Sep 18 17:49:55.264401 ip-10-0-140-54 systemd[1]: Starting Kubernetes Kubelet...
Sep 18 17:49:55.622038 ip-10-0-140-54 kubenswrapper[2561]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 17:49:55.622038 ip-10-0-140-54 kubenswrapper[2561]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 18 17:49:55.622038 ip-10-0-140-54 kubenswrapper[2561]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 17:49:55.622038 ip-10-0-140-54 kubenswrapper[2561]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 18 17:49:55.622038 ip-10-0-140-54 kubenswrapper[2561]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 17:49:55.622841 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.622753    2561 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 18 17:49:55.627274 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627247    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 17:49:55.627274 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627265    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 17:49:55.627274 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627269    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 17:49:55.627274 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627272    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 17:49:55.627274 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627275    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 17:49:55.627274 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627278    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 17:49:55.627274 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627281    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 17:49:55.627274 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627284    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 17:49:55.627557 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627287    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 17:49:55.627557 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627290    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 17:49:55.627557 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627293    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 17:49:55.627557 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627296    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 17:49:55.627557 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627299    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 17:49:55.627557 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627302    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 17:49:55.627557 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627305    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 17:49:55.627557 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627308    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 17:49:55.627557 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627310    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 17:49:55.627557 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627313    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 17:49:55.627557 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627316    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 17:49:55.627557 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627319    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 17:49:55.627557 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627322    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 17:49:55.627557 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627324    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 17:49:55.627557 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627327    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 17:49:55.627557 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627330    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 17:49:55.627557 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627333    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 17:49:55.627557 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627336    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 17:49:55.627557 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627338    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 17:49:55.627557 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627342    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 17:49:55.628023 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627346    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 17:49:55.628023 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627349    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 17:49:55.628023 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627352    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 17:49:55.628023 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627355    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 17:49:55.628023 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627358    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 17:49:55.628023 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627361    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 17:49:55.628023 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627363    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 17:49:55.628023 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627366    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 17:49:55.628023 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627369    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 17:49:55.628023 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627371    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 17:49:55.628023 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627375    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 17:49:55.628023 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627379    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 17:49:55.628023 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627383    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 17:49:55.628023 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627387    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 17:49:55.628023 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627390    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 17:49:55.628023 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627393    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 17:49:55.628023 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627396    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 17:49:55.628023 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627399    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 17:49:55.628023 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627402    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 17:49:55.628536 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627405    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 17:49:55.628536 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627408    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 17:49:55.628536 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627411    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 17:49:55.628536 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627414    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 17:49:55.628536 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627417    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 17:49:55.628536 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627420    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 17:49:55.628536 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627423    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 17:49:55.628536 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627426    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 17:49:55.628536 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627429    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 17:49:55.628536 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627432    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 17:49:55.628536 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627435    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 17:49:55.628536 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627437    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 17:49:55.628536 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627440    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 17:49:55.628536 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627443    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 17:49:55.628536 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627445    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 17:49:55.628536 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627448    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 17:49:55.628536 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627451    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 17:49:55.628536 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627454    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 17:49:55.628536 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627456    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 17:49:55.628536 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627459    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 17:49:55.629026 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627462    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 17:49:55.629026 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627464    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 17:49:55.629026 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627467    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 17:49:55.629026 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627470    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 17:49:55.629026 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627472    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 17:49:55.629026 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627476    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 17:49:55.629026 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627479    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 17:49:55.629026 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627481    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 17:49:55.629026 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627484    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 17:49:55.629026 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627487    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 17:49:55.629026 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627490    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 17:49:55.629026 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627493    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 17:49:55.629026 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627496    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 17:49:55.629026 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627499    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 17:49:55.629026 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627502    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 17:49:55.629026 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627504    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 17:49:55.629026 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627508    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 17:49:55.629026 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627510    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 17:49:55.629026 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627513    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 17:49:55.629026 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627516    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 17:49:55.629518 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627518    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 17:49:55.629518 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627522    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 17:49:55.629518 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627526    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 17:49:55.629518 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627529    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 17:49:55.629518 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627532    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 17:49:55.629518 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627535    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 17:49:55.629518 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627538    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 17:49:55.629518 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627540    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 17:49:55.629518 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627543    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 17:49:55.629518 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627547    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 17:49:55.629518 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627550    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 17:49:55.629518 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.627553    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 17:49:55.629518 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628263    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 17:49:55.629518 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628272    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 17:49:55.629518 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628275    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 17:49:55.629518 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628279    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 17:49:55.629518 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628284    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 17:49:55.629518 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628286    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 17:49:55.629518 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628290    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 17:49:55.629982 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628293    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 17:49:55.629982 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628295    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 17:49:55.629982 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628298    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 17:49:55.629982 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628301    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 17:49:55.629982 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628304    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 17:49:55.629982 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628306    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 17:49:55.629982 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628309    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 17:49:55.629982 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628312    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 17:49:55.629982 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628314    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 17:49:55.629982 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628317    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 17:49:55.629982 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628320    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 17:49:55.629982 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628323    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 17:49:55.629982 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628326    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 17:49:55.629982 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628328    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 17:49:55.629982 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628332    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 17:49:55.629982 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628335    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 17:49:55.629982 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628337    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 17:49:55.629982 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628340    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 17:49:55.629982 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628342    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 17:49:55.630503 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628345    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 17:49:55.630503 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628348    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 17:49:55.630503 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628350    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 17:49:55.630503 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628353    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 17:49:55.630503 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628355    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 17:49:55.630503 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628359    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 17:49:55.630503 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628362    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 17:49:55.630503 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628364    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 17:49:55.630503 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628367    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 17:49:55.630503 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628370    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 17:49:55.630503 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628372    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 17:49:55.630503 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628375    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 17:49:55.630503 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628377    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 17:49:55.630503 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628380    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 17:49:55.630503 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628383    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 17:49:55.630503 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628386    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 17:49:55.630503 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628388    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 17:49:55.630503 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628393    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 17:49:55.630503 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628397    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 17:49:55.630962 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628400    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 17:49:55.630962 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628403    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 17:49:55.630962 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628406    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 17:49:55.630962 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628409    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 17:49:55.630962 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628411    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 17:49:55.630962 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628415    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 17:49:55.630962 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628418    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 17:49:55.630962 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628423    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 17:49:55.630962 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628426    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 17:49:55.630962 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628429    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 17:49:55.630962 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628432    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 17:49:55.630962 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628435    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 17:49:55.630962 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628438    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 17:49:55.630962 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628441    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 17:49:55.630962 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628443    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 17:49:55.630962 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628446    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 17:49:55.630962 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628449    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 17:49:55.630962 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628451    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 17:49:55.630962 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628454    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 17:49:55.630962 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628457    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 17:49:55.631454 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628460    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 17:49:55.631454 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628464    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 17:49:55.631454 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628467    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 17:49:55.631454 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628469    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 17:49:55.631454 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628472    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 17:49:55.631454 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628475    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 17:49:55.631454 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628477    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 17:49:55.631454 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628480    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 17:49:55.631454 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628483    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 17:49:55.631454 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628485    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 17:49:55.631454 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628488    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 17:49:55.631454 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628490    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 17:49:55.631454 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628493    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 17:49:55.631454 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628495    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 17:49:55.631454 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628498    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 17:49:55.631454 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628501    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 17:49:55.631454 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628503    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 17:49:55.631454 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628506    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 17:49:55.631454 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628509    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 17:49:55.631454 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628512    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 17:49:55.631943 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628515    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 17:49:55.631943 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628517    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 17:49:55.631943 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628520    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 17:49:55.631943 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628523    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 17:49:55.631943 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628525    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 17:49:55.631943 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628528    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 17:49:55.631943 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628531    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 17:49:55.631943 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628533    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 17:49:55.631943 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628536    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 17:49:55.631943 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628539    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 17:49:55.631943 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628542    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 17:49:55.631943 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628545    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 17:49:55.631943 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628548    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 17:49:55.631943 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628550    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 17:49:55.631943 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628921    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 17:49:55.631943 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628926    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 17:49:55.631943 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628928    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 17:49:55.631943 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628931    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 17:49:55.631943 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628934    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 17:49:55.631943 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628936    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 17:49:55.632465 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628939    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 17:49:55.632465 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628942    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 17:49:55.632465 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628945    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 17:49:55.632465 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628947    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 17:49:55.632465 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628950    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 17:49:55.632465 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628952    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 17:49:55.632465 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628955    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 17:49:55.632465 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628958    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 17:49:55.632465 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628960    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 17:49:55.632465 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628963    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 17:49:55.632465 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628966    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 17:49:55.632465 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628969    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 17:49:55.632465 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628972    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 17:49:55.632465 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628975    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 17:49:55.632465 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628978    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 17:49:55.632465 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628980    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 17:49:55.632465 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628983    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 17:49:55.632465 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628985    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 17:49:55.632465 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628989    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 17:49:55.632465 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628991    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 17:49:55.632937 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628994    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 17:49:55.632937 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628996    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 17:49:55.632937 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.628999    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 17:49:55.632937 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629001    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 17:49:55.632937 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629004    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 17:49:55.632937 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629007    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 17:49:55.632937 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629012    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 17:49:55.632937 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629016    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 17:49:55.632937 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629018    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 17:49:55.632937 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629022    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 17:49:55.632937 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629025    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 17:49:55.632937 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629028    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 17:49:55.632937 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629031    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 17:49:55.632937 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629033    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 17:49:55.632937 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629036    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 17:49:55.632937 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629039    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 17:49:55.632937 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629041    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 17:49:55.632937 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629044    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 17:49:55.632937 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629047    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 17:49:55.633414 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629049    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 17:49:55.633414 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629052    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 17:49:55.633414 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629054    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 17:49:55.633414 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629057    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 17:49:55.633414 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629060    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 17:49:55.633414 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629063    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 17:49:55.633414 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629066    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 17:49:55.633414 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629068    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 17:49:55.633414 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629071    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 17:49:55.633414 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629073    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 17:49:55.633414 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629076    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 17:49:55.633414 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629080    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 17:49:55.633414 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629084    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 17:49:55.633414 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629087    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 17:49:55.633414 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629090    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 17:49:55.633414 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629094    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 17:49:55.633414 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629097    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 17:49:55.633414 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629100    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 17:49:55.633414 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629103    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 17:49:55.633878 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629106    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 17:49:55.633878 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629109    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 17:49:55.633878 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629112    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 17:49:55.633878 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629129    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 17:49:55.633878 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629132    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 17:49:55.633878 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629135    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 17:49:55.633878 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629137    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 17:49:55.633878 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629140    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 17:49:55.633878 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629143    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 17:49:55.633878 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629145    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 17:49:55.633878 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629149    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 17:49:55.633878 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629151    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 17:49:55.633878 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629154    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 17:49:55.633878 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629156    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 17:49:55.633878 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629159    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 17:49:55.633878 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629162    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 17:49:55.633878 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629164    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 17:49:55.633878 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629167    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 17:49:55.633878 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629170    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 17:49:55.633878 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629172    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 17:49:55.634376 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629175    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 17:49:55.634376 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629178    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 17:49:55.634376 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629180    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 17:49:55.634376 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629183    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 17:49:55.634376 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629186    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 17:49:55.634376 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629189    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 17:49:55.634376 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629192    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 17:49:55.634376 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629194    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 17:49:55.634376 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629196    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 17:49:55.634376 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629199    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 17:49:55.634376 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629201    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 17:49:55.634376 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629204    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 17:49:55.634376 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629207    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 17:49:55.634376 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629209    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 17:49:55.634376 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629212    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 17:49:55.634376 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629289    2561 flags.go:64] FLAG: --address="0.0.0.0"
Sep 18 17:49:55.634376 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629343    2561 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 18 17:49:55.634376 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629354    2561 flags.go:64] FLAG: --anonymous-auth="true"
Sep 18 17:49:55.634376 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629358    2561 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 18 17:49:55.634376 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629362    2561 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 18 17:49:55.634376 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629366    2561 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 18 17:49:55.634909 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629370    2561 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 18 17:49:55.634909 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629375    2561 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 18 17:49:55.634909 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629378    2561 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 18 17:49:55.634909 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629381    2561 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 18 17:49:55.634909 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629385    2561 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 18 17:49:55.634909 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629388    2561 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 18 17:49:55.634909 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629391    2561 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 18 17:49:55.634909 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629395    2561 flags.go:64] FLAG: --cgroup-root=""
Sep 18 17:49:55.634909 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629397    2561 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 18 17:49:55.634909 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629400    2561 flags.go:64] FLAG: --client-ca-file=""
Sep 18 17:49:55.634909 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629404    2561 flags.go:64] FLAG: --cloud-provider="external"
Sep 18 17:49:55.634909 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629408    2561 flags.go:64] FLAG: --cluster-dns="[]"
Sep 18 17:49:55.634909 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629412    2561 flags.go:64] FLAG: --cluster-domain=""
Sep 18 17:49:55.634909 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629414    2561 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 18 17:49:55.634909 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629417    2561 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 18 17:49:55.634909 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629423    2561 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 18 17:49:55.634909 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629426    2561 flags.go:64] FLAG: --container-log-max-files="5"
Sep 18 17:49:55.634909 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629430    2561 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 18 17:49:55.634909 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629434    2561 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 18 17:49:55.634909 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629437    2561 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 18 17:49:55.634909 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629440    2561 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 18 17:49:55.634909 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629443    2561 flags.go:64] FLAG: --contention-profiling="false"
Sep 18 17:49:55.634909 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629446    2561 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 18 17:49:55.634909 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629449    2561 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 18 17:49:55.635503 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629452    2561 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 18 17:49:55.635503 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629455    2561 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 18 17:49:55.635503 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629459    2561 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 18 17:49:55.635503 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629462    2561 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 18 17:49:55.635503 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629466    2561 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 18 17:49:55.635503 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629469    2561 flags.go:64] FLAG: --enable-load-reader="false"
Sep 18 17:49:55.635503 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629472    2561 flags.go:64] FLAG: --enable-server="true"
Sep 18 17:49:55.635503 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629475    2561 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 18 17:49:55.635503 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629479    2561 flags.go:64] FLAG: --event-burst="100"
Sep 18 17:49:55.635503 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629483    2561 flags.go:64] FLAG: --event-qps="50"
Sep 18 17:49:55.635503 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629486    2561 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 18 17:49:55.635503 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629489    2561 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 18 17:49:55.635503 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629492    2561 flags.go:64] FLAG: --eviction-hard=""
Sep 18 17:49:55.635503 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629495    2561 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 18 17:49:55.635503 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629498    2561 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 18 17:49:55.635503 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629501    2561 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 18 17:49:55.635503 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629505    2561 flags.go:64] FLAG: --eviction-soft=""
Sep 18 17:49:55.635503 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629507    2561 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 18 17:49:55.635503 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629510    2561 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 18 17:49:55.635503 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629513    2561 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 18 17:49:55.635503 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629517    2561 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 18 17:49:55.635503 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629520    2561 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 18 17:49:55.635503 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629522    2561 flags.go:64] FLAG: --fail-swap-on="true"
Sep 18 17:49:55.635503 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629527    2561 flags.go:64] FLAG: --feature-gates=""
Sep 18 17:49:55.635503 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629531    2561 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 18 17:49:55.636110 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629534    2561 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 18 17:49:55.636110 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629537    2561 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 18 17:49:55.636110 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629540    2561 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 18 17:49:55.636110 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629543    2561 flags.go:64] FLAG: --healthz-port="10248"
Sep 18 17:49:55.636110 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629547    2561 flags.go:64] FLAG: --help="false"
Sep 18 17:49:55.636110 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629550    2561 flags.go:64] FLAG: --hostname-override="ip-10-0-140-54.ec2.internal"
Sep 18 17:49:55.636110 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629553    2561 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 18 17:49:55.636110 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629556    2561 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 18 17:49:55.636110 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629559    2561 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 18 17:49:55.636110 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629562    2561 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 18 17:49:55.636110 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629566    2561 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 18 17:49:55.636110 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629569    2561 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 18 17:49:55.636110 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629572    2561 flags.go:64] FLAG: --image-service-endpoint=""
Sep 18 17:49:55.636110 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629575    2561 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 18 17:49:55.636110 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629578    2561 flags.go:64] FLAG: --kube-api-burst="100"
Sep 18 17:49:55.636110 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629581    2561 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 18 17:49:55.636110 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629584    2561 flags.go:64] FLAG: --kube-api-qps="50"
Sep 18 17:49:55.636110 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629587    2561 flags.go:64] FLAG: --kube-reserved=""
Sep 18 17:49:55.636110 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629590    2561 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 18 17:49:55.636110 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629593    2561 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 18 17:49:55.636110 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629596    2561 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 18 17:49:55.636110 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629599    2561 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 18 17:49:55.636110 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629602    2561 flags.go:64] FLAG: --lock-file=""
Sep 18 17:49:55.636110 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629605    2561 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 18 17:49:55.636759 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629607    2561 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 18 17:49:55.636759 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629610    2561 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 18 17:49:55.636759 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629616    2561 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 18 17:49:55.636759 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629619    2561 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 18 17:49:55.636759 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629622    2561 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 18 17:49:55.636759 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629625    2561 flags.go:64] FLAG: --logging-format="text"
Sep 18 17:49:55.636759 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629630    2561 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 18 17:49:55.636759 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629633    2561 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 18 17:49:55.636759 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629636    2561 flags.go:64] FLAG: --manifest-url=""
Sep 18 17:49:55.636759 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629639    2561 flags.go:64] FLAG: --manifest-url-header=""
Sep 18 17:49:55.636759 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629643    2561 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 18 17:49:55.636759 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629647    2561 flags.go:64] FLAG: --max-open-files="1000000"
Sep 18 17:49:55.636759 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629650    2561 flags.go:64] FLAG: --max-pods="110"
Sep 18 17:49:55.636759 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629653    2561 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 18 17:49:55.636759 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629656    2561 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 18 17:49:55.636759 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629659    2561 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 18 17:49:55.636759 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629662    2561 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 18 17:49:55.636759 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629665    2561 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 18 17:49:55.636759 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629668    2561 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 18 17:49:55.636759 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629671    2561 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 18 17:49:55.636759 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629678    2561 flags.go:64] FLAG: --node-status-max-images="50"
Sep 18 17:49:55.636759 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629682    2561 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 18 17:49:55.636759 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629685    2561 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 18 17:49:55.636759 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629690    2561 flags.go:64] FLAG: --pod-cidr=""
Sep 18 17:49:55.637366 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629693    2561 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 18 17:49:55.637366 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629699    2561 flags.go:64] FLAG: --pod-manifest-path=""
Sep 18 17:49:55.637366 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629702    2561 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 18 17:49:55.637366 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629705    2561 flags.go:64] FLAG: --pods-per-core="0"
Sep 18 17:49:55.637366 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629708    2561 flags.go:64] FLAG: --port="10250"
Sep 18 17:49:55.637366 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629711    2561 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 18 17:49:55.637366 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629714    2561 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-05a021ad636518451"
Sep 18 17:49:55.637366 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629717    2561 flags.go:64] FLAG: --qos-reserved=""
Sep 18 17:49:55.637366 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629720    2561 flags.go:64] FLAG: --read-only-port="10255"
Sep 18 17:49:55.637366 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629723    2561 flags.go:64] FLAG: --register-node="true"
Sep 18 17:49:55.637366 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629726    2561 flags.go:64] FLAG: --register-with-taints=""
Sep 18 17:49:55.637366 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629729    2561 flags.go:64] FLAG: --registry-burst="10"
Sep 18 17:49:55.637366 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629733    2561 flags.go:64] FLAG: --registry-qps="5"
Sep 18 17:49:55.637366 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629736    2561 flags.go:64] FLAG: --reserved-cpus=""
Sep 18 17:49:55.637366 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629739    2561 flags.go:64] FLAG: --reserved-memory=""
Sep 18 17:49:55.637366 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629744    2561 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 18 17:49:55.637366 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629747    2561 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 18 17:49:55.637366 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629750    2561 flags.go:64] FLAG: --rotate-certificates="false"
Sep 18 17:49:55.637366 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629753    2561 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 18 17:49:55.637366 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629755    2561 flags.go:64] FLAG: --runonce="false"
Sep 18 17:49:55.637366 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629758    2561 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 18 17:49:55.637366 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629761    2561 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 18 17:49:55.637366 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629764    2561 flags.go:64] FLAG: --seccomp-default="false"
Sep 18 17:49:55.637366 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629767    2561 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 18 17:49:55.637366 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629770    2561 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 18 17:49:55.638212 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629773    2561 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 18 17:49:55.638212 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629777    2561 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 18 17:49:55.638212 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629780    2561 flags.go:64] FLAG: --storage-driver-password="root"
Sep 18 17:49:55.638212 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629783    2561 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 18 17:49:55.638212 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629786    2561 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 18 17:49:55.638212 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629788    2561 flags.go:64] FLAG: --storage-driver-user="root"
Sep 18 17:49:55.638212 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629793    2561 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 18 17:49:55.638212 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629797    2561 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 18 17:49:55.638212 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629800    2561 flags.go:64] FLAG: --system-cgroups=""
Sep 18 17:49:55.638212 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629802    2561 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 18 17:49:55.638212 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629808    2561 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 18 17:49:55.638212 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629811    2561 flags.go:64] FLAG: --tls-cert-file=""
Sep 18 17:49:55.638212 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629814    2561 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 18 17:49:55.638212 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629817    2561 flags.go:64] FLAG: --tls-min-version=""
Sep 18 17:49:55.638212 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629821    2561 flags.go:64] FLAG: --tls-private-key-file=""
Sep 18 17:49:55.638212 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629824    2561 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 18 17:49:55.638212 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629827    2561 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 18 17:49:55.638212 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629830    2561 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 18 17:49:55.638212 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629833    2561 flags.go:64] FLAG: --v="2"
Sep 18 17:49:55.638212 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629837    2561 flags.go:64] FLAG: --version="false"
Sep 18 17:49:55.638212 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629841    2561 flags.go:64] FLAG: --vmodule=""
Sep 18 17:49:55.638212 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629845    2561 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 18 17:49:55.638212 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.629850    2561 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 18 17:49:55.638212 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629964    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 17:49:55.638212 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629968    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 17:49:55.639320 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629972    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 17:49:55.639320 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629975    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 17:49:55.639320 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629978    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 17:49:55.639320 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629981    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 17:49:55.639320 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629984    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 17:49:55.639320 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629987    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 17:49:55.639320 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629990    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 17:49:55.639320 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629993    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 17:49:55.639320 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629996    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 17:49:55.639320 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.629998    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 17:49:55.639320 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630001    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 17:49:55.639320 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630004    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 17:49:55.639320 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630007    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 17:49:55.639320 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630012    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 17:49:55.639320 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630014    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 17:49:55.639320 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630017    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 17:49:55.639320 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630019    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 17:49:55.639320 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630022    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 17:49:55.639320 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630025    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 17:49:55.640237 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630027    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 17:49:55.640237 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630030    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 17:49:55.640237 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630032    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 17:49:55.640237 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630035    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 17:49:55.640237 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630039    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 17:49:55.640237 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630042    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 17:49:55.640237 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630045    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 17:49:55.640237 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630048    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 17:49:55.640237 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630051    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 17:49:55.640237 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630054    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 17:49:55.640237 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630058    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 17:49:55.640237 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630061    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 17:49:55.640237 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630064    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 17:49:55.640237 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630067    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 17:49:55.640237 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630070    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 17:49:55.640237 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630074    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 17:49:55.640237 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630077    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 17:49:55.640237 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630080    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 17:49:55.640237 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630082    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 17:49:55.640827 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630085    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 17:49:55.640827 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630087    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 17:49:55.640827 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630090    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 17:49:55.640827 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630093    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 17:49:55.640827 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630096    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 17:49:55.640827 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630099    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 17:49:55.640827 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630102    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 17:49:55.640827 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630108    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 17:49:55.640827 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630111    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 17:49:55.640827 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630127    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 17:49:55.640827 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630130    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 17:49:55.640827 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630133    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 17:49:55.640827 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630137    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 17:49:55.640827 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630139    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 17:49:55.640827 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630143    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 17:49:55.640827 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630145    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 17:49:55.640827 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630148    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 17:49:55.640827 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630151    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 17:49:55.640827 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630153    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 17:49:55.640827 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630156    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 17:49:55.641371 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630159    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 17:49:55.641371 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630162    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 17:49:55.641371 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630164    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 17:49:55.641371 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630169    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 17:49:55.641371 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630172    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 17:49:55.641371 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630174    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 17:49:55.641371 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630177    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 17:49:55.641371 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630180    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 17:49:55.641371 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630182    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 17:49:55.641371 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630185    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 17:49:55.641371 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630188    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 17:49:55.641371 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630191    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 17:49:55.641371 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630193    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 17:49:55.641371 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630196    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 17:49:55.641371 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630198    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 17:49:55.641371 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630201    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 17:49:55.641371 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630204    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 17:49:55.641371 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630206    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 17:49:55.641371 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630209    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 17:49:55.642228 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630213    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 17:49:55.642228 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630216    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 17:49:55.642228 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630218    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 17:49:55.642228 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630221    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 17:49:55.642228 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630223    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 17:49:55.642228 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630227    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 17:49:55.642228 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630229    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 17:49:55.642228 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630232    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 17:49:55.642228 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630235    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 17:49:55.642228 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630238    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 17:49:55.642228 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630240    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 17:49:55.642228 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630243    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 17:49:55.642228 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630245    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 17:49:55.642228 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630248    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 17:49:55.642228 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630250    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 17:49:55.642228 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630253    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 17:49:55.642228 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630257    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 17:49:55.642228 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630260    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 17:49:55.642228 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630263    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 17:49:55.642228 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.630266    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 17:49:55.643044 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.630820    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 17:49:55.643044 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.640334    2561 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 18 17:49:55.643044 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.640521    2561 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 18 17:49:55.643044 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.641323    2561 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 18 17:49:55.643044 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.641337    2561 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 18 17:49:55.643044 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641459    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 17:49:55.643044 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641470    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 17:49:55.643044 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641476    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 17:49:55.643044 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641481    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 17:49:55.643044 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641485    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 17:49:55.643044 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641490    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 17:49:55.643044 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641494    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 17:49:55.643044 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641498    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 17:49:55.643044 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641503    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 17:49:55.643471 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641507    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 17:49:55.643471 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641511    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 17:49:55.643471 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641515    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 17:49:55.643471 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641519    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 17:49:55.643471 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641524    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 17:49:55.643471 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641528    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 17:49:55.643471 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641532    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 17:49:55.643471 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641537    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 17:49:55.643471 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641541    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 17:49:55.643471 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641545    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 17:49:55.643471 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641549    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 17:49:55.643471 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641554    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 17:49:55.643471 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641558    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 17:49:55.643471 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641562    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 17:49:55.643471 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641566    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 17:49:55.643471 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641570    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 17:49:55.643471 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641575    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 17:49:55.643471 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641579    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 17:49:55.643471 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641584    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 17:49:55.643471 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641588    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 17:49:55.644429 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641595    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 17:49:55.644429 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641599    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 17:49:55.644429 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641603    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 17:49:55.644429 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641607    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 17:49:55.644429 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641612    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 17:49:55.644429 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641616    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 17:49:55.644429 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641620    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 17:49:55.644429 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641625    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 17:49:55.644429 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641629    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 17:49:55.644429 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641633    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 17:49:55.644429 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641637    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 17:49:55.644429 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641641    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 17:49:55.644429 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641645    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 17:49:55.644429 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641649    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 17:49:55.644429 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641654    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 17:49:55.644429 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641658    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 17:49:55.644429 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641662    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 17:49:55.644429 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641667    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 17:49:55.644429 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641671    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 17:49:55.644429 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641675    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 17:49:55.645036 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641679    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 17:49:55.645036 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641683    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 17:49:55.645036 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641687    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 17:49:55.645036 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641691    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 17:49:55.645036 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641695    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 17:49:55.645036 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641699    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 17:49:55.645036 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641704    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 17:49:55.645036 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641708    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 17:49:55.645036 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641712    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 17:49:55.645036 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641717    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 17:49:55.645036 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641721    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 17:49:55.645036 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641725    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 17:49:55.645036 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641730    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 17:49:55.645036 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641736    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 17:49:55.645036 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641740    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 17:49:55.645036 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641744    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 17:49:55.645036 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641751    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 17:49:55.645036 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641756    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 17:49:55.645036 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641761    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 17:49:55.645036 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641765    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 17:49:55.645721 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641769    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 17:49:55.645721 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641775    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 17:49:55.645721 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641781    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 17:49:55.645721 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641786    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 17:49:55.645721 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641791    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 17:49:55.645721 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641796    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 17:49:55.645721 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641801    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 17:49:55.645721 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641805    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 17:49:55.645721 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641810    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 17:49:55.645721 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641814    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 17:49:55.645721 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641819    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 17:49:55.645721 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641823    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 17:49:55.645721 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641827    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 17:49:55.645721 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641831    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 17:49:55.645721 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641835    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 17:49:55.645721 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641839    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 17:49:55.645721 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641844    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 17:49:55.645721 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641848    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 17:49:55.645721 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641853    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 17:49:55.646273 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641857    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 17:49:55.646273 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641862    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 17:49:55.646273 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641866    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 17:49:55.646273 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641870    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 17:49:55.646273 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641874    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 17:49:55.646273 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641879    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 17:49:55.646273 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641883    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 17:49:55.646273 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641897    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 17:49:55.646273 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641902    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 17:49:55.646273 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641906    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 17:49:55.646273 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.641910    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 17:49:55.646273 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.641919    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 17:49:55.646273 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642158    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 17:49:55.646273 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642166    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 17:49:55.646273 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642172    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 17:49:55.646788 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642177    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 17:49:55.646788 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642182    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 17:49:55.646788 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642187    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 17:49:55.646788 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642191    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 17:49:55.646788 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642196    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 17:49:55.646788 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642201    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 17:49:55.646788 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642205    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 17:49:55.646788 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642210    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 17:49:55.646788 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642215    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 17:49:55.646788 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642219    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 17:49:55.646788 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642223    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 17:49:55.646788 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642228    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 17:49:55.646788 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642231    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 17:49:55.646788 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642236    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 17:49:55.646788 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642239    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 17:49:55.646788 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642243    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 17:49:55.646788 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642248    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 17:49:55.646788 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642252    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 17:49:55.646788 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642256    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 17:49:55.646788 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642260    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 17:49:55.647294 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642266    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 17:49:55.647294 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642273    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 17:49:55.647294 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642277    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 17:49:55.647294 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642282    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 17:49:55.647294 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642287    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 17:49:55.647294 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642300    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 17:49:55.647294 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642305    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 17:49:55.647294 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642309    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 17:49:55.647294 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642313    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 17:49:55.647294 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642318    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 17:49:55.647294 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642322    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 17:49:55.647294 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642326    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 17:49:55.647294 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642331    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 17:49:55.647294 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642335    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 17:49:55.647294 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642339    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 17:49:55.647294 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642343    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 17:49:55.647294 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642347    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 17:49:55.647294 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642351    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 17:49:55.647294 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642356    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 17:49:55.647819 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642360    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 17:49:55.647819 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642365    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 17:49:55.647819 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642369    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 17:49:55.647819 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642373    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 17:49:55.647819 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642377    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 17:49:55.647819 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642381    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 17:49:55.647819 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642385    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 17:49:55.647819 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642389    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 17:49:55.647819 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642394    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 17:49:55.647819 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642398    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 17:49:55.647819 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642402    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 17:49:55.647819 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642406    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 17:49:55.647819 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642411    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 17:49:55.647819 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642415    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 17:49:55.647819 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642419    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 17:49:55.647819 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642423    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 17:49:55.647819 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642428    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 17:49:55.647819 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642432    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 17:49:55.647819 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642436    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 17:49:55.647819 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642448    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 17:49:55.648470 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642452    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 17:49:55.648470 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642456    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 17:49:55.648470 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642461    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 17:49:55.648470 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642464    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 17:49:55.648470 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642469    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 17:49:55.648470 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642473    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 17:49:55.648470 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642477    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 17:49:55.648470 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642481    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 17:49:55.648470 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642485    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 17:49:55.648470 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642489    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 17:49:55.648470 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642493    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 17:49:55.648470 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642498    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 17:49:55.648470 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642502    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 17:49:55.648470 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642506    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 17:49:55.648470 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642511    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 17:49:55.648470 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642515    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 17:49:55.648470 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642519    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 17:49:55.648470 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642525    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 17:49:55.648470 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642531    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 17:49:55.648926 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642535    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 17:49:55.648926 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642540    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 17:49:55.648926 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642543    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 17:49:55.648926 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642547    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 17:49:55.648926 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642551    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 17:49:55.648926 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642555    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 17:49:55.648926 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642560    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 17:49:55.648926 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642565    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 17:49:55.648926 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642568    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 17:49:55.648926 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642573    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 17:49:55.648926 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642577    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 17:49:55.648926 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642581    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 17:49:55.648926 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642585    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 17:49:55.648926 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642598    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 17:49:55.648926 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642602    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 17:49:55.648926 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642609    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 17:49:55.648926 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642615    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 17:49:55.648926 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:55.642620    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 17:49:55.649415 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.642628    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 17:49:55.649415 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.642860    2561 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 18 17:49:55.649415 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.646513    2561 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 18 17:49:55.649415 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.647539    2561 server.go:1019] "Starting client certificate rotation"
Sep 18 17:49:55.649415 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.647652    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 17:49:55.649415 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.647682    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 17:49:55.650174 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.650157    2561 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 17:49:55.658165 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.658144    2561 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 17:49:55.669207 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.669190    2561 log.go:25] "Validated CRI v1 runtime API"
Sep 18 17:49:55.670765 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.670754    2561 log.go:25] "Validated CRI v1 image API"
Sep 18 17:49:55.671992 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.671972    2561 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 18 17:49:55.674673 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.674650    2561 fs.go:135] Filesystem UUIDs: map[5b66f986-c385-4419-8b2b-3397896f98aa:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 ba8bc9b3-b13b-4d06-a57a-32e155bd64ec:/dev/nvme0n1p4]
Sep 18 17:49:55.674737 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.674673    2561 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 18 17:49:55.679718 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.679613    2561 manager.go:216] Machine: {Timestamp:2026-09-18 17:49:55.678539512 +0000 UTC m=+0.305671124 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3108172 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24804ed83739bb93900a17c62b8467 SystemUUID:ec24804e-d837-39bb-9390-0a17c62b8467 BootID:efbfc0d2-e782-4035-a236-b6a7fd4a41a1 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c7:c0:63:49 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c7:c0:63:49 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:72:cb:9c:03:f4:c5 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 18 17:49:55.679718 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.679713    2561 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 18 17:49:55.679834 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.679823    2561 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 18 17:49:55.680849 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.680831    2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 17:49:55.682527 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.682500    2561 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 18 17:49:55.682668 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.682529    2561 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-54.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 18 17:49:55.683745 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.683734    2561 topology_manager.go:138] "Creating topology manager with none policy"
Sep 18 17:49:55.683782 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.683747    2561 container_manager_linux.go:309] "Creating device plugin manager"
Sep 18 17:49:55.683782 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.683755    2561 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 17:49:55.684381 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.684370    2561 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 17:49:55.685471 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.685462    2561 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 18 17:49:55.689180 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.689168    2561 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 17:49:55.689285 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.689275    2561 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 18 17:49:55.689330 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.689305    2561 kubelet.go:486] "Attempting to sync node with API server"
Sep 18 17:49:55.689330 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.689314    2561 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 18 17:49:55.689330 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.689330    2561 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 18 17:49:55.689446 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.689340    2561 kubelet.go:398] "Adding apiserver pod source"
Sep 18 17:49:55.689446 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.689349    2561 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 18 17:49:55.689744 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.689732    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 17:49:55.689783 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.689757    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 17:49:55.692700 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.692682    2561 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 18 17:49:55.694677 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.694665    2561 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 18 17:49:55.694735 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.694694    2561 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 18 17:49:55.696221 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.696206    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 18 17:49:55.696279 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.696227    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 18 17:49:55.696279 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.696235    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 18 17:49:55.696279 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.696240    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 18 17:49:55.696279 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.696250    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 18 17:49:55.696279 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.696255    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 18 17:49:55.696279 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.696270    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 18 17:49:55.696279 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.696275    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 18 17:49:55.696279 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.696282    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 18 17:49:55.696523 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.696377    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 18 17:49:55.696523 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.696392    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 18 17:49:55.696523 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.696398    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 18 17:49:55.697835 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.697824    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 18 17:49:55.697835 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.697833    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 18 17:49:55.701160 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.701135    2561 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-54.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 17:49:55.701298 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:49:55.701275    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-54.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 18 17:49:55.701363 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:49:55.701324    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 18 17:49:55.702168 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.702154    2561 server.go:1268] "Started kubelet"
Sep 18 17:49:55.702268 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.702227    2561 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 18 17:49:55.702350 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.702302    2561 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 18 17:49:55.702435 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.702368    2561 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 18 17:49:55.705578 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.705555    2561 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 18 17:49:55.706589 ip-10-0-140-54 systemd[1]: Started Kubernetes Kubelet.
Sep 18 17:49:55.707808 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:49:55.707150    2561 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 18 17:49:55.708770 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.708755    2561 server.go:310] "Adding debug handlers to kubelet server"
Sep 18 17:49:55.713352 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:49:55.712370    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-54.ec2.internal.18d67b983c3b39a1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-54.ec2.internal,UID:ip-10-0-140-54.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-54.ec2.internal,},FirstTimestamp:2026-09-18 17:49:55.702110625 +0000 UTC m=+0.329242236,LastTimestamp:2026-09-18 17:49:55.702110625 +0000 UTC m=+0.329242236,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-54.ec2.internal,}"
Sep 18 17:49:55.714262 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.714240    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 18 17:49:55.714974 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.714958    2561 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 18 17:49:55.715108 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:49:55.715087    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-54.ec2.internal\" not found"
Sep 18 17:49:55.715703 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.715687    2561 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 18 17:49:55.715815 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.715689    2561 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 18 17:49:55.715815 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.715818    2561 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 18 17:49:55.715966 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.715856    2561 factory.go:55] Registering systemd factory
Sep 18 17:49:55.715966 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.715869    2561 factory.go:223] Registration of the systemd container factory successfully
Sep 18 17:49:55.715966 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.715939    2561 reconstruct.go:97] "Volume reconstruction finished"
Sep 18 17:49:55.715966 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.715947    2561 reconciler.go:29] "Reconciler: start to sync state"
Sep 18 17:49:55.716103 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.716061    2561 factory.go:153] Registering CRI-O factory
Sep 18 17:49:55.716103 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.716073    2561 factory.go:223] Registration of the crio container factory successfully
Sep 18 17:49:55.716230 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.716130    2561 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 18 17:49:55.716230 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.716149    2561 factory.go:103] Registering Raw factory
Sep 18 17:49:55.716230 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.716177    2561 manager.go:1196] Started watching for new ooms in manager
Sep 18 17:49:55.716632 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.716607    2561 manager.go:349] Starting recovery of all containers
Sep 18 17:49:55.722670 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:49:55.722635    2561 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-54.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 18 17:49:55.722886 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:49:55.722862    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 18 17:49:55.728013 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.727996    2561 manager.go:354] Recovery completed
Sep 18 17:49:55.733678 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.733661    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 17:49:55.735586 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.735564    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-54.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 17:49:55.735681 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.735599    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-54.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 17:49:55.735681 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.735610    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-54.ec2.internal" event="NodeHasSufficientPID"
Sep 18 17:49:55.736564 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.736548    2561 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 18 17:49:55.736564 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.736560    2561 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 18 17:49:55.736691 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.736575    2561 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 17:49:55.736691 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.736601    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 17:49:55.737351 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.737337    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-54.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 17:49:55.737441 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.737362    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-54.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 17:49:55.737441 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.737374    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-54.ec2.internal" event="NodeHasSufficientPID"
Sep 18 17:49:55.737569 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:49:55.737492    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-54.ec2.internal.18d67b983e39fac2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-54.ec2.internal,UID:ip-10-0-140-54.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-54.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-54.ec2.internal,},FirstTimestamp:2026-09-18 17:49:55.735583426 +0000 UTC m=+0.362715041,LastTimestamp:2026-09-18 17:49:55.735583426 +0000 UTC m=+0.362715041,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-54.ec2.internal,}"
Sep 18 17:49:55.738895 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.738880    2561 policy_none.go:49] "None policy: Start"
Sep 18 17:49:55.738895 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.738897    2561 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 18 17:49:55.739014 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.738907    2561 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 18 17:49:55.740483 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.740468    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8lshj"
Sep 18 17:49:55.741041 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.741029    2561 policy_none.go:47] "Start"
Sep 18 17:49:55.747173 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:49:55.747078    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-54.ec2.internal.18d67b983e3a4e3e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-54.ec2.internal,UID:ip-10-0-140-54.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-54.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-54.ec2.internal,},FirstTimestamp:2026-09-18 17:49:55.735604798 +0000 UTC m=+0.362736409,LastTimestamp:2026-09-18 17:49:55.735604798 +0000 UTC m=+0.362736409,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-54.ec2.internal,}"
Sep 18 17:49:55.747557 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.747541    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8lshj"
Sep 18 17:49:55.769701 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.769687    2561 manager.go:337] "Starting Device Plugin manager"
Sep 18 17:49:55.769795 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:49:55.769733    2561 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 18 17:49:55.769795 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.769746    2561 server.go:87] "Starting device plugin registration server"
Sep 18 17:49:55.770088 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.770070    2561 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 18 17:49:55.770314 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.770088    2561 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 18 17:49:55.770314 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.770262    2561 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 18 17:49:55.770445 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.770331    2561 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 18 17:49:55.770445 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.770341    2561 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 18 17:49:55.772163 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:49:55.772142    2561 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 18 17:49:55.772249 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:49:55.772193    2561 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-54.ec2.internal\" not found"
Sep 18 17:49:55.853251 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.853226    2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 18 17:49:55.854420 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.854406    2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 18 17:49:55.854471 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.854462    2561 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 18 17:49:55.854521 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.854485    2561 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 18 17:49:55.854566 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:49:55.854524    2561 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 18 17:49:55.857323 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.857304    2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 17:49:55.870585 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.870560    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 17:49:55.871434 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.871418    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-54.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 17:49:55.871513 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.871448    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-54.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 17:49:55.871513 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.871458    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-54.ec2.internal" event="NodeHasSufficientPID"
Sep 18 17:49:55.871513 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.871479    2561 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-54.ec2.internal"
Sep 18 17:49:55.880261 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.880202    2561 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-54.ec2.internal"
Sep 18 17:49:55.880261 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:49:55.880220    2561 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-54.ec2.internal\": node \"ip-10-0-140-54.ec2.internal\" not found"
Sep 18 17:49:55.909591 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:49:55.909574    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-54.ec2.internal\" not found"
Sep 18 17:49:55.955530 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.955508    2561 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-54.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-54.ec2.internal"]
Sep 18 17:49:55.955601 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.955576    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 17:49:55.956862 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.956846    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-54.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 17:49:55.956924 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.956875    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-54.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 17:49:55.956924 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.956885    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-54.ec2.internal" event="NodeHasSufficientPID"
Sep 18 17:49:55.959224 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.959212    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 17:49:55.959334 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.959321    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-54.ec2.internal"
Sep 18 17:49:55.959376 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.959365    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 17:49:55.959898 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.959883    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-54.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 17:49:55.959977 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.959905    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-54.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 17:49:55.959977 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.959915    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-54.ec2.internal" event="NodeHasSufficientPID"
Sep 18 17:49:55.959977 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.959928    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-54.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 17:49:55.959977 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.959947    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-54.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 17:49:55.959977 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.959960    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-54.ec2.internal" event="NodeHasSufficientPID"
Sep 18 17:49:55.962044 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.962028    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-54.ec2.internal"
Sep 18 17:49:55.962127 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.962054    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 17:49:55.962727 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.962701    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-54.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 17:49:55.962801 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.962731    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-54.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 17:49:55.962801 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:55.962741    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-54.ec2.internal" event="NodeHasSufficientPID"
Sep 18 17:49:55.981649 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:49:55.981631    2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-54.ec2.internal\" not found" node="ip-10-0-140-54.ec2.internal"
Sep 18 17:49:55.985375 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:49:55.985359    2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-54.ec2.internal\" not found" node="ip-10-0-140-54.ec2.internal"
Sep 18 17:49:56.010065 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:49:56.010046    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-54.ec2.internal\" not found"
Sep 18 17:49:56.017348 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:56.017327    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bfcddef9dbb20737aafa5928064e8e0d-config\") pod \"kube-apiserver-proxy-ip-10-0-140-54.ec2.internal\" (UID: \"bfcddef9dbb20737aafa5928064e8e0d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-54.ec2.internal"
Sep 18 17:49:56.017430 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:56.017357    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0b9f4e0872eb36d1a9eb7f46a41c431b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-54.ec2.internal\" (UID: \"0b9f4e0872eb36d1a9eb7f46a41c431b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-54.ec2.internal"
Sep 18 17:49:56.017430 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:56.017382    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b9f4e0872eb36d1a9eb7f46a41c431b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-54.ec2.internal\" (UID: \"0b9f4e0872eb36d1a9eb7f46a41c431b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-54.ec2.internal"
Sep 18 17:49:56.110741 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:49:56.110712    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-54.ec2.internal\" not found"
Sep 18 17:49:56.118068 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:56.118046    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bfcddef9dbb20737aafa5928064e8e0d-config\") pod \"kube-apiserver-proxy-ip-10-0-140-54.ec2.internal\" (UID: \"bfcddef9dbb20737aafa5928064e8e0d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-54.ec2.internal"
Sep 18 17:49:56.118145 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:56.118073    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0b9f4e0872eb36d1a9eb7f46a41c431b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-54.ec2.internal\" (UID: \"0b9f4e0872eb36d1a9eb7f46a41c431b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-54.ec2.internal"
Sep 18 17:49:56.118145 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:56.118089    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b9f4e0872eb36d1a9eb7f46a41c431b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-54.ec2.internal\" (UID: \"0b9f4e0872eb36d1a9eb7f46a41c431b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-54.ec2.internal"
Sep 18 17:49:56.118260 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:56.118175    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bfcddef9dbb20737aafa5928064e8e0d-config\") pod \"kube-apiserver-proxy-ip-10-0-140-54.ec2.internal\" (UID: \"bfcddef9dbb20737aafa5928064e8e0d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-54.ec2.internal"
Sep 18 17:49:56.118260 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:56.118181    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b9f4e0872eb36d1a9eb7f46a41c431b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-54.ec2.internal\" (UID: \"0b9f4e0872eb36d1a9eb7f46a41c431b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-54.ec2.internal"
Sep 18 17:49:56.118260 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:56.118187    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0b9f4e0872eb36d1a9eb7f46a41c431b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-54.ec2.internal\" (UID: \"0b9f4e0872eb36d1a9eb7f46a41c431b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-54.ec2.internal"
Sep 18 17:49:56.211018 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:49:56.210941    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-54.ec2.internal\" not found"
Sep 18 17:49:56.285339 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:56.285311    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-54.ec2.internal"
Sep 18 17:49:56.288500 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:56.288482    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-54.ec2.internal"
Sep 18 17:49:56.311043 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:49:56.311024    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-54.ec2.internal\" not found"
Sep 18 17:49:56.411593 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:49:56.411548    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-54.ec2.internal\" not found"
Sep 18 17:49:56.516134 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:49:56.516059    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-54.ec2.internal\" not found"
Sep 18 17:49:56.618345 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:49:56.618290    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-54.ec2.internal\" not found"
Sep 18 17:49:56.648884 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:56.648859    2561 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 18 17:49:56.649366 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:56.649052    2561 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 17:49:56.719004 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:49:56.718974    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-54.ec2.internal\" not found"
Sep 18 17:49:56.752954 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:56.752919    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-17 17:44:56 +0000 UTC" deadline="2028-05-12 16:36:04.199236317 +0000 UTC"
Sep 18 17:49:56.752954 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:56.752946    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14446h46m7.446293666s"
Sep 18 17:49:56.764787 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:56.764766    2561 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 17:49:56.819844 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:49:56.819788    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-54.ec2.internal\" not found"
Sep 18 17:49:56.869342 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:56.869324    2561 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 17:49:56.915816 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:56.915792    2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-54.ec2.internal"
Sep 18 17:49:56.931086 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:56.931068    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 17:49:56.932897 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:56.932882    2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-54.ec2.internal"
Sep 18 17:49:56.943077 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:56.943060    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 17:49:57.206796 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:57.206564    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfcddef9dbb20737aafa5928064e8e0d.slice/crio-d6b06312a4ed0014667f4f74cdd5754f2a1a7714112cc672729eaeda8ca76448 WatchSource:0}: Error finding container d6b06312a4ed0014667f4f74cdd5754f2a1a7714112cc672729eaeda8ca76448: Status 404 returned error can't find the container with id d6b06312a4ed0014667f4f74cdd5754f2a1a7714112cc672729eaeda8ca76448
Sep 18 17:49:57.207158 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:57.207137    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b9f4e0872eb36d1a9eb7f46a41c431b.slice/crio-6d4a226cb1a63a9d983d953dd70eca5a926f019a9a5953eabf319fb4256fc68d WatchSource:0}: Error finding container 6d4a226cb1a63a9d983d953dd70eca5a926f019a9a5953eabf319fb4256fc68d: Status 404 returned error can't find the container with id 6d4a226cb1a63a9d983d953dd70eca5a926f019a9a5953eabf319fb4256fc68d
Sep 18 17:49:57.212166 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.212149    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 17:49:57.240220 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.240121    2561 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 17:49:57.644289 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.644209    2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 17:49:57.690348 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.690314    2561 apiserver.go:52] "Watching apiserver"
Sep 18 17:49:57.697678 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.697654    2561 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 18 17:49:57.698042 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.698020    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-k2xs7","kube-system/kube-apiserver-proxy-ip-10-0-140-54.ec2.internal","openshift-dns/node-resolver-q2btm","openshift-network-diagnostics/network-check-target-bgwtf","openshift-network-operator/iptables-alerter-2mqnb","openshift-ovn-kubernetes/ovnkube-node-rkcgk","kube-system/global-pull-secret-syncer-mvtdm","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xphc9","openshift-cluster-node-tuning-operator/tuned-mqjzn","openshift-image-registry/node-ca-zvmrv","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-54.ec2.internal","openshift-multus/multus-4wsjj","openshift-multus/multus-additional-cni-plugins-cqzh9","openshift-multus/network-metrics-daemon-tmtsv"]
Sep 18 17:49:57.702037 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.702021    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mqjzn"
Sep 18 17:49:57.704289 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.704271    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tmtsv"
Sep 18 17:49:57.704393 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:49:57.704327    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tmtsv" podUID="45a7fd6c-fe96-4a1a-b3a5-2cb78c5435e8"
Sep 18 17:49:57.704653 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.704628    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 18 17:49:57.704794 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.704770    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 18 17:49:57.704903 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.704887    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-p4ppk\""
Sep 18 17:49:57.706639 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.706619    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-q2btm"
Sep 18 17:49:57.709071 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.709051    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bgwtf"
Sep 18 17:49:57.709183 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:49:57.709151    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bgwtf" podUID="234003fa-8d5d-4108-8f62-859ea05eb9aa"
Sep 18 17:49:57.709183 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.709177    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9hbx7\""
Sep 18 17:49:57.709415 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.709400    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 18 17:49:57.709491 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.709437    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 18 17:49:57.711453 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.711433    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2mqnb"
Sep 18 17:49:57.713726 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.713708    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 18 17:49:57.713974 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.713958    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk"
Sep 18 17:49:57.714479 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.714117    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-v5xp5\""
Sep 18 17:49:57.714479 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.714137    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 18 17:49:57.714479 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.714143    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 18 17:49:57.714479 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.714409    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 18 17:49:57.717032 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.716498    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 18 17:49:57.717032 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.716519    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 18 17:49:57.717032 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.716519    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xl4fg\""
Sep 18 17:49:57.717032 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.716595    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 18 17:49:57.717032 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.716734    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4wsjj"
Sep 18 17:49:57.717032 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.716856    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 18 17:49:57.717032 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.716932    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 18 17:49:57.717797 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.717778    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 18 17:49:57.720823 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.720751    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xphc9"
Sep 18 17:49:57.720823 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.720772    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 18 17:49:57.720969 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.720865    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 18 17:49:57.721139 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.721114    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-w6jlq\""
Sep 18 17:49:57.721401 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.721383    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 18 17:49:57.721479 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.721464    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 18 17:49:57.722219 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.722195    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mvtdm"
Sep 18 17:49:57.722320 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:49:57.722280    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mvtdm" podUID="7a3ae2d8-a30b-4c83-a53c-634fb07621d6"
Sep 18 17:49:57.723235 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.723215    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-p8hm9\""
Sep 18 17:49:57.723333 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.723211    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 18 17:49:57.723592 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.723575    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 18 17:49:57.723679 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.723646    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 18 17:49:57.725381 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.724829    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zvmrv"
Sep 18 17:49:57.725478 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.725458    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/548befd4-2d47-4b4d-a376-c295e0dc141b-cni-binary-copy\") pod \"multus-4wsjj\" (UID: \"548befd4-2d47-4b4d-a376-c295e0dc141b\") " pod="openshift-multus/multus-4wsjj"
Sep 18 17:49:57.725524 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.725486    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/548befd4-2d47-4b4d-a376-c295e0dc141b-host-run-netns\") pod \"multus-4wsjj\" (UID: \"548befd4-2d47-4b4d-a376-c295e0dc141b\") " pod="openshift-multus/multus-4wsjj"
Sep 18 17:49:57.725562 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.725502    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/548befd4-2d47-4b4d-a376-c295e0dc141b-multus-conf-dir\") pod \"multus-4wsjj\" (UID: \"548befd4-2d47-4b4d-a376-c295e0dc141b\") " pod="openshift-multus/multus-4wsjj"
Sep 18 17:49:57.725562 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.725540    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/548befd4-2d47-4b4d-a376-c295e0dc141b-host-run-multus-certs\") pod \"multus-4wsjj\" (UID: \"548befd4-2d47-4b4d-a376-c295e0dc141b\") " pod="openshift-multus/multus-4wsjj"
Sep 18 17:49:57.725632 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.725575    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhc4v\" (UniqueName: \"kubernetes.io/projected/548befd4-2d47-4b4d-a376-c295e0dc141b-kube-api-access-nhc4v\") pod \"multus-4wsjj\" (UID: \"548befd4-2d47-4b4d-a376-c295e0dc141b\") " pod="openshift-multus/multus-4wsjj"
Sep 18 17:49:57.725632 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.725591    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/291a480f-a747-4910-a467-676a77e04eb0-etc-sysctl-d\") pod \"tuned-mqjzn\" (UID: \"291a480f-a747-4910-a467-676a77e04eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-mqjzn"
Sep 18 17:49:57.725632 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.725607    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c5d026b7-af6b-4899-85f4-6554c0f7cd52-run-openvswitch\") pod \"ovnkube-node-rkcgk\" (UID: \"c5d026b7-af6b-4899-85f4-6554c0f7cd52\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk"
Sep 18 17:49:57.725632 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.725621    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c5d026b7-af6b-4899-85f4-6554c0f7cd52-ovnkube-config\") pod \"ovnkube-node-rkcgk\" (UID: \"c5d026b7-af6b-4899-85f4-6554c0f7cd52\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk"
Sep 18 17:49:57.725755 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.725645    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/65f6cbfa-00b5-4a5c-be75-28392cf8a1a8-host-slash\") pod \"iptables-alerter-2mqnb\" (UID: \"65f6cbfa-00b5-4a5c-be75-28392cf8a1a8\") " pod="openshift-network-operator/iptables-alerter-2mqnb"
Sep 18 17:49:57.725755 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.725660    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf9qt\" (UniqueName: \"kubernetes.io/projected/65f6cbfa-00b5-4a5c-be75-28392cf8a1a8-kube-api-access-lf9qt\") pod \"iptables-alerter-2mqnb\" (UID: \"65f6cbfa-00b5-4a5c-be75-28392cf8a1a8\") " pod="openshift-network-operator/iptables-alerter-2mqnb"
Sep 18 17:49:57.725755 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.725699    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/291a480f-a747-4910-a467-676a77e04eb0-var-lib-kubelet\") pod \"tuned-mqjzn\" (UID: \"291a480f-a747-4910-a467-676a77e04eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-mqjzn"
Sep 18 17:49:57.725755 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.725735    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxwb7\" (UniqueName: \"kubernetes.io/projected/1907e944-be36-42b2-bc90-90938a227371-kube-api-access-qxwb7\") pod \"node-resolver-q2btm\" (UID: \"1907e944-be36-42b2-bc90-90938a227371\") " pod="openshift-dns/node-resolver-q2btm"
Sep 18 17:49:57.725882 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.725768    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/291a480f-a747-4910-a467-676a77e04eb0-etc-kubernetes\") pod \"tuned-mqjzn\" (UID: \"291a480f-a747-4910-a467-676a77e04eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-mqjzn"
Sep 18 17:49:57.725882 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.725799    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/291a480f-a747-4910-a467-676a77e04eb0-etc-tuned\") pod \"tuned-mqjzn\" (UID: \"291a480f-a747-4910-a467-676a77e04eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-mqjzn"
Sep 18 17:49:57.725882 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.725820    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1907e944-be36-42b2-bc90-90938a227371-tmp-dir\") pod \"node-resolver-q2btm\" (UID: \"1907e944-be36-42b2-bc90-90938a227371\") " pod="openshift-dns/node-resolver-q2btm"
Sep 18 17:49:57.725882 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.725835    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c5d026b7-af6b-4899-85f4-6554c0f7cd52-run-systemd\") pod \"ovnkube-node-rkcgk\" (UID: \"c5d026b7-af6b-4899-85f4-6554c0f7cd52\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk"
Sep 18 17:49:57.725882 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.725855    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c5d026b7-af6b-4899-85f4-6554c0f7cd52-run-ovn\") pod \"ovnkube-node-rkcgk\" (UID: \"c5d026b7-af6b-4899-85f4-6554c0f7cd52\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk"
Sep 18 17:49:57.725882 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.725877    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/291a480f-a747-4910-a467-676a77e04eb0-lib-modules\") pod \"tuned-mqjzn\" (UID: \"291a480f-a747-4910-a467-676a77e04eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-mqjzn"
Sep 18 17:49:57.726078 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.725904    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/291a480f-a747-4910-a467-676a77e04eb0-host\") pod \"tuned-mqjzn\" (UID: \"291a480f-a747-4910-a467-676a77e04eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-mqjzn"
Sep 18 17:49:57.726078 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.725927    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c5d026b7-af6b-4899-85f4-6554c0f7cd52-ovnkube-script-lib\") pod \"ovnkube-node-rkcgk\" (UID: \"c5d026b7-af6b-4899-85f4-6554c0f7cd52\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk"
Sep 18 17:49:57.726078 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.725965    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5d026b7-af6b-4899-85f4-6554c0f7cd52-host-kubelet\") pod \"ovnkube-node-rkcgk\" (UID: \"c5d026b7-af6b-4899-85f4-6554c0f7cd52\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk"
Sep 18 17:49:57.726078 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.726000    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c5d026b7-af6b-4899-85f4-6554c0f7cd52-systemd-units\") pod \"ovnkube-node-rkcgk\" (UID: \"c5d026b7-af6b-4899-85f4-6554c0f7cd52\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk"
Sep 18 17:49:57.726078 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.726036    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5d026b7-af6b-4899-85f4-6554c0f7cd52-host-run-ovn-kubernetes\") pod \"ovnkube-node-rkcgk\" (UID: \"c5d026b7-af6b-4899-85f4-6554c0f7cd52\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk"
Sep 18 17:49:57.726319 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.726078    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c5d026b7-af6b-4899-85f4-6554c0f7cd52-host-cni-bin\") pod \"ovnkube-node-rkcgk\" (UID: \"c5d026b7-af6b-4899-85f4-6554c0f7cd52\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk"
Sep 18 17:49:57.726319 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.726155    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c5d026b7-af6b-4899-85f4-6554c0f7cd52-host-slash\") pod \"ovnkube-node-rkcgk\" (UID: \"c5d026b7-af6b-4899-85f4-6554c0f7cd52\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk"
Sep 18 17:49:57.726319 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.726229    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c5d026b7-af6b-4899-85f4-6554c0f7cd52-host-run-netns\") pod \"ovnkube-node-rkcgk\" (UID: \"c5d026b7-af6b-4899-85f4-6554c0f7cd52\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk"
Sep 18 17:49:57.726319 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.726252    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/548befd4-2d47-4b4d-a376-c295e0dc141b-host-var-lib-cni-multus\") pod \"multus-4wsjj\" (UID: \"548befd4-2d47-4b4d-a376-c295e0dc141b\") " pod="openshift-multus/multus-4wsjj"
Sep 18 17:49:57.726319 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.726299    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/291a480f-a747-4910-a467-676a77e04eb0-sys\") pod \"tuned-mqjzn\" (UID: \"291a480f-a747-4910-a467-676a77e04eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-mqjzn"
Sep 18 17:49:57.726566 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.726333    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/65f6cbfa-00b5-4a5c-be75-28392cf8a1a8-iptables-alerter-script\") pod \"iptables-alerter-2mqnb\" (UID: \"65f6cbfa-00b5-4a5c-be75-28392cf8a1a8\") " pod="openshift-network-operator/iptables-alerter-2mqnb"
Sep 18 17:49:57.726566 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.726354    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c5d026b7-af6b-4899-85f4-6554c0f7cd52-host-cni-netd\") pod \"ovnkube-node-rkcgk\" (UID: \"c5d026b7-af6b-4899-85f4-6554c0f7cd52\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk"
Sep 18 17:49:57.726566 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.726391    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5d026b7-af6b-4899-85f4-6554c0f7cd52-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rkcgk\" (UID: \"c5d026b7-af6b-4899-85f4-6554c0f7cd52\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk"
Sep 18 17:49:57.726566 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.726427    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c5d026b7-af6b-4899-85f4-6554c0f7cd52-env-overrides\") pod \"ovnkube-node-rkcgk\" (UID: \"c5d026b7-af6b-4899-85f4-6554c0f7cd52\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk"
Sep 18 17:49:57.726566 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.726456    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45a7fd6c-fe96-4a1a-b3a5-2cb78c5435e8-metrics-certs\") pod \"network-metrics-daemon-tmtsv\" (UID: \"45a7fd6c-fe96-4a1a-b3a5-2cb78c5435e8\") " pod="openshift-multus/network-metrics-daemon-tmtsv"
Sep 18 17:49:57.726566 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.726480    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c5d026b7-af6b-4899-85f4-6554c0f7cd52-var-lib-openvswitch\") pod \"ovnkube-node-rkcgk\" (UID: \"c5d026b7-af6b-4899-85f4-6554c0f7cd52\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk"
Sep 18 17:49:57.726566 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.726504    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c5d026b7-af6b-4899-85f4-6554c0f7cd52-node-log\") pod \"ovnkube-node-rkcgk\" (UID: \"c5d026b7-af6b-4899-85f4-6554c0f7cd52\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk"
Sep 18 17:49:57.726566 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.726528    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/291a480f-a747-4910-a467-676a77e04eb0-etc-modprobe-d\") pod \"tuned-mqjzn\" (UID: \"291a480f-a747-4910-a467-676a77e04eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-mqjzn"
Sep 18 17:49:57.726566 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.726556    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crvhz\" (UniqueName: \"kubernetes.io/projected/c5d026b7-af6b-4899-85f4-6554c0f7cd52-kube-api-access-crvhz\") pod \"ovnkube-node-rkcgk\" (UID: \"c5d026b7-af6b-4899-85f4-6554c0f7cd52\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk"
Sep 18 17:49:57.726890 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.726581    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/548befd4-2d47-4b4d-a376-c295e0dc141b-multus-daemon-config\") pod \"multus-4wsjj\" (UID: \"548befd4-2d47-4b4d-a376-c295e0dc141b\") " pod="openshift-multus/multus-4wsjj"
Sep 18 17:49:57.726890 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.726610    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/291a480f-a747-4910-a467-676a77e04eb0-etc-sysctl-conf\") pod \"tuned-mqjzn\" (UID: \"291a480f-a747-4910-a467-676a77e04eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-mqjzn"
Sep 18 17:49:57.726890 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.726650    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c5d026b7-af6b-4899-85f4-6554c0f7cd52-etc-openvswitch\") pod \"ovnkube-node-rkcgk\" (UID: \"c5d026b7-af6b-4899-85f4-6554c0f7cd52\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk"
Sep 18 17:49:57.726890 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.726692    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c5d026b7-af6b-4899-85f4-6554c0f7cd52-ovn-node-metrics-cert\") pod \"ovnkube-node-rkcgk\" (UID: \"c5d026b7-af6b-4899-85f4-6554c0f7cd52\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk"
Sep 18 17:49:57.726890 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.726719    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/291a480f-a747-4910-a467-676a77e04eb0-tmp\") pod \"tuned-mqjzn\" (UID: \"291a480f-a747-4910-a467-676a77e04eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-mqjzn"
Sep 18 17:49:57.726890 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.726743    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/548befd4-2d47-4b4d-a376-c295e0dc141b-cnibin\") pod \"multus-4wsjj\" (UID: \"548befd4-2d47-4b4d-a376-c295e0dc141b\") " pod="openshift-multus/multus-4wsjj"
Sep 18 17:49:57.726890 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.726769    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/548befd4-2d47-4b4d-a376-c295e0dc141b-multus-socket-dir-parent\") pod \"multus-4wsjj\" (UID: \"548befd4-2d47-4b4d-a376-c295e0dc141b\") " pod="openshift-multus/multus-4wsjj"
Sep 18 17:49:57.726890 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.726793    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/548befd4-2d47-4b4d-a376-c295e0dc141b-hostroot\") pod \"multus-4wsjj\" (UID: \"548befd4-2d47-4b4d-a376-c295e0dc141b\") " pod="openshift-multus/multus-4wsjj"
Sep 18 17:49:57.726890 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.726827    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/291a480f-a747-4910-a467-676a77e04eb0-etc-sysconfig\") pod \"tuned-mqjzn\" (UID: \"291a480f-a747-4910-a467-676a77e04eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-mqjzn"
Sep 18 17:49:57.726890 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.726852    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgwwr\" (UniqueName: \"kubernetes.io/projected/291a480f-a747-4910-a467-676a77e04eb0-kube-api-access-hgwwr\") pod \"tuned-mqjzn\" (UID: \"291a480f-a747-4910-a467-676a77e04eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-mqjzn"
Sep 18 17:49:57.727251 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.726915    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1907e944-be36-42b2-bc90-90938a227371-hosts-file\") pod \"node-resolver-q2btm\" (UID: \"1907e944-be36-42b2-bc90-90938a227371\") " pod="openshift-dns/node-resolver-q2btm"
Sep 18 17:49:57.727251 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.726940    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/548befd4-2d47-4b4d-a376-c295e0dc141b-multus-cni-dir\") pod \"multus-4wsjj\" (UID: \"548befd4-2d47-4b4d-a376-c295e0dc141b\") " pod="openshift-multus/multus-4wsjj"
Sep 18 17:49:57.727251 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.726964    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/548befd4-2d47-4b4d-a376-c295e0dc141b-host-run-k8s-cni-cncf-io\") pod \"multus-4wsjj\" (UID: \"548befd4-2d47-4b4d-a376-c295e0dc141b\") " pod="openshift-multus/multus-4wsjj"
Sep 18 17:49:57.727251 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.726995    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scbjn\" (UniqueName: \"kubernetes.io/projected/234003fa-8d5d-4108-8f62-859ea05eb9aa-kube-api-access-scbjn\") pod \"network-check-target-bgwtf\" (UID: \"234003fa-8d5d-4108-8f62-859ea05eb9aa\") " pod="openshift-network-diagnostics/network-check-target-bgwtf"
Sep 18 17:49:57.727251 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.727022    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/548befd4-2d47-4b4d-a376-c295e0dc141b-system-cni-dir\") pod \"multus-4wsjj\" (UID: \"548befd4-2d47-4b4d-a376-c295e0dc141b\") " pod="openshift-multus/multus-4wsjj"
Sep 18 17:49:57.727251 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.727045    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c5d026b7-af6b-4899-85f4-6554c0f7cd52-log-socket\") pod \"ovnkube-node-rkcgk\" (UID: \"c5d026b7-af6b-4899-85f4-6554c0f7cd52\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk"
Sep 18 17:49:57.727251 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.727068    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/291a480f-a747-4910-a467-676a77e04eb0-etc-systemd\") pod \"tuned-mqjzn\" (UID: \"291a480f-a747-4910-a467-676a77e04eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-mqjzn"
Sep 18 17:49:57.727251 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.727094    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/548befd4-2d47-4b4d-a376-c295e0dc141b-host-var-lib-cni-bin\") pod \"multus-4wsjj\" (UID: \"548befd4-2d47-4b4d-a376-c295e0dc141b\") " pod="openshift-multus/multus-4wsjj"
Sep 18 17:49:57.727251 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.727131    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/548befd4-2d47-4b4d-a376-c295e0dc141b-host-var-lib-kubelet\") pod \"multus-4wsjj\" (UID: \"548befd4-2d47-4b4d-a376-c295e0dc141b\") " pod="openshift-multus/multus-4wsjj"
Sep 18 17:49:57.727251 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.727154    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/548befd4-2d47-4b4d-a376-c295e0dc141b-etc-kubernetes\") pod \"multus-4wsjj\" (UID: \"548befd4-2d47-4b4d-a376-c295e0dc141b\") " pod="openshift-multus/multus-4wsjj"
Sep 18 17:49:57.727251 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.727210    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btk9p\" (UniqueName: \"kubernetes.io/projected/45a7fd6c-fe96-4a1a-b3a5-2cb78c5435e8-kube-api-access-btk9p\") pod \"network-metrics-daemon-tmtsv\" (UID: \"45a7fd6c-fe96-4a1a-b3a5-2cb78c5435e8\") " pod="openshift-multus/network-metrics-daemon-tmtsv"
Sep 18 17:49:57.727251 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.727232    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/291a480f-a747-4910-a467-676a77e04eb0-run\") pod \"tuned-mqjzn\" (UID: \"291a480f-a747-4910-a467-676a77e04eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-mqjzn"
Sep 18 17:49:57.727685 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.727506    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-k2xs7"
Sep 18 17:49:57.730072 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.730057    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cqzh9"
Sep 18 17:49:57.732386 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.732272    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 18 17:49:57.732480 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.732441    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 18 17:49:57.732620 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.732601    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-kn2f8\""
Sep 18 17:49:57.732713 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.732697    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bjq6x\""
Sep 18 17:49:57.733275 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.733247    2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 17:49:57.733366 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.733254    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-jrzkw\""
Sep 18 17:49:57.733366 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.733301    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 18 17:49:57.733366 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.733312    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 18 17:49:57.733565 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.733393    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 18 17:49:57.733565 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.733463    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 18 17:49:57.733674 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.733660    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 18 17:49:57.761207 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.761172    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-g8j4d"
Sep 18 17:49:57.771277 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.771256    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-g8j4d"
Sep 18 17:49:57.816643 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.816612    2561 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 18 17:49:57.827686 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.827659    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/291a480f-a747-4910-a467-676a77e04eb0-etc-sysctl-d\") pod \"tuned-mqjzn\" (UID: \"291a480f-a747-4910-a467-676a77e04eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-mqjzn"
Sep 18 17:49:57.827814 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.827694    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c5d026b7-af6b-4899-85f4-6554c0f7cd52-run-openvswitch\") pod \"ovnkube-node-rkcgk\" (UID: \"c5d026b7-af6b-4899-85f4-6554c0f7cd52\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk"
Sep 18 17:49:57.827814 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.827719    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c5d026b7-af6b-4899-85f4-6554c0f7cd52-ovnkube-config\") pod \"ovnkube-node-rkcgk\" (UID: \"c5d026b7-af6b-4899-85f4-6554c0f7cd52\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk"
Sep 18 17:49:57.827814 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.827741    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/65f6cbfa-00b5-4a5c-be75-28392cf8a1a8-host-slash\") pod \"iptables-alerter-2mqnb\" (UID: \"65f6cbfa-00b5-4a5c-be75-28392cf8a1a8\") " pod="openshift-network-operator/iptables-alerter-2mqnb"
Sep 18 17:49:57.827814 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.827796    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/65f6cbfa-00b5-4a5c-be75-28392cf8a1a8-host-slash\") pod \"iptables-alerter-2mqnb\" (UID: \"65f6cbfa-00b5-4a5c-be75-28392cf8a1a8\") " pod="openshift-network-operator/iptables-alerter-2mqnb"
Sep 18 17:49:57.828030 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.827841    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c5d026b7-af6b-4899-85f4-6554c0f7cd52-run-openvswitch\") pod \"ovnkube-node-rkcgk\" (UID: \"c5d026b7-af6b-4899-85f4-6554c0f7cd52\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk"
Sep 18 17:49:57.828030 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.827855    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/291a480f-a747-4910-a467-676a77e04eb0-etc-sysctl-d\") pod \"tuned-mqjzn\" (UID: \"291a480f-a747-4910-a467-676a77e04eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-mqjzn"
Sep 18 17:49:57.828030 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.827870    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf9qt\" (UniqueName: \"kubernetes.io/projected/65f6cbfa-00b5-4a5c-be75-28392cf8a1a8-kube-api-access-lf9qt\") pod \"iptables-alerter-2mqnb\" (UID: \"65f6cbfa-00b5-4a5c-be75-28392cf8a1a8\") " pod="openshift-network-operator/iptables-alerter-2mqnb"
Sep 18 17:49:57.828030 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.827900    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13-agent-certs\") pod \"konnectivity-agent-k2xs7\" (UID: \"313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13\") " pod="kube-system/konnectivity-agent-k2xs7"
Sep 18 17:49:57.828030 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.827927    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13-konnectivity-ca\") pod \"konnectivity-agent-k2xs7\" (UID: \"313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13\") " pod="kube-system/konnectivity-agent-k2xs7"
Sep 18 17:49:57.828030 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.827953    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1dc25d25-efc8-477f-9a1e-5220f1d20345-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cqzh9\" (UID: \"1dc25d25-efc8-477f-9a1e-5220f1d20345\") " pod="openshift-multus/multus-additional-cni-plugins-cqzh9"
Sep 18 17:49:57.828030 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.827980    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/291a480f-a747-4910-a467-676a77e04eb0-var-lib-kubelet\") pod \"tuned-mqjzn\" (UID: \"291a480f-a747-4910-a467-676a77e04eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-mqjzn"
Sep 18 17:49:57.828030 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.828006    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxwb7\" (UniqueName: \"kubernetes.io/projected/1907e944-be36-42b2-bc90-90938a227371-kube-api-access-qxwb7\") pod \"node-resolver-q2btm\" (UID: \"1907e944-be36-42b2-bc90-90938a227371\") " pod="openshift-dns/node-resolver-q2btm"
Sep 18 17:49:57.828423 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.828086    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/291a480f-a747-4910-a467-676a77e04eb0-var-lib-kubelet\") pod \"tuned-mqjzn\" (UID: \"291a480f-a747-4910-a467-676a77e04eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-mqjzn"
Sep 18 17:49:57.828423 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.828154    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1dc25d25-efc8-477f-9a1e-5220f1d20345-cni-binary-copy\") pod \"multus-additional-cni-plugins-cqzh9\" (UID: \"1dc25d25-efc8-477f-9a1e-5220f1d20345\") " pod="openshift-multus/multus-additional-cni-plugins-cqzh9"
Sep 18 17:49:57.828423 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.828177    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4945651f-6aa7-4ce0-82e2-3f5f234b607f-host\") pod \"node-ca-zvmrv\" (UID: \"4945651f-6aa7-4ce0-82e2-3f5f234b607f\") " pod="openshift-image-registry/node-ca-zvmrv"
Sep 18 17:49:57.828423 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.828197    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/291a480f-a747-4910-a467-676a77e04eb0-etc-kubernetes\") pod \"tuned-mqjzn\" (UID: \"291a480f-a747-4910-a467-676a77e04eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-mqjzn"
Sep 18 17:49:57.828423 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.828225    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/291a480f-a747-4910-a467-676a77e04eb0-etc-tuned\") pod \"tuned-mqjzn\" (UID: \"291a480f-a747-4910-a467-676a77e04eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-mqjzn"
Sep 18 17:49:57.828423 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.828266    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1907e944-be36-42b2-bc90-90938a227371-tmp-dir\") pod \"node-resolver-q2btm\" (UID: \"1907e944-be36-42b2-bc90-90938a227371\") " pod="openshift-dns/node-resolver-q2btm"
Sep 18 17:49:57.828423 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.828303    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1dc25d25-efc8-477f-9a1e-5220f1d20345-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cqzh9\" (UID: \"1dc25d25-efc8-477f-9a1e-5220f1d20345\") " pod="openshift-multus/multus-additional-cni-plugins-cqzh9"
Sep 18 17:49:57.828423 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.828310    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/291a480f-a747-4910-a467-676a77e04eb0-etc-kubernetes\") pod \"tuned-mqjzn\" (UID: \"291a480f-a747-4910-a467-676a77e04eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-mqjzn"
Sep 18 17:49:57.828423 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.828276    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c5d026b7-af6b-4899-85f4-6554c0f7cd52-ovnkube-config\") pod \"ovnkube-node-rkcgk\" (UID: \"c5d026b7-af6b-4899-85f4-6554c0f7cd52\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk"
Sep 18 17:49:57.828423 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.828322    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1dc25d25-efc8-477f-9a1e-5220f1d20345-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cqzh9\" (UID: \"1dc25d25-efc8-477f-9a1e-5220f1d20345\") " pod="openshift-multus/multus-additional-cni-plugins-cqzh9"
Sep 18 17:49:57.828423 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.828409    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6fb9873f-b1a1-4d93-910c-5c9d1417c1da-device-dir\") pod \"aws-ebs-csi-driver-node-xphc9\" (UID: \"6fb9873f-b1a1-4d93-910c-5c9d1417c1da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xphc9"
Sep 18 17:49:57.828979 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.828436    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7a3ae2d8-a30b-4c83-a53c-634fb07621d6-dbus\") pod \"global-pull-secret-syncer-mvtdm\" (UID: \"7a3ae2d8-a30b-4c83-a53c-634fb07621d6\") " pod="kube-system/global-pull-secret-syncer-mvtdm"
Sep 18 17:49:57.828979 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.828463    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngwb6\" (UniqueName: \"kubernetes.io/projected/4945651f-6aa7-4ce0-82e2-3f5f234b607f-kube-api-access-ngwb6\") pod \"node-ca-zvmrv\" (UID: \"4945651f-6aa7-4ce0-82e2-3f5f234b607f\") " pod="openshift-image-registry/node-ca-zvmrv"
Sep 18 17:49:57.828979 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.828492    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c5d026b7-af6b-4899-85f4-6554c0f7cd52-run-systemd\") pod \"ovnkube-node-rkcgk\" (UID: \"c5d026b7-af6b-4899-85f4-6554c0f7cd52\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk"
Sep 18 17:49:57.828979 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.828518    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c5d026b7-af6b-4899-85f4-6554c0f7cd52-run-ovn\") pod \"ovnkube-node-rkcgk\" (UID: \"c5d026b7-af6b-4899-85f4-6554c0f7cd52\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk"
Sep 18 17:49:57.828979 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.828545    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/291a480f-a747-4910-a467-676a77e04eb0-lib-modules\") pod \"tuned-mqjzn\" (UID: \"291a480f-a747-4910-a467-676a77e04eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-mqjzn"
Sep 18 17:49:57.828979 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.828543    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1907e944-be36-42b2-bc90-90938a227371-tmp-dir\") pod \"node-resolver-q2btm\" (UID: \"1907e944-be36-42b2-bc90-90938a227371\") " pod="openshift-dns/node-resolver-q2btm"
Sep 18 17:49:57.828979 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.828569    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/291a480f-a747-4910-a467-676a77e04eb0-host\") pod \"tuned-mqjzn\" (UID: \"291a480f-a747-4910-a467-676a77e04eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-mqjzn"
Sep 18 17:49:57.828979 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.828586    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c5d026b7-af6b-4899-85f4-6554c0f7cd52-run-systemd\") pod \"ovnkube-node-rkcgk\" (UID: \"c5d026b7-af6b-4899-85f4-6554c0f7cd52\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk"
Sep 18 17:49:57.828979 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.828595    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2mc4\" (UniqueName: \"kubernetes.io/projected/1dc25d25-efc8-477f-9a1e-5220f1d20345-kube-api-access-h2mc4\") pod \"multus-additional-cni-plugins-cqzh9\" (UID: \"1dc25d25-efc8-477f-9a1e-5220f1d20345\") " pod="openshift-multus/multus-additional-cni-plugins-cqzh9"
Sep 18 17:49:57.828979 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.828600    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c5d026b7-af6b-4899-85f4-6554c0f7cd52-run-ovn\") pod \"ovnkube-node-rkcgk\" (UID: \"c5d026b7-af6b-4899-85f4-6554c0f7cd52\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk"
Sep 18 17:49:57.828979 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.828625    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c5d026b7-af6b-4899-85f4-6554c0f7cd52-ovnkube-script-lib\") pod \"ovnkube-node-rkcgk\" (UID: \"c5d026b7-af6b-4899-85f4-6554c0f7cd52\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk"
Sep 18 17:49:57.828979 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.828650    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/291a480f-a747-4910-a467-676a77e04eb0-host\") pod \"tuned-mqjzn\" (UID: \"291a480f-a747-4910-a467-676a77e04eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-mqjzn"
Sep 18 17:49:57.828979 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.828653    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5d026b7-af6b-4899-85f4-6554c0f7cd52-host-kubelet\") pod \"ovnkube-node-rkcgk\" (UID: \"c5d026b7-af6b-4899-85f4-6554c0f7cd52\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk"
Sep 18 17:49:57.828979 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.828681    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c5d026b7-af6b-4899-85f4-6554c0f7cd52-systemd-units\") pod \"ovnkube-node-rkcgk\" (UID: \"c5d026b7-af6b-4899-85f4-6554c0f7cd52\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk"
Sep 18 17:49:57.828979 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.828707    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5d026b7-af6b-4899-85f4-6554c0f7cd52-host-run-ovn-kubernetes\") pod \"ovnkube-node-rkcgk\" (UID: \"c5d026b7-af6b-4899-85f4-6554c0f7cd52\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk"
Sep 18 17:49:57.828979 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.828735    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/291a480f-a747-4910-a467-676a77e04eb0-lib-modules\") pod \"tuned-mqjzn\" (UID: \"291a480f-a747-4910-a467-676a77e04eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-mqjzn"
Sep 18 17:49:57.828979 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.828738    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5d026b7-af6b-4899-85f4-6554c0f7cd52-host-kubelet\") pod \"ovnkube-node-rkcgk\" (UID: \"c5d026b7-af6b-4899-85f4-6554c0f7cd52\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk"
Sep 18 17:49:57.828979 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.828737    2561 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 18 17:49:57.829731 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.828764    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5d026b7-af6b-4899-85f4-6554c0f7cd52-host-run-ovn-kubernetes\") pod \"ovnkube-node-rkcgk\" (UID: \"c5d026b7-af6b-4899-85f4-6554c0f7cd52\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk"
Sep 18 17:49:57.829731 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.828771    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c5d026b7-af6b-4899-85f4-6554c0f7cd52-systemd-units\") pod \"ovnkube-node-rkcgk\" (UID: \"c5d026b7-af6b-4899-85f4-6554c0f7cd52\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk"
Sep 18 17:49:57.829731 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.828804    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c5d026b7-af6b-4899-85f4-6554c0f7cd52-host-cni-bin\") pod \"ovnkube-node-rkcgk\" (UID: \"c5d026b7-af6b-4899-85f4-6554c0f7cd52\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk"
Sep 18 17:49:57.829731 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.828838    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c5d026b7-af6b-4899-85f4-6554c0f7cd52-host-slash\") pod \"ovnkube-node-rkcgk\" (UID: \"c5d026b7-af6b-4899-85f4-6554c0f7cd52\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk"
Sep 18 17:49:57.829731 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.828844    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c5d026b7-af6b-4899-85f4-6554c0f7cd52-host-cni-bin\") pod \"ovnkube-node-rkcgk\" (UID: \"c5d026b7-af6b-4899-85f4-6554c0f7cd52\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk"
Sep 18 17:49:57.829731 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.828854    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c5d026b7-af6b-4899-85f4-6554c0f7cd52-host-run-netns\") pod \"ovnkube-node-rkcgk\" (UID: \"c5d026b7-af6b-4899-85f4-6554c0f7cd52\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk"
Sep 18 17:49:57.829731 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.828881    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c5d026b7-af6b-4899-85f4-6554c0f7cd52-host-run-netns\") pod \"ovnkube-node-rkcgk\" (UID: \"c5d026b7-af6b-4899-85f4-6554c0f7cd52\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk"
Sep 18 17:49:57.829731 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.828892    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/548befd4-2d47-4b4d-a376-c295e0dc141b-host-var-lib-cni-multus\") pod \"multus-4wsjj\" (UID: \"548befd4-2d47-4b4d-a376-c295e0dc141b\") " pod="openshift-multus/multus-4wsjj"
Sep 18 17:49:57.829731 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.828901    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c5d026b7-af6b-4899-85f4-6554c0f7cd52-host-slash\") pod \"ovnkube-node-rkcgk\" (UID: \"c5d026b7-af6b-4899-85f4-6554c0f7cd52\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk"
Sep 18 17:49:57.829731 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.828920    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/291a480f-a747-4910-a467-676a77e04eb0-sys\") pod \"tuned-mqjzn\" (UID: \"291a480f-a747-4910-a467-676a77e04eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-mqjzn"
Sep 18 17:49:57.829731 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.828943    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/548befd4-2d47-4b4d-a376-c295e0dc141b-host-var-lib-cni-multus\") pod \"multus-4wsjj\" (UID: \"548befd4-2d47-4b4d-a376-c295e0dc141b\") " pod="openshift-multus/multus-4wsjj"
Sep 18 17:49:57.829731 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.828947    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/65f6cbfa-00b5-4a5c-be75-28392cf8a1a8-iptables-alerter-script\") pod \"iptables-alerter-2mqnb\" (UID: \"65f6cbfa-00b5-4a5c-be75-28392cf8a1a8\") " pod="openshift-network-operator/iptables-alerter-2mqnb"
Sep 18 17:49:57.829731 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.828991    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c5d026b7-af6b-4899-85f4-6554c0f7cd52-host-cni-netd\") pod \"ovnkube-node-rkcgk\" (UID: \"c5d026b7-af6b-4899-85f4-6554c0f7cd52\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk"
Sep 18 17:49:57.829731 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.828993    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/291a480f-a747-4910-a467-676a77e04eb0-sys\") pod \"tuned-mqjzn\" (UID: \"291a480f-a747-4910-a467-676a77e04eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-mqjzn"
Sep 18 17:49:57.829731 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.829027    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c5d026b7-af6b-4899-85f4-6554c0f7cd52-host-cni-netd\") pod \"ovnkube-node-rkcgk\" (UID: \"c5d026b7-af6b-4899-85f4-6554c0f7cd52\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk"
Sep 18 17:49:57.829731 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.829022    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5d026b7-af6b-4899-85f4-6554c0f7cd52-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rkcgk\" (UID: \"c5d026b7-af6b-4899-85f4-6554c0f7cd52\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk"
Sep 18 17:49:57.829731 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.829057    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5d026b7-af6b-4899-85f4-6554c0f7cd52-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rkcgk\" (UID: \"c5d026b7-af6b-4899-85f4-6554c0f7cd52\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk"
Sep 18 17:49:57.830760 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.829087    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c5d026b7-af6b-4899-85f4-6554c0f7cd52-env-overrides\") pod \"ovnkube-node-rkcgk\" (UID: \"c5d026b7-af6b-4899-85f4-6554c0f7cd52\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk"
Sep 18 17:49:57.830760 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.829132    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c5d026b7-af6b-4899-85f4-6554c0f7cd52-ovnkube-script-lib\") pod \"ovnkube-node-rkcgk\" (UID: \"c5d026b7-af6b-4899-85f4-6554c0f7cd52\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk"
Sep 18 17:49:57.830760 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.829117    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45a7fd6c-fe96-4a1a-b3a5-2cb78c5435e8-metrics-certs\") pod \"network-metrics-daemon-tmtsv\" (UID: \"45a7fd6c-fe96-4a1a-b3a5-2cb78c5435e8\") " pod="openshift-multus/network-metrics-daemon-tmtsv"
Sep 18 17:49:57.830760 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:49:57.829221    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 17:49:57.830760 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.829220    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6fb9873f-b1a1-4d93-910c-5c9d1417c1da-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xphc9\" (UID: \"6fb9873f-b1a1-4d93-910c-5c9d1417c1da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xphc9"
Sep 18 17:49:57.830760 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:49:57.829303    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45a7fd6c-fe96-4a1a-b3a5-2cb78c5435e8-metrics-certs podName:45a7fd6c-fe96-4a1a-b3a5-2cb78c5435e8 nodeName:}" failed. No retries permitted until 2026-09-18 17:49:58.329275033 +0000 UTC m=+2.956406651 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/45a7fd6c-fe96-4a1a-b3a5-2cb78c5435e8-metrics-certs") pod "network-metrics-daemon-tmtsv" (UID: "45a7fd6c-fe96-4a1a-b3a5-2cb78c5435e8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 17:49:57.830760 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.829321    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c5d026b7-af6b-4899-85f4-6554c0f7cd52-var-lib-openvswitch\") pod \"ovnkube-node-rkcgk\" (UID: \"c5d026b7-af6b-4899-85f4-6554c0f7cd52\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk"
Sep 18 17:49:57.830760 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.829344    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c5d026b7-af6b-4899-85f4-6554c0f7cd52-node-log\") pod \"ovnkube-node-rkcgk\" (UID: \"c5d026b7-af6b-4899-85f4-6554c0f7cd52\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk"
Sep 18 17:49:57.830760 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.829367    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/291a480f-a747-4910-a467-676a77e04eb0-etc-modprobe-d\") pod \"tuned-mqjzn\" (UID: \"291a480f-a747-4910-a467-676a77e04eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-mqjzn"
Sep 18 17:49:57.830760 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.829389    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1dc25d25-efc8-477f-9a1e-5220f1d20345-cnibin\") pod \"multus-additional-cni-plugins-cqzh9\" (UID: \"1dc25d25-efc8-477f-9a1e-5220f1d20345\") " pod="openshift-multus/multus-additional-cni-plugins-cqzh9"
Sep 18 17:49:57.830760 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.829402    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c5d026b7-af6b-4899-85f4-6554c0f7cd52-var-lib-openvswitch\") pod \"ovnkube-node-rkcgk\" (UID: \"c5d026b7-af6b-4899-85f4-6554c0f7cd52\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk"
Sep 18 17:49:57.830760 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.829410    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crvhz\" (UniqueName: \"kubernetes.io/projected/c5d026b7-af6b-4899-85f4-6554c0f7cd52-kube-api-access-crvhz\") pod \"ovnkube-node-rkcgk\" (UID: \"c5d026b7-af6b-4899-85f4-6554c0f7cd52\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk"
Sep 18 17:49:57.830760 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.829432    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/548befd4-2d47-4b4d-a376-c295e0dc141b-multus-daemon-config\") pod \"multus-4wsjj\" (UID: \"548befd4-2d47-4b4d-a376-c295e0dc141b\") " pod="openshift-multus/multus-4wsjj"
Sep 18 17:49:57.830760 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.829444    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c5d026b7-af6b-4899-85f4-6554c0f7cd52-node-log\") pod \"ovnkube-node-rkcgk\" (UID: \"c5d026b7-af6b-4899-85f4-6554c0f7cd52\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk"
Sep 18 17:49:57.830760 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.829446    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/65f6cbfa-00b5-4a5c-be75-28392cf8a1a8-iptables-alerter-script\") pod \"iptables-alerter-2mqnb\" (UID: \"65f6cbfa-00b5-4a5c-be75-28392cf8a1a8\") " pod="openshift-network-operator/iptables-alerter-2mqnb"
Sep 18 17:49:57.830760 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.829452    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/291a480f-a747-4910-a467-676a77e04eb0-etc-sysctl-conf\") pod \"tuned-mqjzn\" (UID: \"291a480f-a747-4910-a467-676a77e04eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-mqjzn"
Sep 18 17:49:57.830760 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.829497    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/291a480f-a747-4910-a467-676a77e04eb0-etc-modprobe-d\") pod \"tuned-mqjzn\" (UID: \"291a480f-a747-4910-a467-676a77e04eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-mqjzn"
Sep 18 17:49:57.831367 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.829522    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c5d026b7-af6b-4899-85f4-6554c0f7cd52-env-overrides\") pod \"ovnkube-node-rkcgk\" (UID: \"c5d026b7-af6b-4899-85f4-6554c0f7cd52\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk"
Sep 18 17:49:57.831367 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.829524    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6fb9873f-b1a1-4d93-910c-5c9d1417c1da-registration-dir\") pod \"aws-ebs-csi-driver-node-xphc9\" (UID: \"6fb9873f-b1a1-4d93-910c-5c9d1417c1da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xphc9"
Sep 18 17:49:57.831367 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.829554    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c5d026b7-af6b-4899-85f4-6554c0f7cd52-etc-openvswitch\") pod \"ovnkube-node-rkcgk\" (UID: \"c5d026b7-af6b-4899-85f4-6554c0f7cd52\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk"
Sep 18 17:49:57.831367 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.829594    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c5d026b7-af6b-4899-85f4-6554c0f7cd52-ovn-node-metrics-cert\") pod \"ovnkube-node-rkcgk\" (UID: \"c5d026b7-af6b-4899-85f4-6554c0f7cd52\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk"
Sep 18 17:49:57.831367 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.829619    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/291a480f-a747-4910-a467-676a77e04eb0-tmp\") pod \"tuned-mqjzn\" (UID: \"291a480f-a747-4910-a467-676a77e04eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-mqjzn"
Sep 18 17:49:57.831367 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.829646    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1dc25d25-efc8-477f-9a1e-5220f1d20345-system-cni-dir\") pod \"multus-additional-cni-plugins-cqzh9\" (UID: \"1dc25d25-efc8-477f-9a1e-5220f1d20345\") " pod="openshift-multus/multus-additional-cni-plugins-cqzh9"
Sep 18 17:49:57.831367 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.829671    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6fb9873f-b1a1-4d93-910c-5c9d1417c1da-etc-selinux\") pod \"aws-ebs-csi-driver-node-xphc9\" (UID: \"6fb9873f-b1a1-4d93-910c-5c9d1417c1da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xphc9"
Sep 18 17:49:57.831367 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.829620    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c5d026b7-af6b-4899-85f4-6554c0f7cd52-etc-openvswitch\") pod \"ovnkube-node-rkcgk\" (UID: \"c5d026b7-af6b-4899-85f4-6554c0f7cd52\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk"
Sep 18 17:49:57.831367 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.829705    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/548befd4-2d47-4b4d-a376-c295e0dc141b-cnibin\") pod \"multus-4wsjj\" (UID: \"548befd4-2d47-4b4d-a376-c295e0dc141b\") " pod="openshift-multus/multus-4wsjj"
Sep 18 17:49:57.831367 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.829704    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/291a480f-a747-4910-a467-676a77e04eb0-etc-sysctl-conf\") pod \"tuned-mqjzn\" (UID: \"291a480f-a747-4910-a467-676a77e04eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-mqjzn"
Sep 18 17:49:57.831367 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.829743    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/548befd4-2d47-4b4d-a376-c295e0dc141b-multus-socket-dir-parent\") pod \"multus-4wsjj\" (UID: \"548befd4-2d47-4b4d-a376-c295e0dc141b\") " pod="openshift-multus/multus-4wsjj"
Sep 18 17:49:57.831367 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.829772    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/548befd4-2d47-4b4d-a376-c295e0dc141b-hostroot\") pod \"multus-4wsjj\" (UID: \"548befd4-2d47-4b4d-a376-c295e0dc141b\") " pod="openshift-multus/multus-4wsjj"
Sep 18 17:49:57.831367 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.829791    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/548befd4-2d47-4b4d-a376-c295e0dc141b-cnibin\") pod \"multus-4wsjj\" (UID: \"548befd4-2d47-4b4d-a376-c295e0dc141b\") " pod="openshift-multus/multus-4wsjj"
Sep 18 17:49:57.831367 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.829796    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/291a480f-a747-4910-a467-676a77e04eb0-etc-sysconfig\") pod \"tuned-mqjzn\" (UID: \"291a480f-a747-4910-a467-676a77e04eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-mqjzn"
Sep 18 17:49:57.831367 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.829834    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/548befd4-2d47-4b4d-a376-c295e0dc141b-hostroot\") pod \"multus-4wsjj\" (UID: \"548befd4-2d47-4b4d-a376-c295e0dc141b\") " pod="openshift-multus/multus-4wsjj"
Sep 18 17:49:57.831367 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.829851    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/291a480f-a747-4910-a467-676a77e04eb0-etc-sysconfig\") pod \"tuned-mqjzn\" (UID: \"291a480f-a747-4910-a467-676a77e04eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-mqjzn"
Sep 18 17:49:57.831367 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.829865    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgwwr\" (UniqueName: \"kubernetes.io/projected/291a480f-a747-4910-a467-676a77e04eb0-kube-api-access-hgwwr\") pod \"tuned-mqjzn\" (UID: \"291a480f-a747-4910-a467-676a77e04eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-mqjzn"
Sep 18 17:49:57.831367 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.829893    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1907e944-be36-42b2-bc90-90938a227371-hosts-file\") pod \"node-resolver-q2btm\" (UID: \"1907e944-be36-42b2-bc90-90938a227371\") " pod="openshift-dns/node-resolver-q2btm"
Sep 18 17:49:57.831989 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.829893    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/548befd4-2d47-4b4d-a376-c295e0dc141b-multus-socket-dir-parent\") pod \"multus-4wsjj\" (UID: \"548befd4-2d47-4b4d-a376-c295e0dc141b\") " pod="openshift-multus/multus-4wsjj"
Sep 18 17:49:57.831989 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.829917    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/548befd4-2d47-4b4d-a376-c295e0dc141b-multus-cni-dir\") pod \"multus-4wsjj\" (UID: \"548befd4-2d47-4b4d-a376-c295e0dc141b\") " pod="openshift-multus/multus-4wsjj"
Sep 18 17:49:57.831989 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.829954    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/548befd4-2d47-4b4d-a376-c295e0dc141b-host-run-k8s-cni-cncf-io\") pod \"multus-4wsjj\" (UID: \"548befd4-2d47-4b4d-a376-c295e0dc141b\") " pod="openshift-multus/multus-4wsjj"
Sep 18 17:49:57.831989 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.829966    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/548befd4-2d47-4b4d-a376-c295e0dc141b-multus-cni-dir\") pod \"multus-4wsjj\" (UID: \"548befd4-2d47-4b4d-a376-c295e0dc141b\") " pod="openshift-multus/multus-4wsjj"
Sep 18 17:49:57.831989 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.829973    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1907e944-be36-42b2-bc90-90938a227371-hosts-file\") pod \"node-resolver-q2btm\" (UID: \"1907e944-be36-42b2-bc90-90938a227371\") " pod="openshift-dns/node-resolver-q2btm"
Sep 18 17:49:57.831989 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.829984    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scbjn\" (UniqueName: \"kubernetes.io/projected/234003fa-8d5d-4108-8f62-859ea05eb9aa-kube-api-access-scbjn\") pod \"network-check-target-bgwtf\" (UID: \"234003fa-8d5d-4108-8f62-859ea05eb9aa\") " pod="openshift-network-diagnostics/network-check-target-bgwtf"
Sep 18 17:49:57.831989 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.830012    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/548befd4-2d47-4b4d-a376-c295e0dc141b-system-cni-dir\") pod \"multus-4wsjj\" (UID: \"548befd4-2d47-4b4d-a376-c295e0dc141b\") " pod="openshift-multus/multus-4wsjj"
Sep 18 17:49:57.831989 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.830031    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/548befd4-2d47-4b4d-a376-c295e0dc141b-host-run-k8s-cni-cncf-io\") pod \"multus-4wsjj\" (UID: \"548befd4-2d47-4b4d-a376-c295e0dc141b\") " pod="openshift-multus/multus-4wsjj"
Sep 18 17:49:57.831989 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.830040    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c5d026b7-af6b-4899-85f4-6554c0f7cd52-log-socket\") pod \"ovnkube-node-rkcgk\" (UID: \"c5d026b7-af6b-4899-85f4-6554c0f7cd52\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk"
Sep 18 17:49:57.831989 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.830067    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/291a480f-a747-4910-a467-676a77e04eb0-etc-systemd\") pod \"tuned-mqjzn\" (UID: \"291a480f-a747-4910-a467-676a77e04eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-mqjzn"
Sep 18 17:49:57.831989 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.830089    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/548befd4-2d47-4b4d-a376-c295e0dc141b-system-cni-dir\") pod \"multus-4wsjj\" (UID: \"548befd4-2d47-4b4d-a376-c295e0dc141b\") " pod="openshift-multus/multus-4wsjj"
Sep 18 17:49:57.831989 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.830114    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c5d026b7-af6b-4899-85f4-6554c0f7cd52-log-socket\") pod \"ovnkube-node-rkcgk\" (UID: \"c5d026b7-af6b-4899-85f4-6554c0f7cd52\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk"
Sep 18 17:49:57.831989 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.830122    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/291a480f-a747-4910-a467-676a77e04eb0-etc-systemd\") pod \"tuned-mqjzn\" (UID: \"291a480f-a747-4910-a467-676a77e04eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-mqjzn"
Sep 18 17:49:57.831989 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.830147    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4945651f-6aa7-4ce0-82e2-3f5f234b607f-serviceca\") pod \"node-ca-zvmrv\" (UID: \"4945651f-6aa7-4ce0-82e2-3f5f234b607f\") " pod="openshift-image-registry/node-ca-zvmrv"
Sep 18 17:49:57.831989 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.830224    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/548befd4-2d47-4b4d-a376-c295e0dc141b-host-var-lib-cni-bin\") pod \"multus-4wsjj\" (UID: \"548befd4-2d47-4b4d-a376-c295e0dc141b\") " pod="openshift-multus/multus-4wsjj"
Sep 18 17:49:57.831989 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.830260    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/548befd4-2d47-4b4d-a376-c295e0dc141b-host-var-lib-kubelet\") pod \"multus-4wsjj\" (UID: \"548befd4-2d47-4b4d-a376-c295e0dc141b\") " pod="openshift-multus/multus-4wsjj"
Sep 18 17:49:57.831989 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.830296    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/548befd4-2d47-4b4d-a376-c295e0dc141b-etc-kubernetes\") pod \"multus-4wsjj\" (UID: \"548befd4-2d47-4b4d-a376-c295e0dc141b\") " pod="openshift-multus/multus-4wsjj"
Sep 18 17:49:57.831989 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.830313    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/548befd4-2d47-4b4d-a376-c295e0dc141b-host-var-lib-cni-bin\") pod \"multus-4wsjj\" (UID: \"548befd4-2d47-4b4d-a376-c295e0dc141b\") " pod="openshift-multus/multus-4wsjj"
Sep 18 17:49:57.832532 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.830329    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btk9p\" (UniqueName: \"kubernetes.io/projected/45a7fd6c-fe96-4a1a-b3a5-2cb78c5435e8-kube-api-access-btk9p\") pod \"network-metrics-daemon-tmtsv\" (UID: \"45a7fd6c-fe96-4a1a-b3a5-2cb78c5435e8\") " pod="openshift-multus/network-metrics-daemon-tmtsv"
Sep 18 17:49:57.832532 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.830352    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6fb9873f-b1a1-4d93-910c-5c9d1417c1da-socket-dir\") pod \"aws-ebs-csi-driver-node-xphc9\" (UID: \"6fb9873f-b1a1-4d93-910c-5c9d1417c1da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xphc9"
Sep 18 17:49:57.832532 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.830362    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/548befd4-2d47-4b4d-a376-c295e0dc141b-host-var-lib-kubelet\") pod \"multus-4wsjj\" (UID: \"548befd4-2d47-4b4d-a376-c295e0dc141b\") " pod="openshift-multus/multus-4wsjj"
Sep 18 17:49:57.832532 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.830372    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6fb9873f-b1a1-4d93-910c-5c9d1417c1da-sys-fs\") pod \"aws-ebs-csi-driver-node-xphc9\" (UID: \"6fb9873f-b1a1-4d93-910c-5c9d1417c1da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xphc9"
Sep 18 17:49:57.832532 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.830395    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srrwg\" (UniqueName: \"kubernetes.io/projected/6fb9873f-b1a1-4d93-910c-5c9d1417c1da-kube-api-access-srrwg\") pod \"aws-ebs-csi-driver-node-xphc9\" (UID: \"6fb9873f-b1a1-4d93-910c-5c9d1417c1da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xphc9"
Sep 18 17:49:57.832532 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.830412    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/548befd4-2d47-4b4d-a376-c295e0dc141b-etc-kubernetes\") pod \"multus-4wsjj\" (UID: \"548befd4-2d47-4b4d-a376-c295e0dc141b\") " pod="openshift-multus/multus-4wsjj"
Sep 18 17:49:57.832532 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.830419    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7a3ae2d8-a30b-4c83-a53c-634fb07621d6-kubelet-config\") pod \"global-pull-secret-syncer-mvtdm\" (UID: \"7a3ae2d8-a30b-4c83-a53c-634fb07621d6\") " pod="kube-system/global-pull-secret-syncer-mvtdm"
Sep 18 17:49:57.832532 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.830459    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/291a480f-a747-4910-a467-676a77e04eb0-run\") pod \"tuned-mqjzn\" (UID: \"291a480f-a747-4910-a467-676a77e04eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-mqjzn"
Sep 18 17:49:57.832532 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.830488    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7a3ae2d8-a30b-4c83-a53c-634fb07621d6-original-pull-secret\") pod \"global-pull-secret-syncer-mvtdm\" (UID: \"7a3ae2d8-a30b-4c83-a53c-634fb07621d6\") " pod="kube-system/global-pull-secret-syncer-mvtdm"
Sep 18 17:49:57.832532 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.830501    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/548befd4-2d47-4b4d-a376-c295e0dc141b-multus-daemon-config\") pod \"multus-4wsjj\" (UID: \"548befd4-2d47-4b4d-a376-c295e0dc141b\") " pod="openshift-multus/multus-4wsjj"
Sep 18 17:49:57.832532 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.830520    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/548befd4-2d47-4b4d-a376-c295e0dc141b-cni-binary-copy\") pod \"multus-4wsjj\" (UID: \"548befd4-2d47-4b4d-a376-c295e0dc141b\") " pod="openshift-multus/multus-4wsjj"
Sep 18 17:49:57.832532 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.830546    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/548befd4-2d47-4b4d-a376-c295e0dc141b-host-run-netns\") pod \"multus-4wsjj\" (UID: \"548befd4-2d47-4b4d-a376-c295e0dc141b\") " pod="openshift-multus/multus-4wsjj"
Sep 18 17:49:57.832532 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.830559    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/291a480f-a747-4910-a467-676a77e04eb0-run\") pod \"tuned-mqjzn\" (UID: \"291a480f-a747-4910-a467-676a77e04eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-mqjzn"
Sep 18 17:49:57.832532 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.830575    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/548befd4-2d47-4b4d-a376-c295e0dc141b-multus-conf-dir\") pod \"multus-4wsjj\" (UID: \"548befd4-2d47-4b4d-a376-c295e0dc141b\") " pod="openshift-multus/multus-4wsjj"
Sep 18 17:49:57.832532 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.830602    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/548befd4-2d47-4b4d-a376-c295e0dc141b-host-run-multus-certs\") pod \"multus-4wsjj\" (UID: \"548befd4-2d47-4b4d-a376-c295e0dc141b\") " pod="openshift-multus/multus-4wsjj"
Sep 18 17:49:57.832532 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.830627    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhc4v\" (UniqueName: \"kubernetes.io/projected/548befd4-2d47-4b4d-a376-c295e0dc141b-kube-api-access-nhc4v\") pod \"multus-4wsjj\" (UID: \"548befd4-2d47-4b4d-a376-c295e0dc141b\") " pod="openshift-multus/multus-4wsjj"
Sep 18 17:49:57.832532 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.830657    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/548befd4-2d47-4b4d-a376-c295e0dc141b-host-run-netns\") pod \"multus-4wsjj\" (UID: \"548befd4-2d47-4b4d-a376-c295e0dc141b\") " pod="openshift-multus/multus-4wsjj"
Sep 18 17:49:57.833236 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.830715    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/548befd4-2d47-4b4d-a376-c295e0dc141b-host-run-multus-certs\") pod \"multus-4wsjj\" (UID: \"548befd4-2d47-4b4d-a376-c295e0dc141b\") " pod="openshift-multus/multus-4wsjj"
Sep 18 17:49:57.833236 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.830709    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/548befd4-2d47-4b4d-a376-c295e0dc141b-multus-conf-dir\") pod \"multus-4wsjj\" (UID: \"548befd4-2d47-4b4d-a376-c295e0dc141b\") " pod="openshift-multus/multus-4wsjj"
Sep 18 17:49:57.833236 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.830951    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/548befd4-2d47-4b4d-a376-c295e0dc141b-cni-binary-copy\") pod \"multus-4wsjj\" (UID: \"548befd4-2d47-4b4d-a376-c295e0dc141b\") " pod="openshift-multus/multus-4wsjj"
Sep 18 17:49:57.833236 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.832934    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c5d026b7-af6b-4899-85f4-6554c0f7cd52-ovn-node-metrics-cert\") pod \"ovnkube-node-rkcgk\" (UID: \"c5d026b7-af6b-4899-85f4-6554c0f7cd52\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk"
Sep 18 17:49:57.833236 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.833028    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/291a480f-a747-4910-a467-676a77e04eb0-etc-tuned\") pod \"tuned-mqjzn\" (UID: \"291a480f-a747-4910-a467-676a77e04eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-mqjzn"
Sep 18 17:49:57.833236 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.833062    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/291a480f-a747-4910-a467-676a77e04eb0-tmp\") pod \"tuned-mqjzn\" (UID: \"291a480f-a747-4910-a467-676a77e04eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-mqjzn"
Sep 18 17:49:57.836269 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.836250    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxwb7\" (UniqueName: \"kubernetes.io/projected/1907e944-be36-42b2-bc90-90938a227371-kube-api-access-qxwb7\") pod \"node-resolver-q2btm\" (UID: \"1907e944-be36-42b2-bc90-90938a227371\") " pod="openshift-dns/node-resolver-q2btm"
Sep 18 17:49:57.836370 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.836290    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf9qt\" (UniqueName: \"kubernetes.io/projected/65f6cbfa-00b5-4a5c-be75-28392cf8a1a8-kube-api-access-lf9qt\") pod \"iptables-alerter-2mqnb\" (UID: \"65f6cbfa-00b5-4a5c-be75-28392cf8a1a8\") " pod="openshift-network-operator/iptables-alerter-2mqnb"
Sep 18 17:49:57.838060 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.838039    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crvhz\" (UniqueName: \"kubernetes.io/projected/c5d026b7-af6b-4899-85f4-6554c0f7cd52-kube-api-access-crvhz\") pod \"ovnkube-node-rkcgk\" (UID: \"c5d026b7-af6b-4899-85f4-6554c0f7cd52\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk"
Sep 18 17:49:57.838329 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:49:57.838313    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 17:49:57.838329 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:49:57.838331    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 17:49:57.838468 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:49:57.838341    2561 projected.go:196] Error preparing data for projected volume kube-api-access-scbjn for pod openshift-network-diagnostics/network-check-target-bgwtf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 17:49:57.838468 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:49:57.838401    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/234003fa-8d5d-4108-8f62-859ea05eb9aa-kube-api-access-scbjn podName:234003fa-8d5d-4108-8f62-859ea05eb9aa nodeName:}" failed. No retries permitted until 2026-09-18 17:49:58.338373972 +0000 UTC m=+2.965505582 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-scbjn" (UniqueName: "kubernetes.io/projected/234003fa-8d5d-4108-8f62-859ea05eb9aa-kube-api-access-scbjn") pod "network-check-target-bgwtf" (UID: "234003fa-8d5d-4108-8f62-859ea05eb9aa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 17:49:57.839463 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.839443    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btk9p\" (UniqueName: \"kubernetes.io/projected/45a7fd6c-fe96-4a1a-b3a5-2cb78c5435e8-kube-api-access-btk9p\") pod \"network-metrics-daemon-tmtsv\" (UID: \"45a7fd6c-fe96-4a1a-b3a5-2cb78c5435e8\") " pod="openshift-multus/network-metrics-daemon-tmtsv"
Sep 18 17:49:57.839547 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.839489    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhc4v\" (UniqueName: \"kubernetes.io/projected/548befd4-2d47-4b4d-a376-c295e0dc141b-kube-api-access-nhc4v\") pod \"multus-4wsjj\" (UID: \"548befd4-2d47-4b4d-a376-c295e0dc141b\") " pod="openshift-multus/multus-4wsjj"
Sep 18 17:49:57.840814 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.840799    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgwwr\" (UniqueName: \"kubernetes.io/projected/291a480f-a747-4910-a467-676a77e04eb0-kube-api-access-hgwwr\") pod \"tuned-mqjzn\" (UID: \"291a480f-a747-4910-a467-676a77e04eb0\") " pod="openshift-cluster-node-tuning-operator/tuned-mqjzn"
Sep 18 17:49:57.859461 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.859415    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-54.ec2.internal" event={"ID":"0b9f4e0872eb36d1a9eb7f46a41c431b","Type":"ContainerStarted","Data":"6d4a226cb1a63a9d983d953dd70eca5a926f019a9a5953eabf319fb4256fc68d"}
Sep 18 17:49:57.860629 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.860609    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-54.ec2.internal" event={"ID":"bfcddef9dbb20737aafa5928064e8e0d","Type":"ContainerStarted","Data":"d6b06312a4ed0014667f4f74cdd5754f2a1a7714112cc672729eaeda8ca76448"}
Sep 18 17:49:57.931505 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.931426    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1dc25d25-efc8-477f-9a1e-5220f1d20345-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cqzh9\" (UID: \"1dc25d25-efc8-477f-9a1e-5220f1d20345\") " pod="openshift-multus/multus-additional-cni-plugins-cqzh9"
Sep 18 17:49:57.931505 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.931472    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1dc25d25-efc8-477f-9a1e-5220f1d20345-cni-binary-copy\") pod \"multus-additional-cni-plugins-cqzh9\" (UID: \"1dc25d25-efc8-477f-9a1e-5220f1d20345\") " pod="openshift-multus/multus-additional-cni-plugins-cqzh9"
Sep 18 17:49:57.931505 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.931495    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4945651f-6aa7-4ce0-82e2-3f5f234b607f-host\") pod \"node-ca-zvmrv\" (UID: \"4945651f-6aa7-4ce0-82e2-3f5f234b607f\") " pod="openshift-image-registry/node-ca-zvmrv"
Sep 18 17:49:57.931749 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.931532    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1dc25d25-efc8-477f-9a1e-5220f1d20345-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cqzh9\" (UID: \"1dc25d25-efc8-477f-9a1e-5220f1d20345\") " pod="openshift-multus/multus-additional-cni-plugins-cqzh9"
Sep 18 17:49:57.931749 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.931559    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1dc25d25-efc8-477f-9a1e-5220f1d20345-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cqzh9\" (UID: \"1dc25d25-efc8-477f-9a1e-5220f1d20345\") " pod="openshift-multus/multus-additional-cni-plugins-cqzh9"
Sep 18 17:49:57.931749 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.931584    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6fb9873f-b1a1-4d93-910c-5c9d1417c1da-device-dir\") pod \"aws-ebs-csi-driver-node-xphc9\" (UID: \"6fb9873f-b1a1-4d93-910c-5c9d1417c1da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xphc9"
Sep 18 17:49:57.931749 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.931585    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4945651f-6aa7-4ce0-82e2-3f5f234b607f-host\") pod \"node-ca-zvmrv\" (UID: \"4945651f-6aa7-4ce0-82e2-3f5f234b607f\") " pod="openshift-image-registry/node-ca-zvmrv"
Sep 18 17:49:57.931749 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.931631    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7a3ae2d8-a30b-4c83-a53c-634fb07621d6-dbus\") pod \"global-pull-secret-syncer-mvtdm\" (UID: \"7a3ae2d8-a30b-4c83-a53c-634fb07621d6\") " pod="kube-system/global-pull-secret-syncer-mvtdm"
Sep 18 17:49:57.931749 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.931654    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1dc25d25-efc8-477f-9a1e-5220f1d20345-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cqzh9\" (UID: \"1dc25d25-efc8-477f-9a1e-5220f1d20345\") " pod="openshift-multus/multus-additional-cni-plugins-cqzh9"
Sep 18 17:49:57.931749 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.931711    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6fb9873f-b1a1-4d93-910c-5c9d1417c1da-device-dir\") pod \"aws-ebs-csi-driver-node-xphc9\" (UID: \"6fb9873f-b1a1-4d93-910c-5c9d1417c1da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xphc9"
Sep 18 17:49:57.932099 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.931774    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngwb6\" (UniqueName: \"kubernetes.io/projected/4945651f-6aa7-4ce0-82e2-3f5f234b607f-kube-api-access-ngwb6\") pod \"node-ca-zvmrv\" (UID: \"4945651f-6aa7-4ce0-82e2-3f5f234b607f\") " pod="openshift-image-registry/node-ca-zvmrv"
Sep 18 17:49:57.932099 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.931788    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7a3ae2d8-a30b-4c83-a53c-634fb07621d6-dbus\") pod \"global-pull-secret-syncer-mvtdm\" (UID: \"7a3ae2d8-a30b-4c83-a53c-634fb07621d6\") " pod="kube-system/global-pull-secret-syncer-mvtdm"
Sep 18 17:49:57.932099 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.931812    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2mc4\" (UniqueName: \"kubernetes.io/projected/1dc25d25-efc8-477f-9a1e-5220f1d20345-kube-api-access-h2mc4\") pod \"multus-additional-cni-plugins-cqzh9\" (UID: \"1dc25d25-efc8-477f-9a1e-5220f1d20345\") " pod="openshift-multus/multus-additional-cni-plugins-cqzh9"
Sep 18 17:49:57.932099 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.931882    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6fb9873f-b1a1-4d93-910c-5c9d1417c1da-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xphc9\" (UID: \"6fb9873f-b1a1-4d93-910c-5c9d1417c1da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xphc9"
Sep 18 17:49:57.932099 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.931907    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1dc25d25-efc8-477f-9a1e-5220f1d20345-cnibin\") pod \"multus-additional-cni-plugins-cqzh9\" (UID: \"1dc25d25-efc8-477f-9a1e-5220f1d20345\") " pod="openshift-multus/multus-additional-cni-plugins-cqzh9"
Sep 18 17:49:57.932099 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.931966    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1dc25d25-efc8-477f-9a1e-5220f1d20345-cnibin\") pod \"multus-additional-cni-plugins-cqzh9\" (UID: \"1dc25d25-efc8-477f-9a1e-5220f1d20345\") " pod="openshift-multus/multus-additional-cni-plugins-cqzh9"
Sep 18 17:49:57.932099 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.932023    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6fb9873f-b1a1-4d93-910c-5c9d1417c1da-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xphc9\" (UID: \"6fb9873f-b1a1-4d93-910c-5c9d1417c1da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xphc9"
Sep 18 17:49:57.932728 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.932099    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1dc25d25-efc8-477f-9a1e-5220f1d20345-cni-binary-copy\") pod \"multus-additional-cni-plugins-cqzh9\" (UID: \"1dc25d25-efc8-477f-9a1e-5220f1d20345\") " pod="openshift-multus/multus-additional-cni-plugins-cqzh9"
Sep 18 17:49:57.932728 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.932142    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6fb9873f-b1a1-4d93-910c-5c9d1417c1da-registration-dir\") pod \"aws-ebs-csi-driver-node-xphc9\" (UID: \"6fb9873f-b1a1-4d93-910c-5c9d1417c1da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xphc9"
Sep 18 17:49:57.932728 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.932185    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1dc25d25-efc8-477f-9a1e-5220f1d20345-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cqzh9\" (UID: \"1dc25d25-efc8-477f-9a1e-5220f1d20345\") " pod="openshift-multus/multus-additional-cni-plugins-cqzh9"
Sep 18 17:49:57.932728 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.932184    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1dc25d25-efc8-477f-9a1e-5220f1d20345-system-cni-dir\") pod \"multus-additional-cni-plugins-cqzh9\" (UID: \"1dc25d25-efc8-477f-9a1e-5220f1d20345\") " pod="openshift-multus/multus-additional-cni-plugins-cqzh9"
Sep 18 17:49:57.932728 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.932242    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1dc25d25-efc8-477f-9a1e-5220f1d20345-system-cni-dir\") pod \"multus-additional-cni-plugins-cqzh9\" (UID: \"1dc25d25-efc8-477f-9a1e-5220f1d20345\") " pod="openshift-multus/multus-additional-cni-plugins-cqzh9"
Sep 18 17:49:57.932728 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.932242    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6fb9873f-b1a1-4d93-910c-5c9d1417c1da-registration-dir\") pod \"aws-ebs-csi-driver-node-xphc9\" (UID: \"6fb9873f-b1a1-4d93-910c-5c9d1417c1da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xphc9"
Sep 18 17:49:57.932728 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.932245    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6fb9873f-b1a1-4d93-910c-5c9d1417c1da-etc-selinux\") pod \"aws-ebs-csi-driver-node-xphc9\" (UID: \"6fb9873f-b1a1-4d93-910c-5c9d1417c1da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xphc9"
Sep 18 17:49:57.932728 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.932289    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6fb9873f-b1a1-4d93-910c-5c9d1417c1da-etc-selinux\") pod \"aws-ebs-csi-driver-node-xphc9\" (UID: \"6fb9873f-b1a1-4d93-910c-5c9d1417c1da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xphc9"
Sep 18 17:49:57.932728 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.932320    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4945651f-6aa7-4ce0-82e2-3f5f234b607f-serviceca\") pod \"node-ca-zvmrv\" (UID: \"4945651f-6aa7-4ce0-82e2-3f5f234b607f\") " pod="openshift-image-registry/node-ca-zvmrv"
Sep 18 17:49:57.932728 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.932346    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6fb9873f-b1a1-4d93-910c-5c9d1417c1da-socket-dir\") pod \"aws-ebs-csi-driver-node-xphc9\" (UID: \"6fb9873f-b1a1-4d93-910c-5c9d1417c1da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xphc9"
Sep 18 17:49:57.932728 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.932363    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6fb9873f-b1a1-4d93-910c-5c9d1417c1da-sys-fs\") pod \"aws-ebs-csi-driver-node-xphc9\" (UID: \"6fb9873f-b1a1-4d93-910c-5c9d1417c1da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xphc9"
Sep 18 17:49:57.932728 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.932379    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srrwg\" (UniqueName: \"kubernetes.io/projected/6fb9873f-b1a1-4d93-910c-5c9d1417c1da-kube-api-access-srrwg\") pod \"aws-ebs-csi-driver-node-xphc9\" (UID: \"6fb9873f-b1a1-4d93-910c-5c9d1417c1da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xphc9"
Sep 18 17:49:57.932728 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.932400    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7a3ae2d8-a30b-4c83-a53c-634fb07621d6-kubelet-config\") pod \"global-pull-secret-syncer-mvtdm\" (UID: \"7a3ae2d8-a30b-4c83-a53c-634fb07621d6\") " pod="kube-system/global-pull-secret-syncer-mvtdm"
Sep 18 17:49:57.932728 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.932454    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7a3ae2d8-a30b-4c83-a53c-634fb07621d6-kubelet-config\") pod \"global-pull-secret-syncer-mvtdm\" (UID: \"7a3ae2d8-a30b-4c83-a53c-634fb07621d6\") " pod="kube-system/global-pull-secret-syncer-mvtdm"
Sep 18 17:49:57.932728 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.932454    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6fb9873f-b1a1-4d93-910c-5c9d1417c1da-sys-fs\") pod \"aws-ebs-csi-driver-node-xphc9\" (UID: \"6fb9873f-b1a1-4d93-910c-5c9d1417c1da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xphc9"
Sep 18 17:49:57.932728 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.932540    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6fb9873f-b1a1-4d93-910c-5c9d1417c1da-socket-dir\") pod \"aws-ebs-csi-driver-node-xphc9\" (UID: \"6fb9873f-b1a1-4d93-910c-5c9d1417c1da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xphc9"
Sep 18 17:49:57.933384 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.932570    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7a3ae2d8-a30b-4c83-a53c-634fb07621d6-original-pull-secret\") pod \"global-pull-secret-syncer-mvtdm\" (UID: \"7a3ae2d8-a30b-4c83-a53c-634fb07621d6\") " pod="kube-system/global-pull-secret-syncer-mvtdm"
Sep 18 17:49:57.933384 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.932604    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13-agent-certs\") pod \"konnectivity-agent-k2xs7\" (UID: \"313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13\") " pod="kube-system/konnectivity-agent-k2xs7"
Sep 18 17:49:57.933384 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.932612    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1dc25d25-efc8-477f-9a1e-5220f1d20345-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cqzh9\" (UID: \"1dc25d25-efc8-477f-9a1e-5220f1d20345\") " pod="openshift-multus/multus-additional-cni-plugins-cqzh9"
Sep 18 17:49:57.933384 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.932620    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13-konnectivity-ca\") pod \"konnectivity-agent-k2xs7\" (UID: \"313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13\") " pod="kube-system/konnectivity-agent-k2xs7"
Sep 18 17:49:57.933384 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:49:57.932684    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 17:49:57.933384 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:49:57.932757    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a3ae2d8-a30b-4c83-a53c-634fb07621d6-original-pull-secret podName:7a3ae2d8-a30b-4c83-a53c-634fb07621d6 nodeName:}" failed. No retries permitted until 2026-09-18 17:49:58.432736792 +0000 UTC m=+3.059868412 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7a3ae2d8-a30b-4c83-a53c-634fb07621d6-original-pull-secret") pod "global-pull-secret-syncer-mvtdm" (UID: "7a3ae2d8-a30b-4c83-a53c-634fb07621d6") : object "kube-system"/"original-pull-secret" not registered
Sep 18 17:49:57.933384 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.932880    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4945651f-6aa7-4ce0-82e2-3f5f234b607f-serviceca\") pod \"node-ca-zvmrv\" (UID: \"4945651f-6aa7-4ce0-82e2-3f5f234b607f\") " pod="openshift-image-registry/node-ca-zvmrv"
Sep 18 17:49:57.933384 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.933060    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13-konnectivity-ca\") pod \"konnectivity-agent-k2xs7\" (UID: \"313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13\") " pod="kube-system/konnectivity-agent-k2xs7"
Sep 18 17:49:57.935138 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.935108    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13-agent-certs\") pod \"konnectivity-agent-k2xs7\" (UID: \"313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13\") " pod="kube-system/konnectivity-agent-k2xs7"
Sep 18 17:49:57.942274 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.942250    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2mc4\" (UniqueName: \"kubernetes.io/projected/1dc25d25-efc8-477f-9a1e-5220f1d20345-kube-api-access-h2mc4\") pod \"multus-additional-cni-plugins-cqzh9\" (UID: \"1dc25d25-efc8-477f-9a1e-5220f1d20345\") " pod="openshift-multus/multus-additional-cni-plugins-cqzh9"
Sep 18 17:49:57.942958 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.942935    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srrwg\" (UniqueName: \"kubernetes.io/projected/6fb9873f-b1a1-4d93-910c-5c9d1417c1da-kube-api-access-srrwg\") pod \"aws-ebs-csi-driver-node-xphc9\" (UID: \"6fb9873f-b1a1-4d93-910c-5c9d1417c1da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xphc9"
Sep 18 17:49:57.950287 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:57.950263    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngwb6\" (UniqueName: \"kubernetes.io/projected/4945651f-6aa7-4ce0-82e2-3f5f234b607f-kube-api-access-ngwb6\") pod \"node-ca-zvmrv\" (UID: \"4945651f-6aa7-4ce0-82e2-3f5f234b607f\") " pod="openshift-image-registry/node-ca-zvmrv"
Sep 18 17:49:58.011431 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:58.011390    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mqjzn"
Sep 18 17:49:58.022587 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:58.022378    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-q2btm"
Sep 18 17:49:58.028208 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:58.028162    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod291a480f_a747_4910_a467_676a77e04eb0.slice/crio-0ee2a8cba32ed34dd4d073623c4076be644e16834bda995587d5d76b3b43cd82 WatchSource:0}: Error finding container 0ee2a8cba32ed34dd4d073623c4076be644e16834bda995587d5d76b3b43cd82: Status 404 returned error can't find the container with id 0ee2a8cba32ed34dd4d073623c4076be644e16834bda995587d5d76b3b43cd82
Sep 18 17:49:58.030413 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:58.030390    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2mqnb"
Sep 18 17:49:58.035315 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:58.035294    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk"
Sep 18 17:49:58.039992 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:58.039952    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1907e944_be36_42b2_bc90_90938a227371.slice/crio-3216b6a42d530162352beaef71da4c7213c426da8db67faef17c7462894bbdd7 WatchSource:0}: Error finding container 3216b6a42d530162352beaef71da4c7213c426da8db67faef17c7462894bbdd7: Status 404 returned error can't find the container with id 3216b6a42d530162352beaef71da4c7213c426da8db67faef17c7462894bbdd7
Sep 18 17:49:58.041104 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:58.041083    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4wsjj"
Sep 18 17:49:58.047182 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:58.047159    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xphc9"
Sep 18 17:49:58.053681 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:58.053657    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zvmrv"
Sep 18 17:49:58.060035 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:58.060016    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-k2xs7"
Sep 18 17:49:58.064076 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:58.063516    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cqzh9"
Sep 18 17:49:58.064076 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:58.063937    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65f6cbfa_00b5_4a5c_be75_28392cf8a1a8.slice/crio-ccdb8768655ac493e1d6ea329528cc6da347da5304533624bcfc58a181cbc01c WatchSource:0}: Error finding container ccdb8768655ac493e1d6ea329528cc6da347da5304533624bcfc58a181cbc01c: Status 404 returned error can't find the container with id ccdb8768655ac493e1d6ea329528cc6da347da5304533624bcfc58a181cbc01c
Sep 18 17:49:58.065678 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:58.065653    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod548befd4_2d47_4b4d_a376_c295e0dc141b.slice/crio-1f66903e39e6ce5162400a90b4e8391ef76fa4a8823b7a64ad64b4ba5aaf852d WatchSource:0}: Error finding container 1f66903e39e6ce5162400a90b4e8391ef76fa4a8823b7a64ad64b4ba5aaf852d: Status 404 returned error can't find the container with id 1f66903e39e6ce5162400a90b4e8391ef76fa4a8823b7a64ad64b4ba5aaf852d
Sep 18 17:49:58.067097 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:58.067084    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5d026b7_af6b_4899_85f4_6554c0f7cd52.slice/crio-3431068ad333076da5dbc50e523e9ad3bb14d967b1a481d25975aa83e86669c0 WatchSource:0}: Error finding container 3431068ad333076da5dbc50e523e9ad3bb14d967b1a481d25975aa83e86669c0: Status 404 returned error can't find the container with id 3431068ad333076da5dbc50e523e9ad3bb14d967b1a481d25975aa83e86669c0
Sep 18 17:49:58.073181 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:58.071345    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fb9873f_b1a1_4d93_910c_5c9d1417c1da.slice/crio-f3fa2ec1b1a8676b335c88d75da43b19a361ad7305b94d56b92ab85b0d19ad65 WatchSource:0}: Error finding container f3fa2ec1b1a8676b335c88d75da43b19a361ad7305b94d56b92ab85b0d19ad65: Status 404 returned error can't find the container with id f3fa2ec1b1a8676b335c88d75da43b19a361ad7305b94d56b92ab85b0d19ad65
Sep 18 17:49:58.081375 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:49:58.080988    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4945651f_6aa7_4ce0_82e2_3f5f234b607f.slice/crio-84ce419b1b61fe799cc0d154c2fe29fe7fc977d43219038fce5be586be54c58d WatchSource:0}: Error finding container 84ce419b1b61fe799cc0d154c2fe29fe7fc977d43219038fce5be586be54c58d: Status 404 returned error can't find the container with id 84ce419b1b61fe799cc0d154c2fe29fe7fc977d43219038fce5be586be54c58d
Sep 18 17:49:58.336586 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:58.336500    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45a7fd6c-fe96-4a1a-b3a5-2cb78c5435e8-metrics-certs\") pod \"network-metrics-daemon-tmtsv\" (UID: \"45a7fd6c-fe96-4a1a-b3a5-2cb78c5435e8\") " pod="openshift-multus/network-metrics-daemon-tmtsv"
Sep 18 17:49:58.336735 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:49:58.336695    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 17:49:58.336792 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:49:58.336776    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45a7fd6c-fe96-4a1a-b3a5-2cb78c5435e8-metrics-certs podName:45a7fd6c-fe96-4a1a-b3a5-2cb78c5435e8 nodeName:}" failed. No retries permitted until 2026-09-18 17:49:59.336759666 +0000 UTC m=+3.963891285 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/45a7fd6c-fe96-4a1a-b3a5-2cb78c5435e8-metrics-certs") pod "network-metrics-daemon-tmtsv" (UID: "45a7fd6c-fe96-4a1a-b3a5-2cb78c5435e8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 17:49:58.437425 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:58.437387    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scbjn\" (UniqueName: \"kubernetes.io/projected/234003fa-8d5d-4108-8f62-859ea05eb9aa-kube-api-access-scbjn\") pod \"network-check-target-bgwtf\" (UID: \"234003fa-8d5d-4108-8f62-859ea05eb9aa\") " pod="openshift-network-diagnostics/network-check-target-bgwtf"
Sep 18 17:49:58.437566 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:58.437449    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7a3ae2d8-a30b-4c83-a53c-634fb07621d6-original-pull-secret\") pod \"global-pull-secret-syncer-mvtdm\" (UID: \"7a3ae2d8-a30b-4c83-a53c-634fb07621d6\") " pod="kube-system/global-pull-secret-syncer-mvtdm"
Sep 18 17:49:58.437636 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:49:58.437621    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 17:49:58.437703 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:49:58.437692    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a3ae2d8-a30b-4c83-a53c-634fb07621d6-original-pull-secret podName:7a3ae2d8-a30b-4c83-a53c-634fb07621d6 nodeName:}" failed. No retries permitted until 2026-09-18 17:49:59.437671933 +0000 UTC m=+4.064803535 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7a3ae2d8-a30b-4c83-a53c-634fb07621d6-original-pull-secret") pod "global-pull-secret-syncer-mvtdm" (UID: "7a3ae2d8-a30b-4c83-a53c-634fb07621d6") : object "kube-system"/"original-pull-secret" not registered
Sep 18 17:49:58.438178 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:49:58.438150    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 17:49:58.438178 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:49:58.438173    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 17:49:58.438263 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:49:58.438186    2561 projected.go:196] Error preparing data for projected volume kube-api-access-scbjn for pod openshift-network-diagnostics/network-check-target-bgwtf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 17:49:58.438263 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:49:58.438257    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/234003fa-8d5d-4108-8f62-859ea05eb9aa-kube-api-access-scbjn podName:234003fa-8d5d-4108-8f62-859ea05eb9aa nodeName:}" failed. No retries permitted until 2026-09-18 17:49:59.438234132 +0000 UTC m=+4.065365737 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-scbjn" (UniqueName: "kubernetes.io/projected/234003fa-8d5d-4108-8f62-859ea05eb9aa-kube-api-access-scbjn") pod "network-check-target-bgwtf" (UID: "234003fa-8d5d-4108-8f62-859ea05eb9aa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 17:49:58.775142 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:58.775019    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 17:44:58 +0000 UTC" deadline="2028-05-28 04:30:00.176288348 +0000 UTC"
Sep 18 17:49:58.775142 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:58.775073    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14818h40m1.401218107s"
Sep 18 17:49:58.864062 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:58.864022    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cqzh9" event={"ID":"1dc25d25-efc8-477f-9a1e-5220f1d20345","Type":"ContainerStarted","Data":"f42d8dab31a2ed2c9ee6234aa54429e2b76055a94e7e82cfcf405afae9772f33"}
Sep 18 17:49:58.865472 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:58.865439    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2mqnb" event={"ID":"65f6cbfa-00b5-4a5c-be75-28392cf8a1a8","Type":"ContainerStarted","Data":"ccdb8768655ac493e1d6ea329528cc6da347da5304533624bcfc58a181cbc01c"}
Sep 18 17:49:58.866814 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:58.866784    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mqjzn" event={"ID":"291a480f-a747-4910-a467-676a77e04eb0","Type":"ContainerStarted","Data":"0ee2a8cba32ed34dd4d073623c4076be644e16834bda995587d5d76b3b43cd82"}
Sep 18 17:49:58.868168 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:58.868115    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-k2xs7" event={"ID":"313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13","Type":"ContainerStarted","Data":"6c300411945c61731e7bcdbdee6522d267c8b32cfbba605076f59fa58a623aca"}
Sep 18 17:49:58.869457 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:58.869429    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zvmrv" event={"ID":"4945651f-6aa7-4ce0-82e2-3f5f234b607f","Type":"ContainerStarted","Data":"84ce419b1b61fe799cc0d154c2fe29fe7fc977d43219038fce5be586be54c58d"}
Sep 18 17:49:58.870962 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:58.870938    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xphc9" event={"ID":"6fb9873f-b1a1-4d93-910c-5c9d1417c1da","Type":"ContainerStarted","Data":"f3fa2ec1b1a8676b335c88d75da43b19a361ad7305b94d56b92ab85b0d19ad65"}
Sep 18 17:49:58.872509 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:58.872484    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk" event={"ID":"c5d026b7-af6b-4899-85f4-6554c0f7cd52","Type":"ContainerStarted","Data":"3431068ad333076da5dbc50e523e9ad3bb14d967b1a481d25975aa83e86669c0"}
Sep 18 17:49:58.873893 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:58.873873    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4wsjj" event={"ID":"548befd4-2d47-4b4d-a376-c295e0dc141b","Type":"ContainerStarted","Data":"1f66903e39e6ce5162400a90b4e8391ef76fa4a8823b7a64ad64b4ba5aaf852d"}
Sep 18 17:49:58.875166 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:58.875098    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-q2btm" event={"ID":"1907e944-be36-42b2-bc90-90938a227371","Type":"ContainerStarted","Data":"3216b6a42d530162352beaef71da4c7213c426da8db67faef17c7462894bbdd7"}
Sep 18 17:49:59.346688 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:59.346647    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45a7fd6c-fe96-4a1a-b3a5-2cb78c5435e8-metrics-certs\") pod \"network-metrics-daemon-tmtsv\" (UID: \"45a7fd6c-fe96-4a1a-b3a5-2cb78c5435e8\") " pod="openshift-multus/network-metrics-daemon-tmtsv"
Sep 18 17:49:59.346885 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:49:59.346867    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 17:49:59.346953 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:49:59.346934    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45a7fd6c-fe96-4a1a-b3a5-2cb78c5435e8-metrics-certs podName:45a7fd6c-fe96-4a1a-b3a5-2cb78c5435e8 nodeName:}" failed. No retries permitted until 2026-09-18 17:50:01.346915624 +0000 UTC m=+5.974047224 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/45a7fd6c-fe96-4a1a-b3a5-2cb78c5435e8-metrics-certs") pod "network-metrics-daemon-tmtsv" (UID: "45a7fd6c-fe96-4a1a-b3a5-2cb78c5435e8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 17:49:59.448923 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:59.448069    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scbjn\" (UniqueName: \"kubernetes.io/projected/234003fa-8d5d-4108-8f62-859ea05eb9aa-kube-api-access-scbjn\") pod \"network-check-target-bgwtf\" (UID: \"234003fa-8d5d-4108-8f62-859ea05eb9aa\") " pod="openshift-network-diagnostics/network-check-target-bgwtf"
Sep 18 17:49:59.448923 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:59.448130    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7a3ae2d8-a30b-4c83-a53c-634fb07621d6-original-pull-secret\") pod \"global-pull-secret-syncer-mvtdm\" (UID: \"7a3ae2d8-a30b-4c83-a53c-634fb07621d6\") " pod="kube-system/global-pull-secret-syncer-mvtdm"
Sep 18 17:49:59.448923 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:49:59.448274    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 17:49:59.448923 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:49:59.448343    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a3ae2d8-a30b-4c83-a53c-634fb07621d6-original-pull-secret podName:7a3ae2d8-a30b-4c83-a53c-634fb07621d6 nodeName:}" failed. No retries permitted until 2026-09-18 17:50:01.448324231 +0000 UTC m=+6.075455840 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7a3ae2d8-a30b-4c83-a53c-634fb07621d6-original-pull-secret") pod "global-pull-secret-syncer-mvtdm" (UID: "7a3ae2d8-a30b-4c83-a53c-634fb07621d6") : object "kube-system"/"original-pull-secret" not registered
Sep 18 17:49:59.448923 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:49:59.448786    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 17:49:59.448923 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:49:59.448805    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 17:49:59.448923 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:49:59.448818    2561 projected.go:196] Error preparing data for projected volume kube-api-access-scbjn for pod openshift-network-diagnostics/network-check-target-bgwtf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 17:49:59.448923 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:49:59.448866    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/234003fa-8d5d-4108-8f62-859ea05eb9aa-kube-api-access-scbjn podName:234003fa-8d5d-4108-8f62-859ea05eb9aa nodeName:}" failed. No retries permitted until 2026-09-18 17:50:01.448850756 +0000 UTC m=+6.075982361 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-scbjn" (UniqueName: "kubernetes.io/projected/234003fa-8d5d-4108-8f62-859ea05eb9aa-kube-api-access-scbjn") pod "network-check-target-bgwtf" (UID: "234003fa-8d5d-4108-8f62-859ea05eb9aa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 17:49:59.776090 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:59.775993    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 17:44:58 +0000 UTC" deadline="2028-03-13 18:17:40.572132782 +0000 UTC"
Sep 18 17:49:59.776090 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:59.776034    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13008h27m40.796103432s"
Sep 18 17:49:59.857584 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:59.857540    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tmtsv"
Sep 18 17:49:59.857769 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:49:59.857681    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tmtsv" podUID="45a7fd6c-fe96-4a1a-b3a5-2cb78c5435e8"
Sep 18 17:49:59.858123 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:59.858099    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mvtdm"
Sep 18 17:49:59.858249 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:49:59.858227    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mvtdm" podUID="7a3ae2d8-a30b-4c83-a53c-634fb07621d6"
Sep 18 17:49:59.858366 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:49:59.858302    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bgwtf"
Sep 18 17:49:59.858420 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:49:59.858387    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bgwtf" podUID="234003fa-8d5d-4108-8f62-859ea05eb9aa"
Sep 18 17:50:01.362930 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:01.362885    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45a7fd6c-fe96-4a1a-b3a5-2cb78c5435e8-metrics-certs\") pod \"network-metrics-daemon-tmtsv\" (UID: \"45a7fd6c-fe96-4a1a-b3a5-2cb78c5435e8\") " pod="openshift-multus/network-metrics-daemon-tmtsv"
Sep 18 17:50:01.363494 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:01.363054    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 17:50:01.363494 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:01.363116    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45a7fd6c-fe96-4a1a-b3a5-2cb78c5435e8-metrics-certs podName:45a7fd6c-fe96-4a1a-b3a5-2cb78c5435e8 nodeName:}" failed. No retries permitted until 2026-09-18 17:50:05.363097713 +0000 UTC m=+9.990229317 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/45a7fd6c-fe96-4a1a-b3a5-2cb78c5435e8-metrics-certs") pod "network-metrics-daemon-tmtsv" (UID: "45a7fd6c-fe96-4a1a-b3a5-2cb78c5435e8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 17:50:01.464186 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:01.463742    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scbjn\" (UniqueName: \"kubernetes.io/projected/234003fa-8d5d-4108-8f62-859ea05eb9aa-kube-api-access-scbjn\") pod \"network-check-target-bgwtf\" (UID: \"234003fa-8d5d-4108-8f62-859ea05eb9aa\") " pod="openshift-network-diagnostics/network-check-target-bgwtf"
Sep 18 17:50:01.464186 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:01.463799    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7a3ae2d8-a30b-4c83-a53c-634fb07621d6-original-pull-secret\") pod \"global-pull-secret-syncer-mvtdm\" (UID: \"7a3ae2d8-a30b-4c83-a53c-634fb07621d6\") " pod="kube-system/global-pull-secret-syncer-mvtdm"
Sep 18 17:50:01.464186 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:01.463923    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 17:50:01.464186 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:01.463933    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 17:50:01.464186 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:01.463960    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 17:50:01.464186 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:01.463974    2561 projected.go:196] Error preparing data for projected volume kube-api-access-scbjn for pod openshift-network-diagnostics/network-check-target-bgwtf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 17:50:01.464186 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:01.464007    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a3ae2d8-a30b-4c83-a53c-634fb07621d6-original-pull-secret podName:7a3ae2d8-a30b-4c83-a53c-634fb07621d6 nodeName:}" failed. No retries permitted until 2026-09-18 17:50:05.463966254 +0000 UTC m=+10.091097862 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7a3ae2d8-a30b-4c83-a53c-634fb07621d6-original-pull-secret") pod "global-pull-secret-syncer-mvtdm" (UID: "7a3ae2d8-a30b-4c83-a53c-634fb07621d6") : object "kube-system"/"original-pull-secret" not registered
Sep 18 17:50:01.464186 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:01.464031    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/234003fa-8d5d-4108-8f62-859ea05eb9aa-kube-api-access-scbjn podName:234003fa-8d5d-4108-8f62-859ea05eb9aa nodeName:}" failed. No retries permitted until 2026-09-18 17:50:05.464013661 +0000 UTC m=+10.091145280 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-scbjn" (UniqueName: "kubernetes.io/projected/234003fa-8d5d-4108-8f62-859ea05eb9aa-kube-api-access-scbjn") pod "network-check-target-bgwtf" (UID: "234003fa-8d5d-4108-8f62-859ea05eb9aa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 17:50:01.855303 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:01.855218    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tmtsv"
Sep 18 17:50:01.855477 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:01.855219    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bgwtf"
Sep 18 17:50:01.855477 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:01.855409    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tmtsv" podUID="45a7fd6c-fe96-4a1a-b3a5-2cb78c5435e8"
Sep 18 17:50:01.855605 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:01.855492    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bgwtf" podUID="234003fa-8d5d-4108-8f62-859ea05eb9aa"
Sep 18 17:50:01.855605 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:01.855560    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mvtdm"
Sep 18 17:50:01.855726 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:01.855642    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mvtdm" podUID="7a3ae2d8-a30b-4c83-a53c-634fb07621d6"
Sep 18 17:50:03.855540 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:03.855503    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bgwtf"
Sep 18 17:50:03.855922 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:03.855547    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tmtsv"
Sep 18 17:50:03.855922 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:03.855649    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bgwtf" podUID="234003fa-8d5d-4108-8f62-859ea05eb9aa"
Sep 18 17:50:03.855922 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:03.855806    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tmtsv" podUID="45a7fd6c-fe96-4a1a-b3a5-2cb78c5435e8"
Sep 18 17:50:03.855922 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:03.855506    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mvtdm"
Sep 18 17:50:03.855922 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:03.855915    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mvtdm" podUID="7a3ae2d8-a30b-4c83-a53c-634fb07621d6"
Sep 18 17:50:05.396898 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:05.396794    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45a7fd6c-fe96-4a1a-b3a5-2cb78c5435e8-metrics-certs\") pod \"network-metrics-daemon-tmtsv\" (UID: \"45a7fd6c-fe96-4a1a-b3a5-2cb78c5435e8\") " pod="openshift-multus/network-metrics-daemon-tmtsv"
Sep 18 17:50:05.397625 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:05.397140    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 17:50:05.397625 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:05.397269    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45a7fd6c-fe96-4a1a-b3a5-2cb78c5435e8-metrics-certs podName:45a7fd6c-fe96-4a1a-b3a5-2cb78c5435e8 nodeName:}" failed. No retries permitted until 2026-09-18 17:50:13.397213423 +0000 UTC m=+18.024345028 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/45a7fd6c-fe96-4a1a-b3a5-2cb78c5435e8-metrics-certs") pod "network-metrics-daemon-tmtsv" (UID: "45a7fd6c-fe96-4a1a-b3a5-2cb78c5435e8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 17:50:05.498042 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:05.497702    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scbjn\" (UniqueName: \"kubernetes.io/projected/234003fa-8d5d-4108-8f62-859ea05eb9aa-kube-api-access-scbjn\") pod \"network-check-target-bgwtf\" (UID: \"234003fa-8d5d-4108-8f62-859ea05eb9aa\") " pod="openshift-network-diagnostics/network-check-target-bgwtf"
Sep 18 17:50:05.498042 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:05.497763    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7a3ae2d8-a30b-4c83-a53c-634fb07621d6-original-pull-secret\") pod \"global-pull-secret-syncer-mvtdm\" (UID: \"7a3ae2d8-a30b-4c83-a53c-634fb07621d6\") " pod="kube-system/global-pull-secret-syncer-mvtdm"
Sep 18 17:50:05.498042 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:05.497927    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 17:50:05.498042 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:05.497945    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 17:50:05.498042 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:05.497962    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 17:50:05.498042 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:05.497977    2561 projected.go:196] Error preparing data for projected volume kube-api-access-scbjn for pod openshift-network-diagnostics/network-check-target-bgwtf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 17:50:05.498042 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:05.497999    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a3ae2d8-a30b-4c83-a53c-634fb07621d6-original-pull-secret podName:7a3ae2d8-a30b-4c83-a53c-634fb07621d6 nodeName:}" failed. No retries permitted until 2026-09-18 17:50:13.497979717 +0000 UTC m=+18.125111322 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7a3ae2d8-a30b-4c83-a53c-634fb07621d6-original-pull-secret") pod "global-pull-secret-syncer-mvtdm" (UID: "7a3ae2d8-a30b-4c83-a53c-634fb07621d6") : object "kube-system"/"original-pull-secret" not registered
Sep 18 17:50:05.498042 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:05.498018    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/234003fa-8d5d-4108-8f62-859ea05eb9aa-kube-api-access-scbjn podName:234003fa-8d5d-4108-8f62-859ea05eb9aa nodeName:}" failed. No retries permitted until 2026-09-18 17:50:13.498008995 +0000 UTC m=+18.125140595 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-scbjn" (UniqueName: "kubernetes.io/projected/234003fa-8d5d-4108-8f62-859ea05eb9aa-kube-api-access-scbjn") pod "network-check-target-bgwtf" (UID: "234003fa-8d5d-4108-8f62-859ea05eb9aa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 17:50:05.856627 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:05.856578    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mvtdm"
Sep 18 17:50:05.856799 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:05.856642    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tmtsv"
Sep 18 17:50:05.856799 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:05.856714    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bgwtf"
Sep 18 17:50:05.856924 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:05.856823    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bgwtf" podUID="234003fa-8d5d-4108-8f62-859ea05eb9aa"
Sep 18 17:50:05.856924 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:05.856881    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mvtdm" podUID="7a3ae2d8-a30b-4c83-a53c-634fb07621d6"
Sep 18 17:50:05.857041 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:05.856974    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tmtsv" podUID="45a7fd6c-fe96-4a1a-b3a5-2cb78c5435e8"
Sep 18 17:50:07.855613 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:07.855574    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bgwtf"
Sep 18 17:50:07.856067 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:07.855714    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bgwtf" podUID="234003fa-8d5d-4108-8f62-859ea05eb9aa"
Sep 18 17:50:07.856151 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:07.856100    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tmtsv"
Sep 18 17:50:07.856232 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:07.856217    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tmtsv" podUID="45a7fd6c-fe96-4a1a-b3a5-2cb78c5435e8"
Sep 18 17:50:07.856411 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:07.856353    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mvtdm"
Sep 18 17:50:07.856570 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:07.856434    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mvtdm" podUID="7a3ae2d8-a30b-4c83-a53c-634fb07621d6"
Sep 18 17:50:09.855058 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:09.855019    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tmtsv"
Sep 18 17:50:09.855058 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:09.855058    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mvtdm"
Sep 18 17:50:09.855591 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:09.855021    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bgwtf"
Sep 18 17:50:09.855591 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:09.855180    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tmtsv" podUID="45a7fd6c-fe96-4a1a-b3a5-2cb78c5435e8"
Sep 18 17:50:09.855591 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:09.855276    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bgwtf" podUID="234003fa-8d5d-4108-8f62-859ea05eb9aa"
Sep 18 17:50:09.855591 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:09.855363    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mvtdm" podUID="7a3ae2d8-a30b-4c83-a53c-634fb07621d6"
Sep 18 17:50:11.855334 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:11.855291    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bgwtf"
Sep 18 17:50:11.855775 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:11.855288    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mvtdm"
Sep 18 17:50:11.855775 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:11.855407    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bgwtf" podUID="234003fa-8d5d-4108-8f62-859ea05eb9aa"
Sep 18 17:50:11.855775 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:11.855507    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mvtdm" podUID="7a3ae2d8-a30b-4c83-a53c-634fb07621d6"
Sep 18 17:50:11.855775 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:11.855305    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tmtsv"
Sep 18 17:50:11.855775 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:11.855615    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tmtsv" podUID="45a7fd6c-fe96-4a1a-b3a5-2cb78c5435e8"
Sep 18 17:50:13.456052 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:13.456013    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45a7fd6c-fe96-4a1a-b3a5-2cb78c5435e8-metrics-certs\") pod \"network-metrics-daemon-tmtsv\" (UID: \"45a7fd6c-fe96-4a1a-b3a5-2cb78c5435e8\") " pod="openshift-multus/network-metrics-daemon-tmtsv"
Sep 18 17:50:13.456627 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:13.456156    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 17:50:13.456627 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:13.456214    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45a7fd6c-fe96-4a1a-b3a5-2cb78c5435e8-metrics-certs podName:45a7fd6c-fe96-4a1a-b3a5-2cb78c5435e8 nodeName:}" failed. No retries permitted until 2026-09-18 17:50:29.456199294 +0000 UTC m=+34.083330896 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/45a7fd6c-fe96-4a1a-b3a5-2cb78c5435e8-metrics-certs") pod "network-metrics-daemon-tmtsv" (UID: "45a7fd6c-fe96-4a1a-b3a5-2cb78c5435e8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 17:50:13.556711 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:13.556670    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scbjn\" (UniqueName: \"kubernetes.io/projected/234003fa-8d5d-4108-8f62-859ea05eb9aa-kube-api-access-scbjn\") pod \"network-check-target-bgwtf\" (UID: \"234003fa-8d5d-4108-8f62-859ea05eb9aa\") " pod="openshift-network-diagnostics/network-check-target-bgwtf"
Sep 18 17:50:13.556876 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:13.556727    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7a3ae2d8-a30b-4c83-a53c-634fb07621d6-original-pull-secret\") pod \"global-pull-secret-syncer-mvtdm\" (UID: \"7a3ae2d8-a30b-4c83-a53c-634fb07621d6\") " pod="kube-system/global-pull-secret-syncer-mvtdm"
Sep 18 17:50:13.556876 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:13.556836    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 17:50:13.556876 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:13.556864    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 17:50:13.557021 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:13.556892    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 17:50:13.557021 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:13.556901    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a3ae2d8-a30b-4c83-a53c-634fb07621d6-original-pull-secret podName:7a3ae2d8-a30b-4c83-a53c-634fb07621d6 nodeName:}" failed. No retries permitted until 2026-09-18 17:50:29.556882231 +0000 UTC m=+34.184013833 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7a3ae2d8-a30b-4c83-a53c-634fb07621d6-original-pull-secret") pod "global-pull-secret-syncer-mvtdm" (UID: "7a3ae2d8-a30b-4c83-a53c-634fb07621d6") : object "kube-system"/"original-pull-secret" not registered
Sep 18 17:50:13.557021 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:13.556905    2561 projected.go:196] Error preparing data for projected volume kube-api-access-scbjn for pod openshift-network-diagnostics/network-check-target-bgwtf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 17:50:13.557021 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:13.556957    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/234003fa-8d5d-4108-8f62-859ea05eb9aa-kube-api-access-scbjn podName:234003fa-8d5d-4108-8f62-859ea05eb9aa nodeName:}" failed. No retries permitted until 2026-09-18 17:50:29.556940128 +0000 UTC m=+34.184071729 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-scbjn" (UniqueName: "kubernetes.io/projected/234003fa-8d5d-4108-8f62-859ea05eb9aa-kube-api-access-scbjn") pod "network-check-target-bgwtf" (UID: "234003fa-8d5d-4108-8f62-859ea05eb9aa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 17:50:13.855610 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:13.855536    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tmtsv"
Sep 18 17:50:13.855768 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:13.855537    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bgwtf"
Sep 18 17:50:13.855768 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:13.855667    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tmtsv" podUID="45a7fd6c-fe96-4a1a-b3a5-2cb78c5435e8"
Sep 18 17:50:13.855768 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:13.855538    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mvtdm"
Sep 18 17:50:13.855768 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:13.855725    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bgwtf" podUID="234003fa-8d5d-4108-8f62-859ea05eb9aa"
Sep 18 17:50:13.855978 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:13.855791    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mvtdm" podUID="7a3ae2d8-a30b-4c83-a53c-634fb07621d6"
Sep 18 17:50:15.855912 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:15.855875    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mvtdm"
Sep 18 17:50:15.856409 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:15.855964    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tmtsv"
Sep 18 17:50:15.856409 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:15.855991    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mvtdm" podUID="7a3ae2d8-a30b-4c83-a53c-634fb07621d6"
Sep 18 17:50:15.856409 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:15.856092    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tmtsv" podUID="45a7fd6c-fe96-4a1a-b3a5-2cb78c5435e8"
Sep 18 17:50:15.856409 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:15.856152    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bgwtf"
Sep 18 17:50:15.856409 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:15.856255    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bgwtf" podUID="234003fa-8d5d-4108-8f62-859ea05eb9aa"
Sep 18 17:50:16.914861 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:16.914650    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mqjzn" event={"ID":"291a480f-a747-4910-a467-676a77e04eb0","Type":"ContainerStarted","Data":"073abac721803aa627af11fdbd2f200ac6b40d970ba39ed636f4c73f8457c7cb"}
Sep 18 17:50:16.916272 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:16.916250    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-54.ec2.internal" event={"ID":"bfcddef9dbb20737aafa5928064e8e0d","Type":"ContainerStarted","Data":"8da28156d3f76f87f508358b39bb4f9926b3416ba2ea15b36851821219c97e24"}
Sep 18 17:50:16.918110 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:16.918086    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk" event={"ID":"c5d026b7-af6b-4899-85f4-6554c0f7cd52","Type":"ContainerStarted","Data":"65ab8eba00d48767df48a629cefbfc171f92976542005e4d7304a5b6e64bab88"}
Sep 18 17:50:16.931584 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:16.931545    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-mqjzn" podStartSLOduration=2.301107618 podStartE2EDuration="20.931531897s" totalImagesPullingTime="18.630424279s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 17:49:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:50:16.930975085 +0000 UTC m=+21.558106705" watchObservedRunningTime="2026-09-18 17:50:16.931531897 +0000 UTC m=+21.558663582"
Sep 18 17:50:16.943975 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:16.943944    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-54.ec2.internal" podStartSLOduration=19.943929814 podStartE2EDuration="19.943929814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 17:49:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:50:16.943776727 +0000 UTC m=+21.570908365" watchObservedRunningTime="2026-09-18 17:50:16.943929814 +0000 UTC m=+21.571061445"
Sep 18 17:50:17.855675 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:17.855491    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tmtsv"
Sep 18 17:50:17.855807 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:17.855499    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mvtdm"
Sep 18 17:50:17.855807 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:17.855755    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tmtsv" podUID="45a7fd6c-fe96-4a1a-b3a5-2cb78c5435e8"
Sep 18 17:50:17.855875 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:17.855552    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bgwtf"
Sep 18 17:50:17.855875 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:17.855818    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mvtdm" podUID="7a3ae2d8-a30b-4c83-a53c-634fb07621d6"
Sep 18 17:50:17.855933 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:17.855884    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bgwtf" podUID="234003fa-8d5d-4108-8f62-859ea05eb9aa"
Sep 18 17:50:17.921303 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:17.921279    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-k2xs7" event={"ID":"313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13","Type":"ContainerStarted","Data":"d5bacffbc203ee887d96650fdb3f869556a5095ebff5dbd31317bd4d13cc8e00"}
Sep 18 17:50:17.922763 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:17.922741    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zvmrv" event={"ID":"4945651f-6aa7-4ce0-82e2-3f5f234b607f","Type":"ContainerStarted","Data":"f79d708603eb0d61312aff6818e6709c6895ce6e939a314c19bda33fa401b789"}
Sep 18 17:50:17.924227 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:17.924205    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xphc9" event={"ID":"6fb9873f-b1a1-4d93-910c-5c9d1417c1da","Type":"ContainerStarted","Data":"c58660c1091e11fd709d0a9ed31d085abafebcc8fbbdb13ae7e3f5931e1a4307"}
Sep 18 17:50:17.926778 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:17.926761    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkcgk_c5d026b7-af6b-4899-85f4-6554c0f7cd52/ovn-acl-logging/0.log"
Sep 18 17:50:17.927091 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:17.927074    2561 generic.go:356] "Generic (PLEG): container finished" podID="c5d026b7-af6b-4899-85f4-6554c0f7cd52" containerID="541b6c4d5464aee51385a6b07445b88be06d1cb70c4dc240922c83829a079864" exitCode=1
Sep 18 17:50:17.927157 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:17.927134    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk" event={"ID":"c5d026b7-af6b-4899-85f4-6554c0f7cd52","Type":"ContainerStarted","Data":"463a60cb0de1241f47841ed5a6d1bc53f4f30d399f32ed1f73cb428cd253add2"}
Sep 18 17:50:17.927205 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:17.927190    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk" event={"ID":"c5d026b7-af6b-4899-85f4-6554c0f7cd52","Type":"ContainerStarted","Data":"cac4bd67974ce46a8f75503628aadc7bce42d62520a57cbaadd72139315549fa"}
Sep 18 17:50:17.927248 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:17.927222    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk" event={"ID":"c5d026b7-af6b-4899-85f4-6554c0f7cd52","Type":"ContainerStarted","Data":"41d206e91d4b1e7d7366f64d69e43d1317f498b17d78ac8a70a891299e8b45e5"}
Sep 18 17:50:17.927282 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:17.927259    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk" event={"ID":"c5d026b7-af6b-4899-85f4-6554c0f7cd52","Type":"ContainerStarted","Data":"e4c34fbae52e1be1eceaab91447cc84183f16db2479f4a3abbc5105aeafd0db6"}
Sep 18 17:50:17.927324 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:17.927287    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk" event={"ID":"c5d026b7-af6b-4899-85f4-6554c0f7cd52","Type":"ContainerDied","Data":"541b6c4d5464aee51385a6b07445b88be06d1cb70c4dc240922c83829a079864"}
Sep 18 17:50:17.928410 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:17.928386    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4wsjj" event={"ID":"548befd4-2d47-4b4d-a376-c295e0dc141b","Type":"ContainerStarted","Data":"956eaf5e50441733aa2bcee531a0e5c14ea691d9570c2a4b9b86082341a3002c"}
Sep 18 17:50:17.929714 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:17.929696    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-q2btm" event={"ID":"1907e944-be36-42b2-bc90-90938a227371","Type":"ContainerStarted","Data":"1664ba1df11505a04fed892ddeec8879d0a9fec917efab945cb002b0e74e6637"}
Sep 18 17:50:17.931007 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:17.930989    2561 generic.go:356] "Generic (PLEG): container finished" podID="0b9f4e0872eb36d1a9eb7f46a41c431b" containerID="2eb3a8657d586f134c264120f61d26735532f074ba900c5c7f52132606ab3cec" exitCode=0
Sep 18 17:50:17.931103 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:17.931047    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-54.ec2.internal" event={"ID":"0b9f4e0872eb36d1a9eb7f46a41c431b","Type":"ContainerDied","Data":"2eb3a8657d586f134c264120f61d26735532f074ba900c5c7f52132606ab3cec"}
Sep 18 17:50:17.932621 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:17.932568    2561 generic.go:356] "Generic (PLEG): container finished" podID="1dc25d25-efc8-477f-9a1e-5220f1d20345" containerID="0e02cdb16f7637be71165b9062c5590183e862f81c8481207a546c8a8a06303b" exitCode=0
Sep 18 17:50:17.932709 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:17.932645    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cqzh9" event={"ID":"1dc25d25-efc8-477f-9a1e-5220f1d20345","Type":"ContainerDied","Data":"0e02cdb16f7637be71165b9062c5590183e862f81c8481207a546c8a8a06303b"}
Sep 18 17:50:17.935223 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:17.935183    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-k2xs7" podStartSLOduration=2.3628219489999998 podStartE2EDuration="20.935171317s" totalImagesPullingTime="18.572349368s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:49:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:50:17.934727797 +0000 UTC m=+22.561859417" watchObservedRunningTime="2026-09-18 17:50:17.935171317 +0000 UTC m=+22.562302936"
Sep 18 17:50:17.968887 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:17.968839    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-q2btm" podStartSLOduration=15.015000808 podStartE2EDuration="21.968820088s" totalImagesPullingTime="6.95381928s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 17:49:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:50:17.96851574 +0000 UTC m=+22.595647360" watchObservedRunningTime="2026-09-18 17:50:17.968820088 +0000 UTC m=+22.595951708"
Sep 18 17:50:17.984722 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:17.984676    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-4wsjj" podStartSLOduration=2.974056172 podStartE2EDuration="21.98466009s" totalImagesPullingTime="19.010603918s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 17:49:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:50:17.98445581 +0000 UTC m=+22.611587429" watchObservedRunningTime="2026-09-18 17:50:17.98466009 +0000 UTC m=+22.611791709"
Sep 18 17:50:18.026015 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:18.025971    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-zvmrv" podStartSLOduration=6.522779226 podStartE2EDuration="21.025951344s" totalImagesPullingTime="14.503172118s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 17:49:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:50:18.025725353 +0000 UTC m=+22.652856972" watchObservedRunningTime="2026-09-18 17:50:18.025951344 +0000 UTC m=+22.653082966"
Sep 18 17:50:18.405205 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:18.405039    2561 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 18 17:50:18.661039 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:18.660922    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-k2xs7"
Sep 18 17:50:18.661611 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:18.661595    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-k2xs7"
Sep 18 17:50:18.781792 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:18.781630    2561 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-18T17:50:18.405194575Z","UUID":"667706c9-68e8-4b3c-9bd8-0858c505e36c","Handler":null,"Name":"","Endpoint":""}
Sep 18 17:50:18.782920 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:18.782900    2561 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 18 17:50:18.782920 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:18.782926    2561 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 18 17:50:18.938039 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:18.937938    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xphc9" event={"ID":"6fb9873f-b1a1-4d93-910c-5c9d1417c1da","Type":"ContainerStarted","Data":"e6ba63d142bf8fb2e3dc494a764483209df6436832957f45febd41d80c85cc79"}
Sep 18 17:50:18.940349 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:18.940278    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-54.ec2.internal" event={"ID":"0b9f4e0872eb36d1a9eb7f46a41c431b","Type":"ContainerStarted","Data":"6bd9b409e6956d054f404204b0d7bffd130dcddeb96c57af74fb2d9aaf1e36f4"}
Sep 18 17:50:18.942245 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:18.942219    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2mqnb" event={"ID":"65f6cbfa-00b5-4a5c-be75-28392cf8a1a8","Type":"ContainerStarted","Data":"525f189611696fb7eb598492da191e29d7542130ba8a3fca61794afa6caf526b"}
Sep 18 17:50:18.959982 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:18.959601    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-54.ec2.internal" podStartSLOduration=21.959588419 podStartE2EDuration="21.959588419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 17:49:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:50:18.959057393 +0000 UTC m=+23.586189048" watchObservedRunningTime="2026-09-18 17:50:18.959588419 +0000 UTC m=+23.586720039"
Sep 18 17:50:19.855307 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:19.855279    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mvtdm"
Sep 18 17:50:19.855488 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:19.855280    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bgwtf"
Sep 18 17:50:19.855488 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:19.855393    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mvtdm" podUID="7a3ae2d8-a30b-4c83-a53c-634fb07621d6"
Sep 18 17:50:19.855488 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:19.855291    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tmtsv"
Sep 18 17:50:19.855488 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:19.855469    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bgwtf" podUID="234003fa-8d5d-4108-8f62-859ea05eb9aa"
Sep 18 17:50:19.855682 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:19.855560    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tmtsv" podUID="45a7fd6c-fe96-4a1a-b3a5-2cb78c5435e8"
Sep 18 17:50:19.949081 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:19.949007    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xphc9" event={"ID":"6fb9873f-b1a1-4d93-910c-5c9d1417c1da","Type":"ContainerStarted","Data":"0edc8f7cc59597ec437020cc53ebfcc18d7f464718cf78e92114aee80f8f35f2"}
Sep 18 17:50:19.952065 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:19.952045    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkcgk_c5d026b7-af6b-4899-85f4-6554c0f7cd52/ovn-acl-logging/0.log"
Sep 18 17:50:19.952497 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:19.952476    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk" event={"ID":"c5d026b7-af6b-4899-85f4-6554c0f7cd52","Type":"ContainerStarted","Data":"3bee85b9b6cb209428b42e145c633b84c18569c031b866d9aa3c7dafd3fdb139"}
Sep 18 17:50:19.952597 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:19.952532    2561 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 18 17:50:19.972073 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:19.972036    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-2mqnb" podStartSLOduration=5.378173136 podStartE2EDuration="23.972024799s" totalImagesPullingTime="18.593851663s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 17:49:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:50:18.978074822 +0000 UTC m=+23.605206444" watchObservedRunningTime="2026-09-18 17:50:19.972024799 +0000 UTC m=+24.599156464"
Sep 18 17:50:19.972269 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:19.972249    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xphc9" podStartSLOduration=3.3942887 podStartE2EDuration="23.9722446s" totalImagesPullingTime="20.5779559s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 17:49:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:50:19.971901755 +0000 UTC m=+24.599033404" watchObservedRunningTime="2026-09-18 17:50:19.9722446 +0000 UTC m=+24.599376220"
Sep 18 17:50:21.855455 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:21.855427    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tmtsv"
Sep 18 17:50:21.855947 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:21.855428    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bgwtf"
Sep 18 17:50:21.855947 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:21.855545    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mvtdm"
Sep 18 17:50:21.855947 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:21.855551    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tmtsv" podUID="45a7fd6c-fe96-4a1a-b3a5-2cb78c5435e8"
Sep 18 17:50:21.855947 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:21.855620    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bgwtf" podUID="234003fa-8d5d-4108-8f62-859ea05eb9aa"
Sep 18 17:50:21.855947 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:21.855713    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mvtdm" podUID="7a3ae2d8-a30b-4c83-a53c-634fb07621d6"
Sep 18 17:50:23.855771 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:23.855595    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bgwtf"
Sep 18 17:50:23.856364 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:23.855657    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tmtsv"
Sep 18 17:50:23.856364 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:23.855880    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bgwtf" podUID="234003fa-8d5d-4108-8f62-859ea05eb9aa"
Sep 18 17:50:23.856364 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:23.855698    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mvtdm"
Sep 18 17:50:23.856364 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:23.855949    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tmtsv" podUID="45a7fd6c-fe96-4a1a-b3a5-2cb78c5435e8"
Sep 18 17:50:23.856364 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:23.856009    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mvtdm" podUID="7a3ae2d8-a30b-4c83-a53c-634fb07621d6"
Sep 18 17:50:23.892436 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:23.892417    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-k2xs7"
Sep 18 17:50:23.892558 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:23.892504    2561 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 18 17:50:23.892902 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:23.892887    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-k2xs7"
Sep 18 17:50:23.963839 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:23.963820    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkcgk_c5d026b7-af6b-4899-85f4-6554c0f7cd52/ovn-acl-logging/0.log"
Sep 18 17:50:23.964216 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:23.964198    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk" event={"ID":"c5d026b7-af6b-4899-85f4-6554c0f7cd52","Type":"ContainerStarted","Data":"1fa8c61ade118bf9872b3fd0054940d932ffd99afc665fc0d4f8be50b6ad8d9c"}
Sep 18 17:50:23.964471 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:23.964457    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk"
Sep 18 17:50:23.964569 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:23.964553    2561 scope.go:117] "RemoveContainer" containerID="541b6c4d5464aee51385a6b07445b88be06d1cb70c4dc240922c83829a079864"
Sep 18 17:50:23.966181 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:23.966153    2561 generic.go:356] "Generic (PLEG): container finished" podID="1dc25d25-efc8-477f-9a1e-5220f1d20345" containerID="64da0419aeeb6ffa361a4d8c7cf95c01ed17823b451ec7ebdaa5bf237736aff3" exitCode=0
Sep 18 17:50:23.966295 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:23.966239    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cqzh9" event={"ID":"1dc25d25-efc8-477f-9a1e-5220f1d20345","Type":"ContainerDied","Data":"64da0419aeeb6ffa361a4d8c7cf95c01ed17823b451ec7ebdaa5bf237736aff3"}
Sep 18 17:50:23.981941 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:23.981924    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk"
Sep 18 17:50:24.912392 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:24.912363    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mvtdm"]
Sep 18 17:50:24.913037 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:24.912505    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mvtdm"
Sep 18 17:50:24.913037 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:24.912591    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mvtdm" podUID="7a3ae2d8-a30b-4c83-a53c-634fb07621d6"
Sep 18 17:50:24.916524 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:24.916460    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tmtsv"]
Sep 18 17:50:24.916679 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:24.916633    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tmtsv"
Sep 18 17:50:24.916769 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:24.916748    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tmtsv" podUID="45a7fd6c-fe96-4a1a-b3a5-2cb78c5435e8"
Sep 18 17:50:24.917205 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:24.917183    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bgwtf"]
Sep 18 17:50:24.917295 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:24.917271    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bgwtf"
Sep 18 17:50:24.917356 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:24.917340    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bgwtf" podUID="234003fa-8d5d-4108-8f62-859ea05eb9aa"
Sep 18 17:50:24.970983 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:24.970959    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkcgk_c5d026b7-af6b-4899-85f4-6554c0f7cd52/ovn-acl-logging/0.log"
Sep 18 17:50:24.971331 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:24.971310    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk" event={"ID":"c5d026b7-af6b-4899-85f4-6554c0f7cd52","Type":"ContainerStarted","Data":"cef450349ed736aa9805bba7c0f4b1bc4f1edf3118a72e0dac91444204809283"}
Sep 18 17:50:24.980995 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:24.980976    2561 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 18 17:50:24.981287 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:24.981214    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk"
Sep 18 17:50:24.998759 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:24.998687    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk"
Sep 18 17:50:25.015742 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:25.015689    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk" podStartSLOduration=10.404694286 podStartE2EDuration="29.015674085s" totalImagesPullingTime="18.610979799s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 17:49:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:50:25.014152127 +0000 UTC m=+29.641283748" watchObservedRunningTime="2026-09-18 17:50:25.015674085 +0000 UTC m=+29.642805705"
Sep 18 17:50:25.956133 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:25.955942    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk"
Sep 18 17:50:25.975639 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:25.975611    2561 generic.go:356] "Generic (PLEG): container finished" podID="1dc25d25-efc8-477f-9a1e-5220f1d20345" containerID="cd77d06b9ec7fea75a1a088a61bfe833c85c05c99ecea1de699e531de13db74d" exitCode=0
Sep 18 17:50:25.975727 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:25.975686    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cqzh9" event={"ID":"1dc25d25-efc8-477f-9a1e-5220f1d20345","Type":"ContainerDied","Data":"cd77d06b9ec7fea75a1a088a61bfe833c85c05c99ecea1de699e531de13db74d"}
Sep 18 17:50:26.855481 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:26.855451    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tmtsv"
Sep 18 17:50:26.855594 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:26.855455    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bgwtf"
Sep 18 17:50:26.855594 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:26.855575    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tmtsv" podUID="45a7fd6c-fe96-4a1a-b3a5-2cb78c5435e8"
Sep 18 17:50:26.855684 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:26.855451    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mvtdm"
Sep 18 17:50:26.855684 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:26.855647    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bgwtf" podUID="234003fa-8d5d-4108-8f62-859ea05eb9aa"
Sep 18 17:50:26.855753 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:26.855701    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mvtdm" podUID="7a3ae2d8-a30b-4c83-a53c-634fb07621d6"
Sep 18 17:50:26.980208 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:26.980183    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cqzh9" event={"ID":"1dc25d25-efc8-477f-9a1e-5220f1d20345","Type":"ContainerStarted","Data":"71cbd095c332a7e29fe932859e63e81d5a6301fd7493847c2d70a405d4729d6a"}
Sep 18 17:50:27.984359 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:27.984329    2561 generic.go:356] "Generic (PLEG): container finished" podID="1dc25d25-efc8-477f-9a1e-5220f1d20345" containerID="71cbd095c332a7e29fe932859e63e81d5a6301fd7493847c2d70a405d4729d6a" exitCode=0
Sep 18 17:50:27.984712 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:27.984366    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cqzh9" event={"ID":"1dc25d25-efc8-477f-9a1e-5220f1d20345","Type":"ContainerDied","Data":"71cbd095c332a7e29fe932859e63e81d5a6301fd7493847c2d70a405d4729d6a"}
Sep 18 17:50:28.854855 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:28.854821    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bgwtf"
Sep 18 17:50:28.855035 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:28.854821    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tmtsv"
Sep 18 17:50:28.855035 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:28.854939    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bgwtf" podUID="234003fa-8d5d-4108-8f62-859ea05eb9aa"
Sep 18 17:50:28.855035 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:28.855009    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mvtdm"
Sep 18 17:50:28.855035 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:28.855012    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tmtsv" podUID="45a7fd6c-fe96-4a1a-b3a5-2cb78c5435e8"
Sep 18 17:50:28.855245 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:28.855098    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mvtdm" podUID="7a3ae2d8-a30b-4c83-a53c-634fb07621d6"
Sep 18 17:50:29.478289 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.478240    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45a7fd6c-fe96-4a1a-b3a5-2cb78c5435e8-metrics-certs\") pod \"network-metrics-daemon-tmtsv\" (UID: \"45a7fd6c-fe96-4a1a-b3a5-2cb78c5435e8\") " pod="openshift-multus/network-metrics-daemon-tmtsv"
Sep 18 17:50:29.478766 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:29.478391    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 17:50:29.478766 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:29.478466    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45a7fd6c-fe96-4a1a-b3a5-2cb78c5435e8-metrics-certs podName:45a7fd6c-fe96-4a1a-b3a5-2cb78c5435e8 nodeName:}" failed. No retries permitted until 2026-09-18 17:51:01.478447218 +0000 UTC m=+66.105578835 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/45a7fd6c-fe96-4a1a-b3a5-2cb78c5435e8-metrics-certs") pod "network-metrics-daemon-tmtsv" (UID: "45a7fd6c-fe96-4a1a-b3a5-2cb78c5435e8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 17:50:29.579372 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.579333    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scbjn\" (UniqueName: \"kubernetes.io/projected/234003fa-8d5d-4108-8f62-859ea05eb9aa-kube-api-access-scbjn\") pod \"network-check-target-bgwtf\" (UID: \"234003fa-8d5d-4108-8f62-859ea05eb9aa\") " pod="openshift-network-diagnostics/network-check-target-bgwtf"
Sep 18 17:50:29.579528 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.579385    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7a3ae2d8-a30b-4c83-a53c-634fb07621d6-original-pull-secret\") pod \"global-pull-secret-syncer-mvtdm\" (UID: \"7a3ae2d8-a30b-4c83-a53c-634fb07621d6\") " pod="kube-system/global-pull-secret-syncer-mvtdm"
Sep 18 17:50:29.579528 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:29.579512    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 17:50:29.579621 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:29.579551    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 17:50:29.579621 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:29.579571    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a3ae2d8-a30b-4c83-a53c-634fb07621d6-original-pull-secret podName:7a3ae2d8-a30b-4c83-a53c-634fb07621d6 nodeName:}" failed. No retries permitted until 2026-09-18 17:51:01.579553737 +0000 UTC m=+66.206685347 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7a3ae2d8-a30b-4c83-a53c-634fb07621d6-original-pull-secret") pod "global-pull-secret-syncer-mvtdm" (UID: "7a3ae2d8-a30b-4c83-a53c-634fb07621d6") : object "kube-system"/"original-pull-secret" not registered
Sep 18 17:50:29.579621 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:29.579576    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 17:50:29.579621 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:29.579591    2561 projected.go:196] Error preparing data for projected volume kube-api-access-scbjn for pod openshift-network-diagnostics/network-check-target-bgwtf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 17:50:29.579786 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:29.579639    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/234003fa-8d5d-4108-8f62-859ea05eb9aa-kube-api-access-scbjn podName:234003fa-8d5d-4108-8f62-859ea05eb9aa nodeName:}" failed. No retries permitted until 2026-09-18 17:51:01.579623715 +0000 UTC m=+66.206755318 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-scbjn" (UniqueName: "kubernetes.io/projected/234003fa-8d5d-4108-8f62-859ea05eb9aa-kube-api-access-scbjn") pod "network-check-target-bgwtf" (UID: "234003fa-8d5d-4108-8f62-859ea05eb9aa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 17:50:29.629165 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.629134    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-54.ec2.internal" event="NodeReady"
Sep 18 17:50:29.629373 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.629307    2561 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 18 17:50:29.674081 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.674053    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-vzcw7"]
Sep 18 17:50:29.675932 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.675914    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-vzcw7"
Sep 18 17:50:29.678716 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.678674    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-qsqqj"]
Sep 18 17:50:29.679821 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.679800    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 18 17:50:29.679921 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.679858    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 18 17:50:29.679921 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.679879    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-k966h\""
Sep 18 17:50:29.680695 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.680676    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-78c879b97f-xtp8m"]
Sep 18 17:50:29.681335 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.681280    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-qsqqj"
Sep 18 17:50:29.682367 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.682349    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7dc96485b6-66p6f"]
Sep 18 17:50:29.682497 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.682479    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78c879b97f-xtp8m"
Sep 18 17:50:29.684131 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.684095    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-dx752"]
Sep 18 17:50:29.684228 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.684198    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7dc96485b6-66p6f"
Sep 18 17:50:29.685743 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.685657    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 18 17:50:29.685936 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.685839    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xjzc9"]
Sep 18 17:50:29.686066 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.685947    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 18 17:50:29.686066 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.685968    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-dx752"
Sep 18 17:50:29.686833 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.686444    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xssnq\""
Sep 18 17:50:29.686833 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.686517    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 18 17:50:29.686833 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.686779    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 18 17:50:29.687091 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.687070    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-t2gd7\""
Sep 18 17:50:29.687184 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.687089    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 18 17:50:29.687242 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.687148    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 18 17:50:29.688719 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.688699    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xjzc9"
Sep 18 17:50:29.689398 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.689379    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 18 17:50:29.690272 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.690244    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 18 17:50:29.690272 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.690247    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 18 17:50:29.691520 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.691500    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-vzcw7"]
Sep 18 17:50:29.694010 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.693452    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 18 17:50:29.694010 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.693585    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 18 17:50:29.694010 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.693674    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 18 17:50:29.694010 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.693692    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 18 17:50:29.694010 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.693841    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 18 17:50:29.694010 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.693917    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 18 17:50:29.698257 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.695761    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 18 17:50:29.698257 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.695955    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 18 17:50:29.698257 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.696092    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 18 17:50:29.698257 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.696665    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-krc6s\""
Sep 18 17:50:29.698257 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.696668    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 18 17:50:29.698257 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.697384    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-wffwh\""
Sep 18 17:50:29.698613 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.698507    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 18 17:50:29.701600 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.699289    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 18 17:50:29.701600 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.699583    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-tzxvx\""
Sep 18 17:50:29.701600 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.700315    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-qsqqj"]
Sep 18 17:50:29.703281 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.703259    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 18 17:50:29.707783 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.707761    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-qd9mb"]
Sep 18 17:50:29.710074 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.710043    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-5kjlz"]
Sep 18 17:50:29.710276 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.710260    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-qd9mb"
Sep 18 17:50:29.712043 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.712022    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7dc96485b6-66p6f"]
Sep 18 17:50:29.712213 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.712194    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-5kjlz"
Sep 18 17:50:29.712834 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.712815    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-9ttqt"]
Sep 18 17:50:29.714978 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.714929    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-4hdg8"]
Sep 18 17:50:29.715101 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.715047    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-9ttqt"
Sep 18 17:50:29.716108 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.716057    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 18 17:50:29.716434 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.716418    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 18 17:50:29.716851 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.716825    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-78c879b97f-xtp8m"]
Sep 18 17:50:29.716932 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.716907    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-4hdg8"
Sep 18 17:50:29.716932 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.716849    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 18 17:50:29.717766 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.717628    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 18 17:50:29.717766 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.717754    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 18 17:50:29.717916 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.717893    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-j8h26\""
Sep 18 17:50:29.717975 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.717936    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 17:50:29.717975 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.717964    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 18 17:50:29.718174 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.718158    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 18 17:50:29.718174 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.718096    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 17:50:29.718294 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.718191    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-qtjrw\""
Sep 18 17:50:29.722049 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.721909    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 18 17:50:29.722195 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.722179    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-97s7k\""
Sep 18 17:50:29.722301 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.722277    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 18 17:50:29.722384 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.722308    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 18 17:50:29.723029 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.723012    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 18 17:50:29.723029 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.723027    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-zhrt4\""
Sep 18 17:50:29.730082 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.725845    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-qd9mb"]
Sep 18 17:50:29.730082 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.726998    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-kw7df"]
Sep 18 17:50:29.730082 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.728700    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 18 17:50:29.730082 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.729049    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-9hjjd"]
Sep 18 17:50:29.730328 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.730213    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kw7df"
Sep 18 17:50:29.731577 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.731559    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-5kjlz"]
Sep 18 17:50:29.731703 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.731686    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9hjjd"
Sep 18 17:50:29.732302 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.732284    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-9ttqt"]
Sep 18 17:50:29.733674 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.733590    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 18 17:50:29.733674 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.733650    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 18 17:50:29.734014 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.733996    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 18 17:50:29.734340 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.734323    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 18 17:50:29.734418 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.734385    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-hhc9g\""
Sep 18 17:50:29.735787 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.735768    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 18 17:50:29.736247 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.735815    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-s9cmf\""
Sep 18 17:50:29.736247 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.736082    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 18 17:50:29.746932 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.746755    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-dx752"]
Sep 18 17:50:29.747204 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.747161    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-4hdg8"]
Sep 18 17:50:29.748105 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.748089    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xjzc9"]
Sep 18 17:50:29.749869 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.749851    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9hjjd"]
Sep 18 17:50:29.750794 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.750773    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kw7df"]
Sep 18 17:50:29.780203 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.780177    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/20117f3e-e50b-4b03-aa48-604fc42d1737-default-certificate\") pod \"router-default-7dc96485b6-66p6f\" (UID: \"20117f3e-e50b-4b03-aa48-604fc42d1737\") " pod="openshift-ingress/router-default-7dc96485b6-66p6f"
Sep 18 17:50:29.780306 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.780214    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6xxc\" (UniqueName: \"kubernetes.io/projected/26ece3e7-900c-4efe-b497-653cf58b5e60-kube-api-access-d6xxc\") pod \"kube-storage-version-migrator-operator-55889f7b47-dx752\" (UID: \"26ece3e7-900c-4efe-b497-653cf58b5e60\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-dx752"
Sep 18 17:50:29.780306 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.780253    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmmvv\" (UniqueName: \"kubernetes.io/projected/a1e4f37f-b0ba-469f-8923-d0c9d38e619b-kube-api-access-mmmvv\") pod \"cluster-monitoring-operator-847b8bd8d7-xjzc9\" (UID: \"a1e4f37f-b0ba-469f-8923-d0c9d38e619b\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xjzc9"
Sep 18 17:50:29.780417 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.780303    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20117f3e-e50b-4b03-aa48-604fc42d1737-service-ca-bundle\") pod \"router-default-7dc96485b6-66p6f\" (UID: \"20117f3e-e50b-4b03-aa48-604fc42d1737\") " pod="openshift-ingress/router-default-7dc96485b6-66p6f"
Sep 18 17:50:29.780417 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.780323    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/20117f3e-e50b-4b03-aa48-604fc42d1737-stats-auth\") pod \"router-default-7dc96485b6-66p6f\" (UID: \"20117f3e-e50b-4b03-aa48-604fc42d1737\") " pod="openshift-ingress/router-default-7dc96485b6-66p6f"
Sep 18 17:50:29.780417 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.780364    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/60520815-25c1-4a4e-a036-c3fb968adbaf-config\") pod \"service-ca-operator-57cdc4b768-qsqqj\" (UID: \"60520815-25c1-4a4e-a036-c3fb968adbaf\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-qsqqj"
Sep 18 17:50:29.780417 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.780395    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/60520815-25c1-4a4e-a036-c3fb968adbaf-serving-cert\") pod \"service-ca-operator-57cdc4b768-qsqqj\" (UID: \"60520815-25c1-4a4e-a036-c3fb968adbaf\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-qsqqj"
Sep 18 17:50:29.780610 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.780431    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1096ace1-2554-43bc-bf2e-5e6d09e1c48f-ca-trust-extracted\") pod \"image-registry-78c879b97f-xtp8m\" (UID: \"1096ace1-2554-43bc-bf2e-5e6d09e1c48f\") " pod="openshift-image-registry/image-registry-78c879b97f-xtp8m"
Sep 18 17:50:29.780610 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.780457    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1096ace1-2554-43bc-bf2e-5e6d09e1c48f-registry-certificates\") pod \"image-registry-78c879b97f-xtp8m\" (UID: \"1096ace1-2554-43bc-bf2e-5e6d09e1c48f\") " pod="openshift-image-registry/image-registry-78c879b97f-xtp8m"
Sep 18 17:50:29.780610 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.780487    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r557q\" (UniqueName: \"kubernetes.io/projected/20117f3e-e50b-4b03-aa48-604fc42d1737-kube-api-access-r557q\") pod \"router-default-7dc96485b6-66p6f\" (UID: \"20117f3e-e50b-4b03-aa48-604fc42d1737\") " pod="openshift-ingress/router-default-7dc96485b6-66p6f"
Sep 18 17:50:29.780610 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.780503    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1096ace1-2554-43bc-bf2e-5e6d09e1c48f-installation-pull-secrets\") pod \"image-registry-78c879b97f-xtp8m\" (UID: \"1096ace1-2554-43bc-bf2e-5e6d09e1c48f\") " pod="openshift-image-registry/image-registry-78c879b97f-xtp8m"
Sep 18 17:50:29.780610 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.780523    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/26ece3e7-900c-4efe-b497-653cf58b5e60-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-dx752\" (UID: \"26ece3e7-900c-4efe-b497-653cf58b5e60\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-dx752"
Sep 18 17:50:29.780839 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.780612    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1096ace1-2554-43bc-bf2e-5e6d09e1c48f-trusted-ca\") pod \"image-registry-78c879b97f-xtp8m\" (UID: \"1096ace1-2554-43bc-bf2e-5e6d09e1c48f\") " pod="openshift-image-registry/image-registry-78c879b97f-xtp8m"
Sep 18 17:50:29.780839 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.780634    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ltk8\" (UniqueName: \"kubernetes.io/projected/1096ace1-2554-43bc-bf2e-5e6d09e1c48f-kube-api-access-8ltk8\") pod \"image-registry-78c879b97f-xtp8m\" (UID: \"1096ace1-2554-43bc-bf2e-5e6d09e1c48f\") " pod="openshift-image-registry/image-registry-78c879b97f-xtp8m"
Sep 18 17:50:29.780839 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.780654    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/26ece3e7-900c-4efe-b497-653cf58b5e60-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-dx752\" (UID: \"26ece3e7-900c-4efe-b497-653cf58b5e60\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-dx752"
Sep 18 17:50:29.780839 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.780686    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1096ace1-2554-43bc-bf2e-5e6d09e1c48f-bound-sa-token\") pod \"image-registry-78c879b97f-xtp8m\" (UID: \"1096ace1-2554-43bc-bf2e-5e6d09e1c48f\") " pod="openshift-image-registry/image-registry-78c879b97f-xtp8m"
Sep 18 17:50:29.780839 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.780704    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20117f3e-e50b-4b03-aa48-604fc42d1737-metrics-certs\") pod \"router-default-7dc96485b6-66p6f\" (UID: \"20117f3e-e50b-4b03-aa48-604fc42d1737\") " pod="openshift-ingress/router-default-7dc96485b6-66p6f"
Sep 18 17:50:29.780839 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.780738    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a1e4f37f-b0ba-469f-8923-d0c9d38e619b-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-xjzc9\" (UID: \"a1e4f37f-b0ba-469f-8923-d0c9d38e619b\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xjzc9"
Sep 18 17:50:29.780839 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.780756    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8vrr\" (UniqueName: \"kubernetes.io/projected/6d0c7ec9-9691-4341-afd6-ba415fdb1326-kube-api-access-h8vrr\") pod \"volume-data-source-validator-5c4db8bdf9-vzcw7\" (UID: \"6d0c7ec9-9691-4341-afd6-ba415fdb1326\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-vzcw7"
Sep 18 17:50:29.780839 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.780771    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpnbv\" (UniqueName: \"kubernetes.io/projected/60520815-25c1-4a4e-a036-c3fb968adbaf-kube-api-access-kpnbv\") pod \"service-ca-operator-57cdc4b768-qsqqj\" (UID: \"60520815-25c1-4a4e-a036-c3fb968adbaf\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-qsqqj"
Sep 18 17:50:29.780839 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.780802    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1096ace1-2554-43bc-bf2e-5e6d09e1c48f-image-registry-private-configuration\") pod \"image-registry-78c879b97f-xtp8m\" (UID: \"1096ace1-2554-43bc-bf2e-5e6d09e1c48f\") " pod="openshift-image-registry/image-registry-78c879b97f-xtp8m"
Sep 18 17:50:29.780839 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.780832    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1096ace1-2554-43bc-bf2e-5e6d09e1c48f-registry-tls\") pod \"image-registry-78c879b97f-xtp8m\" (UID: \"1096ace1-2554-43bc-bf2e-5e6d09e1c48f\") " pod="openshift-image-registry/image-registry-78c879b97f-xtp8m"
Sep 18 17:50:29.781456 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.780872    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a1e4f37f-b0ba-469f-8923-d0c9d38e619b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-xjzc9\" (UID: \"a1e4f37f-b0ba-469f-8923-d0c9d38e619b\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xjzc9"
Sep 18 17:50:29.881425 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.881354    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a1e4f37f-b0ba-469f-8923-d0c9d38e619b-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-xjzc9\" (UID: \"a1e4f37f-b0ba-469f-8923-d0c9d38e619b\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xjzc9"
Sep 18 17:50:29.881425 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.881395    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8vrr\" (UniqueName: \"kubernetes.io/projected/6d0c7ec9-9691-4341-afd6-ba415fdb1326-kube-api-access-h8vrr\") pod \"volume-data-source-validator-5c4db8bdf9-vzcw7\" (UID: \"6d0c7ec9-9691-4341-afd6-ba415fdb1326\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-vzcw7"
Sep 18 17:50:29.881647 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.881438    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn24x\" (UniqueName: \"kubernetes.io/projected/60a031b6-31f6-47a2-859c-bfd960f0b2c7-kube-api-access-fn24x\") pod \"dns-default-9hjjd\" (UID: \"60a031b6-31f6-47a2-859c-bfd960f0b2c7\") " pod="openshift-dns/dns-default-9hjjd"
Sep 18 17:50:29.881647 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.881460    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1286716b-4028-484e-b51d-58eb4b9f7305-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-qd9mb\" (UID: \"1286716b-4028-484e-b51d-58eb4b9f7305\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-qd9mb"
Sep 18 17:50:29.881647 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.881589    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a1e4f37f-b0ba-469f-8923-d0c9d38e619b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-xjzc9\" (UID: \"a1e4f37f-b0ba-469f-8923-d0c9d38e619b\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xjzc9"
Sep 18 17:50:29.881647 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.881641    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/20117f3e-e50b-4b03-aa48-604fc42d1737-default-certificate\") pod \"router-default-7dc96485b6-66p6f\" (UID: \"20117f3e-e50b-4b03-aa48-604fc42d1737\") " pod="openshift-ingress/router-default-7dc96485b6-66p6f"
Sep 18 17:50:29.881838 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.881665    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgmbt\" (UniqueName: \"kubernetes.io/projected/2e2c3ab1-da7b-40e3-8d33-4d7ec5980634-kube-api-access-pgmbt\") pod \"insights-operator-6f9c644dd9-4hdg8\" (UID: \"2e2c3ab1-da7b-40e3-8d33-4d7ec5980634\") " pod="openshift-insights/insights-operator-6f9c644dd9-4hdg8"
Sep 18 17:50:29.881838 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.881682    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/60a031b6-31f6-47a2-859c-bfd960f0b2c7-config-volume\") pod \"dns-default-9hjjd\" (UID: \"60a031b6-31f6-47a2-859c-bfd960f0b2c7\") " pod="openshift-dns/dns-default-9hjjd"
Sep 18 17:50:29.881838 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:29.881697    2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 17:50:29.881838 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.881719    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6xxc\" (UniqueName: \"kubernetes.io/projected/26ece3e7-900c-4efe-b497-653cf58b5e60-kube-api-access-d6xxc\") pod \"kube-storage-version-migrator-operator-55889f7b47-dx752\" (UID: \"26ece3e7-900c-4efe-b497-653cf58b5e60\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-dx752"
Sep 18 17:50:29.881838 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:29.881780    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1e4f37f-b0ba-469f-8923-d0c9d38e619b-cluster-monitoring-operator-tls podName:a1e4f37f-b0ba-469f-8923-d0c9d38e619b nodeName:}" failed. No retries permitted until 2026-09-18 17:50:30.381761121 +0000 UTC m=+35.008892743 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a1e4f37f-b0ba-469f-8923-d0c9d38e619b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-xjzc9" (UID: "a1e4f37f-b0ba-469f-8923-d0c9d38e619b") : secret "cluster-monitoring-operator-tls" not found
Sep 18 17:50:29.881838 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.881805    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/75f2a1f7-95a7-462e-9ea7-23c51dd29599-config\") pod \"console-operator-fd4bb46c-5kjlz\" (UID: \"75f2a1f7-95a7-462e-9ea7-23c51dd29599\") " pod="openshift-console-operator/console-operator-fd4bb46c-5kjlz"
Sep 18 17:50:29.882033 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.881845    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1096ace1-2554-43bc-bf2e-5e6d09e1c48f-registry-certificates\") pod \"image-registry-78c879b97f-xtp8m\" (UID: \"1096ace1-2554-43bc-bf2e-5e6d09e1c48f\") " pod="openshift-image-registry/image-registry-78c879b97f-xtp8m"
Sep 18 17:50:29.882033 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.881874    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/20117f3e-e50b-4b03-aa48-604fc42d1737-stats-auth\") pod \"router-default-7dc96485b6-66p6f\" (UID: \"20117f3e-e50b-4b03-aa48-604fc42d1737\") " pod="openshift-ingress/router-default-7dc96485b6-66p6f"
Sep 18 17:50:29.882033 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.881901    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/60520815-25c1-4a4e-a036-c3fb968adbaf-config\") pod \"service-ca-operator-57cdc4b768-qsqqj\" (UID: \"60520815-25c1-4a4e-a036-c3fb968adbaf\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-qsqqj"
Sep 18 17:50:29.882033 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.881927    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/26ece3e7-900c-4efe-b497-653cf58b5e60-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-dx752\" (UID: \"26ece3e7-900c-4efe-b497-653cf58b5e60\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-dx752"
Sep 18 17:50:29.882033 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.881960    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7pc2\" (UniqueName: \"kubernetes.io/projected/75f2a1f7-95a7-462e-9ea7-23c51dd29599-kube-api-access-k7pc2\") pod \"console-operator-fd4bb46c-5kjlz\" (UID: \"75f2a1f7-95a7-462e-9ea7-23c51dd29599\") " pod="openshift-console-operator/console-operator-fd4bb46c-5kjlz"
Sep 18 17:50:29.882033 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.881994    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20117f3e-e50b-4b03-aa48-604fc42d1737-metrics-certs\") pod \"router-default-7dc96485b6-66p6f\" (UID: \"20117f3e-e50b-4b03-aa48-604fc42d1737\") " pod="openshift-ingress/router-default-7dc96485b6-66p6f"
Sep 18 17:50:29.882033 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.882019    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ac028d85-f1a2-4c5a-91ad-fb7141335c07-cert\") pod \"ingress-canary-kw7df\" (UID: \"ac028d85-f1a2-4c5a-91ad-fb7141335c07\") " pod="openshift-ingress-canary/ingress-canary-kw7df"
Sep 18 17:50:29.882336 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.882049    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpnbv\" (UniqueName: \"kubernetes.io/projected/60520815-25c1-4a4e-a036-c3fb968adbaf-kube-api-access-kpnbv\") pod \"service-ca-operator-57cdc4b768-qsqqj\" (UID: \"60520815-25c1-4a4e-a036-c3fb968adbaf\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-qsqqj"
Sep 18 17:50:29.882336 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.882081    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1096ace1-2554-43bc-bf2e-5e6d09e1c48f-image-registry-private-configuration\") pod \"image-registry-78c879b97f-xtp8m\" (UID: \"1096ace1-2554-43bc-bf2e-5e6d09e1c48f\") " pod="openshift-image-registry/image-registry-78c879b97f-xtp8m"
Sep 18 17:50:29.882336 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.882109    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1096ace1-2554-43bc-bf2e-5e6d09e1c48f-registry-tls\") pod \"image-registry-78c879b97f-xtp8m\" (UID: \"1096ace1-2554-43bc-bf2e-5e6d09e1c48f\") " pod="openshift-image-registry/image-registry-78c879b97f-xtp8m"
Sep 18 17:50:29.882336 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.882152    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znz5k\" (UniqueName: \"kubernetes.io/projected/ac028d85-f1a2-4c5a-91ad-fb7141335c07-kube-api-access-znz5k\") pod \"ingress-canary-kw7df\" (UID: \"ac028d85-f1a2-4c5a-91ad-fb7141335c07\") " pod="openshift-ingress-canary/ingress-canary-kw7df"
Sep 18 17:50:29.882336 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.882152    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a1e4f37f-b0ba-469f-8923-d0c9d38e619b-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-xjzc9\" (UID: \"a1e4f37f-b0ba-469f-8923-d0c9d38e619b\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xjzc9"
Sep 18 17:50:29.882336 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:29.882246    2561 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 17:50:29.882336 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:29.882307    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20117f3e-e50b-4b03-aa48-604fc42d1737-metrics-certs podName:20117f3e-e50b-4b03-aa48-604fc42d1737 nodeName:}" failed. No retries permitted until 2026-09-18 17:50:30.382292179 +0000 UTC m=+35.009423789 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/20117f3e-e50b-4b03-aa48-604fc42d1737-metrics-certs") pod "router-default-7dc96485b6-66p6f" (UID: "20117f3e-e50b-4b03-aa48-604fc42d1737") : secret "router-metrics-certs-default" not found
Sep 18 17:50:29.882675 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.882340    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6n6f\" (UniqueName: \"kubernetes.io/projected/1286716b-4028-484e-b51d-58eb4b9f7305-kube-api-access-g6n6f\") pod \"cluster-samples-operator-685d7d68b5-qd9mb\" (UID: \"1286716b-4028-484e-b51d-58eb4b9f7305\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-qd9mb"
Sep 18 17:50:29.882675 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.882392    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/60520815-25c1-4a4e-a036-c3fb968adbaf-serving-cert\") pod \"service-ca-operator-57cdc4b768-qsqqj\" (UID: \"60520815-25c1-4a4e-a036-c3fb968adbaf\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-qsqqj"
Sep 18 17:50:29.882675 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:29.882402    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 17:50:29.882675 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:29.882414    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78c879b97f-xtp8m: secret "image-registry-tls" not found
Sep 18 17:50:29.882675 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.882422    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2e2c3ab1-da7b-40e3-8d33-4d7ec5980634-snapshots\") pod \"insights-operator-6f9c644dd9-4hdg8\" (UID: \"2e2c3ab1-da7b-40e3-8d33-4d7ec5980634\") " pod="openshift-insights/insights-operator-6f9c644dd9-4hdg8"
Sep 18 17:50:29.882675 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:29.882452    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1096ace1-2554-43bc-bf2e-5e6d09e1c48f-registry-tls podName:1096ace1-2554-43bc-bf2e-5e6d09e1c48f nodeName:}" failed. No retries permitted until 2026-09-18 17:50:30.382438192 +0000 UTC m=+35.009569797 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1096ace1-2554-43bc-bf2e-5e6d09e1c48f-registry-tls") pod "image-registry-78c879b97f-xtp8m" (UID: "1096ace1-2554-43bc-bf2e-5e6d09e1c48f") : secret "image-registry-tls" not found
Sep 18 17:50:29.882675 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.882470    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e2c3ab1-da7b-40e3-8d33-4d7ec5980634-tmp\") pod \"insights-operator-6f9c644dd9-4hdg8\" (UID: \"2e2c3ab1-da7b-40e3-8d33-4d7ec5980634\") " pod="openshift-insights/insights-operator-6f9c644dd9-4hdg8"
Sep 18 17:50:29.882675 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.882490    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1096ace1-2554-43bc-bf2e-5e6d09e1c48f-registry-certificates\") pod \"image-registry-78c879b97f-xtp8m\" (UID: \"1096ace1-2554-43bc-bf2e-5e6d09e1c48f\") " pod="openshift-image-registry/image-registry-78c879b97f-xtp8m"
Sep 18 17:50:29.882675 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.882547    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/26ece3e7-900c-4efe-b497-653cf58b5e60-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-dx752\" (UID: \"26ece3e7-900c-4efe-b497-653cf58b5e60\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-dx752"
Sep 18 17:50:29.882675 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.882559    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmmvv\" (UniqueName: \"kubernetes.io/projected/a1e4f37f-b0ba-469f-8923-d0c9d38e619b-kube-api-access-mmmvv\") pod \"cluster-monitoring-operator-847b8bd8d7-xjzc9\" (UID: \"a1e4f37f-b0ba-469f-8923-d0c9d38e619b\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xjzc9"
Sep 18 17:50:29.882675 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.882590    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/26ece3e7-900c-4efe-b497-653cf58b5e60-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-dx752\" (UID: \"26ece3e7-900c-4efe-b497-653cf58b5e60\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-dx752"
Sep 18 17:50:29.883203 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.882685    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20117f3e-e50b-4b03-aa48-604fc42d1737-service-ca-bundle\") pod \"router-default-7dc96485b6-66p6f\" (UID: \"20117f3e-e50b-4b03-aa48-604fc42d1737\") " pod="openshift-ingress/router-default-7dc96485b6-66p6f"
Sep 18 17:50:29.883203 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.882722    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2e2c3ab1-da7b-40e3-8d33-4d7ec5980634-serving-cert\") pod \"insights-operator-6f9c644dd9-4hdg8\" (UID: \"2e2c3ab1-da7b-40e3-8d33-4d7ec5980634\") " pod="openshift-insights/insights-operator-6f9c644dd9-4hdg8"
Sep 18 17:50:29.883203 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.882772    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/60a031b6-31f6-47a2-859c-bfd960f0b2c7-metrics-tls\") pod \"dns-default-9hjjd\" (UID: \"60a031b6-31f6-47a2-859c-bfd960f0b2c7\") " pod="openshift-dns/dns-default-9hjjd"
Sep 18 17:50:29.883203 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:29.882872    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/20117f3e-e50b-4b03-aa48-604fc42d1737-service-ca-bundle podName:20117f3e-e50b-4b03-aa48-604fc42d1737 nodeName:}" failed. No retries permitted until 2026-09-18 17:50:30.382851342 +0000 UTC m=+35.009982957 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/20117f3e-e50b-4b03-aa48-604fc42d1737-service-ca-bundle") pod "router-default-7dc96485b6-66p6f" (UID: "20117f3e-e50b-4b03-aa48-604fc42d1737") : configmap references non-existent config key: service-ca.crt
Sep 18 17:50:29.883203 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.882952    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1096ace1-2554-43bc-bf2e-5e6d09e1c48f-ca-trust-extracted\") pod \"image-registry-78c879b97f-xtp8m\" (UID: \"1096ace1-2554-43bc-bf2e-5e6d09e1c48f\") " pod="openshift-image-registry/image-registry-78c879b97f-xtp8m"
Sep 18 17:50:29.883203 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.882982    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1096ace1-2554-43bc-bf2e-5e6d09e1c48f-installation-pull-secrets\") pod \"image-registry-78c879b97f-xtp8m\" (UID: \"1096ace1-2554-43bc-bf2e-5e6d09e1c48f\") " pod="openshift-image-registry/image-registry-78c879b97f-xtp8m"
Sep 18 17:50:29.883203 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.883008    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60a031b6-31f6-47a2-859c-bfd960f0b2c7-tmp-dir\") pod \"dns-default-9hjjd\" (UID: \"60a031b6-31f6-47a2-859c-bfd960f0b2c7\") " pod="openshift-dns/dns-default-9hjjd"
Sep 18 17:50:29.883203 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.883019    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/60520815-25c1-4a4e-a036-c3fb968adbaf-config\") pod \"service-ca-operator-57cdc4b768-qsqqj\" (UID: \"60520815-25c1-4a4e-a036-c3fb968adbaf\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-qsqqj"
Sep 18 17:50:29.883203 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.883050    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r557q\" (UniqueName: \"kubernetes.io/projected/20117f3e-e50b-4b03-aa48-604fc42d1737-kube-api-access-r557q\") pod \"router-default-7dc96485b6-66p6f\" (UID: \"20117f3e-e50b-4b03-aa48-604fc42d1737\") " pod="openshift-ingress/router-default-7dc96485b6-66p6f"
Sep 18 17:50:29.883203 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.883078    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/75f2a1f7-95a7-462e-9ea7-23c51dd29599-trusted-ca\") pod \"console-operator-fd4bb46c-5kjlz\" (UID: \"75f2a1f7-95a7-462e-9ea7-23c51dd29599\") " pod="openshift-console-operator/console-operator-fd4bb46c-5kjlz"
Sep 18 17:50:29.883203 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.883110    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1096ace1-2554-43bc-bf2e-5e6d09e1c48f-trusted-ca\") pod \"image-registry-78c879b97f-xtp8m\" (UID: \"1096ace1-2554-43bc-bf2e-5e6d09e1c48f\") " pod="openshift-image-registry/image-registry-78c879b97f-xtp8m"
Sep 18 17:50:29.883203 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.883155    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ltk8\" (UniqueName: \"kubernetes.io/projected/1096ace1-2554-43bc-bf2e-5e6d09e1c48f-kube-api-access-8ltk8\") pod \"image-registry-78c879b97f-xtp8m\" (UID: \"1096ace1-2554-43bc-bf2e-5e6d09e1c48f\") " pod="openshift-image-registry/image-registry-78c879b97f-xtp8m"
Sep 18 17:50:29.883203 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.883195    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e2c3ab1-da7b-40e3-8d33-4d7ec5980634-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-4hdg8\" (UID: \"2e2c3ab1-da7b-40e3-8d33-4d7ec5980634\") " pod="openshift-insights/insights-operator-6f9c644dd9-4hdg8"
Sep 18 17:50:29.883817 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.883229    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/75f2a1f7-95a7-462e-9ea7-23c51dd29599-serving-cert\") pod \"console-operator-fd4bb46c-5kjlz\" (UID: \"75f2a1f7-95a7-462e-9ea7-23c51dd29599\") " pod="openshift-console-operator/console-operator-fd4bb46c-5kjlz"
Sep 18 17:50:29.883817 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.883257    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1096ace1-2554-43bc-bf2e-5e6d09e1c48f-bound-sa-token\") pod \"image-registry-78c879b97f-xtp8m\" (UID: \"1096ace1-2554-43bc-bf2e-5e6d09e1c48f\") " pod="openshift-image-registry/image-registry-78c879b97f-xtp8m"
Sep 18 17:50:29.883817 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.883267    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1096ace1-2554-43bc-bf2e-5e6d09e1c48f-ca-trust-extracted\") pod \"image-registry-78c879b97f-xtp8m\" (UID: \"1096ace1-2554-43bc-bf2e-5e6d09e1c48f\") " pod="openshift-image-registry/image-registry-78c879b97f-xtp8m"
Sep 18 17:50:29.883817 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.883282    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e2c3ab1-da7b-40e3-8d33-4d7ec5980634-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-4hdg8\" (UID: \"2e2c3ab1-da7b-40e3-8d33-4d7ec5980634\") " pod="openshift-insights/insights-operator-6f9c644dd9-4hdg8"
Sep 18 17:50:29.883817 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.883331    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvfvh\" (UniqueName: \"kubernetes.io/projected/6586ff5a-f123-4692-a672-1844f9ce5a11-kube-api-access-rvfvh\") pod \"network-check-source-7cbc45fddc-9ttqt\" (UID: \"6586ff5a-f123-4692-a672-1844f9ce5a11\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-9ttqt"
Sep 18 17:50:29.884422 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.884401    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1096ace1-2554-43bc-bf2e-5e6d09e1c48f-trusted-ca\") pod \"image-registry-78c879b97f-xtp8m\" (UID: \"1096ace1-2554-43bc-bf2e-5e6d09e1c48f\") " pod="openshift-image-registry/image-registry-78c879b97f-xtp8m"
Sep 18 17:50:29.886947 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.886923    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/26ece3e7-900c-4efe-b497-653cf58b5e60-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-dx752\" (UID: \"26ece3e7-900c-4efe-b497-653cf58b5e60\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-dx752"
Sep 18 17:50:29.887102 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.887083    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/20117f3e-e50b-4b03-aa48-604fc42d1737-default-certificate\") pod \"router-default-7dc96485b6-66p6f\" (UID: \"20117f3e-e50b-4b03-aa48-604fc42d1737\") " pod="openshift-ingress/router-default-7dc96485b6-66p6f"
Sep 18 17:50:29.887102 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.887093    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1096ace1-2554-43bc-bf2e-5e6d09e1c48f-image-registry-private-configuration\") pod \"image-registry-78c879b97f-xtp8m\" (UID: \"1096ace1-2554-43bc-bf2e-5e6d09e1c48f\") " pod="openshift-image-registry/image-registry-78c879b97f-xtp8m"
Sep 18 17:50:29.887512 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.887489    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/20117f3e-e50b-4b03-aa48-604fc42d1737-stats-auth\") pod \"router-default-7dc96485b6-66p6f\" (UID: \"20117f3e-e50b-4b03-aa48-604fc42d1737\") " pod="openshift-ingress/router-default-7dc96485b6-66p6f"
Sep 18 17:50:29.887645 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.887620    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/60520815-25c1-4a4e-a036-c3fb968adbaf-serving-cert\") pod \"service-ca-operator-57cdc4b768-qsqqj\" (UID: \"60520815-25c1-4a4e-a036-c3fb968adbaf\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-qsqqj"
Sep 18 17:50:29.888545 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.888529    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1096ace1-2554-43bc-bf2e-5e6d09e1c48f-installation-pull-secrets\") pod \"image-registry-78c879b97f-xtp8m\" (UID: \"1096ace1-2554-43bc-bf2e-5e6d09e1c48f\") " pod="openshift-image-registry/image-registry-78c879b97f-xtp8m"
Sep 18 17:50:29.892323 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.892298    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6xxc\" (UniqueName: \"kubernetes.io/projected/26ece3e7-900c-4efe-b497-653cf58b5e60-kube-api-access-d6xxc\") pod \"kube-storage-version-migrator-operator-55889f7b47-dx752\" (UID: \"26ece3e7-900c-4efe-b497-653cf58b5e60\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-dx752"
Sep 18 17:50:29.893108 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.893082    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmmvv\" (UniqueName: \"kubernetes.io/projected/a1e4f37f-b0ba-469f-8923-d0c9d38e619b-kube-api-access-mmmvv\") pod \"cluster-monitoring-operator-847b8bd8d7-xjzc9\" (UID: \"a1e4f37f-b0ba-469f-8923-d0c9d38e619b\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xjzc9"
Sep 18 17:50:29.893209 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.893111    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1096ace1-2554-43bc-bf2e-5e6d09e1c48f-bound-sa-token\") pod \"image-registry-78c879b97f-xtp8m\" (UID: \"1096ace1-2554-43bc-bf2e-5e6d09e1c48f\") " pod="openshift-image-registry/image-registry-78c879b97f-xtp8m"
Sep 18 17:50:29.893399 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.893379    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8vrr\" (UniqueName: \"kubernetes.io/projected/6d0c7ec9-9691-4341-afd6-ba415fdb1326-kube-api-access-h8vrr\") pod \"volume-data-source-validator-5c4db8bdf9-vzcw7\" (UID: \"6d0c7ec9-9691-4341-afd6-ba415fdb1326\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-vzcw7"
Sep 18 17:50:29.893939 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.893916    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r557q\" (UniqueName: \"kubernetes.io/projected/20117f3e-e50b-4b03-aa48-604fc42d1737-kube-api-access-r557q\") pod \"router-default-7dc96485b6-66p6f\" (UID: \"20117f3e-e50b-4b03-aa48-604fc42d1737\") " pod="openshift-ingress/router-default-7dc96485b6-66p6f"
Sep 18 17:50:29.894018 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.893997    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpnbv\" (UniqueName: \"kubernetes.io/projected/60520815-25c1-4a4e-a036-c3fb968adbaf-kube-api-access-kpnbv\") pod \"service-ca-operator-57cdc4b768-qsqqj\" (UID: \"60520815-25c1-4a4e-a036-c3fb968adbaf\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-qsqqj"
Sep 18 17:50:29.894930 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.894902    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ltk8\" (UniqueName: \"kubernetes.io/projected/1096ace1-2554-43bc-bf2e-5e6d09e1c48f-kube-api-access-8ltk8\") pod \"image-registry-78c879b97f-xtp8m\" (UID: \"1096ace1-2554-43bc-bf2e-5e6d09e1c48f\") " pod="openshift-image-registry/image-registry-78c879b97f-xtp8m"
Sep 18 17:50:29.984342 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.984256    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7pc2\" (UniqueName: \"kubernetes.io/projected/75f2a1f7-95a7-462e-9ea7-23c51dd29599-kube-api-access-k7pc2\") pod \"console-operator-fd4bb46c-5kjlz\" (UID: \"75f2a1f7-95a7-462e-9ea7-23c51dd29599\") " pod="openshift-console-operator/console-operator-fd4bb46c-5kjlz"
Sep 18 17:50:29.984342 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.984308    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ac028d85-f1a2-4c5a-91ad-fb7141335c07-cert\") pod \"ingress-canary-kw7df\" (UID: \"ac028d85-f1a2-4c5a-91ad-fb7141335c07\") " pod="openshift-ingress-canary/ingress-canary-kw7df"
Sep 18 17:50:29.984654 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.984346    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znz5k\" (UniqueName: \"kubernetes.io/projected/ac028d85-f1a2-4c5a-91ad-fb7141335c07-kube-api-access-znz5k\") pod \"ingress-canary-kw7df\" (UID: \"ac028d85-f1a2-4c5a-91ad-fb7141335c07\") " pod="openshift-ingress-canary/ingress-canary-kw7df"
Sep 18 17:50:29.984654 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.984368    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6n6f\" (UniqueName: \"kubernetes.io/projected/1286716b-4028-484e-b51d-58eb4b9f7305-kube-api-access-g6n6f\") pod \"cluster-samples-operator-685d7d68b5-qd9mb\" (UID: \"1286716b-4028-484e-b51d-58eb4b9f7305\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-qd9mb"
Sep 18 17:50:29.984654 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.984396    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2e2c3ab1-da7b-40e3-8d33-4d7ec5980634-snapshots\") pod \"insights-operator-6f9c644dd9-4hdg8\" (UID: \"2e2c3ab1-da7b-40e3-8d33-4d7ec5980634\") " pod="openshift-insights/insights-operator-6f9c644dd9-4hdg8"
Sep 18 17:50:29.984654 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.984421    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e2c3ab1-da7b-40e3-8d33-4d7ec5980634-tmp\") pod \"insights-operator-6f9c644dd9-4hdg8\" (UID: \"2e2c3ab1-da7b-40e3-8d33-4d7ec5980634\") " pod="openshift-insights/insights-operator-6f9c644dd9-4hdg8"
Sep 18 17:50:29.984654 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.984457    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2e2c3ab1-da7b-40e3-8d33-4d7ec5980634-serving-cert\") pod \"insights-operator-6f9c644dd9-4hdg8\" (UID: \"2e2c3ab1-da7b-40e3-8d33-4d7ec5980634\") " pod="openshift-insights/insights-operator-6f9c644dd9-4hdg8"
Sep 18 17:50:29.984654 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.984479    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/60a031b6-31f6-47a2-859c-bfd960f0b2c7-metrics-tls\") pod \"dns-default-9hjjd\" (UID: \"60a031b6-31f6-47a2-859c-bfd960f0b2c7\") " pod="openshift-dns/dns-default-9hjjd"
Sep 18 17:50:29.984654 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.984508    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60a031b6-31f6-47a2-859c-bfd960f0b2c7-tmp-dir\") pod \"dns-default-9hjjd\" (UID: \"60a031b6-31f6-47a2-859c-bfd960f0b2c7\") " pod="openshift-dns/dns-default-9hjjd"
Sep 18 17:50:29.984654 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.984543    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/75f2a1f7-95a7-462e-9ea7-23c51dd29599-trusted-ca\") pod \"console-operator-fd4bb46c-5kjlz\" (UID: \"75f2a1f7-95a7-462e-9ea7-23c51dd29599\") " pod="openshift-console-operator/console-operator-fd4bb46c-5kjlz"
Sep 18 17:50:29.984654 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:29.984604    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 17:50:29.985719 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:29.984685    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60a031b6-31f6-47a2-859c-bfd960f0b2c7-metrics-tls podName:60a031b6-31f6-47a2-859c-bfd960f0b2c7 nodeName:}" failed. No retries permitted until 2026-09-18 17:50:30.484659036 +0000 UTC m=+35.111790655 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/60a031b6-31f6-47a2-859c-bfd960f0b2c7-metrics-tls") pod "dns-default-9hjjd" (UID: "60a031b6-31f6-47a2-859c-bfd960f0b2c7") : secret "dns-default-metrics-tls" not found
Sep 18 17:50:29.985719 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.984864    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e2c3ab1-da7b-40e3-8d33-4d7ec5980634-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-4hdg8\" (UID: \"2e2c3ab1-da7b-40e3-8d33-4d7ec5980634\") " pod="openshift-insights/insights-operator-6f9c644dd9-4hdg8"
Sep 18 17:50:29.985719 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.984898    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/75f2a1f7-95a7-462e-9ea7-23c51dd29599-serving-cert\") pod \"console-operator-fd4bb46c-5kjlz\" (UID: \"75f2a1f7-95a7-462e-9ea7-23c51dd29599\") " pod="openshift-console-operator/console-operator-fd4bb46c-5kjlz"
Sep 18 17:50:29.985719 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.984933    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e2c3ab1-da7b-40e3-8d33-4d7ec5980634-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-4hdg8\" (UID: \"2e2c3ab1-da7b-40e3-8d33-4d7ec5980634\") " pod="openshift-insights/insights-operator-6f9c644dd9-4hdg8"
Sep 18 17:50:29.985719 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.984963    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvfvh\" (UniqueName: \"kubernetes.io/projected/6586ff5a-f123-4692-a672-1844f9ce5a11-kube-api-access-rvfvh\") pod \"network-check-source-7cbc45fddc-9ttqt\" (UID: \"6586ff5a-f123-4692-a672-1844f9ce5a11\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-9ttqt"
Sep 18 17:50:29.985719 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.985025    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn24x\" (UniqueName: \"kubernetes.io/projected/60a031b6-31f6-47a2-859c-bfd960f0b2c7-kube-api-access-fn24x\") pod \"dns-default-9hjjd\" (UID: \"60a031b6-31f6-47a2-859c-bfd960f0b2c7\") " pod="openshift-dns/dns-default-9hjjd"
Sep 18 17:50:29.985719 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.985051    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1286716b-4028-484e-b51d-58eb4b9f7305-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-qd9mb\" (UID: \"1286716b-4028-484e-b51d-58eb4b9f7305\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-qd9mb"
Sep 18 17:50:29.985719 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.985105    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgmbt\" (UniqueName: \"kubernetes.io/projected/2e2c3ab1-da7b-40e3-8d33-4d7ec5980634-kube-api-access-pgmbt\") pod \"insights-operator-6f9c644dd9-4hdg8\" (UID: \"2e2c3ab1-da7b-40e3-8d33-4d7ec5980634\") " pod="openshift-insights/insights-operator-6f9c644dd9-4hdg8"
Sep 18 17:50:29.985719 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.985144    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/60a031b6-31f6-47a2-859c-bfd960f0b2c7-config-volume\") pod \"dns-default-9hjjd\" (UID: \"60a031b6-31f6-47a2-859c-bfd960f0b2c7\") " pod="openshift-dns/dns-default-9hjjd"
Sep 18 17:50:29.985719 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.985172    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/75f2a1f7-95a7-462e-9ea7-23c51dd29599-config\") pod \"console-operator-fd4bb46c-5kjlz\" (UID: \"75f2a1f7-95a7-462e-9ea7-23c51dd29599\") " pod="openshift-console-operator/console-operator-fd4bb46c-5kjlz"
Sep 18 17:50:29.985719 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.985167    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60a031b6-31f6-47a2-859c-bfd960f0b2c7-tmp-dir\") pod \"dns-default-9hjjd\" (UID: \"60a031b6-31f6-47a2-859c-bfd960f0b2c7\") " pod="openshift-dns/dns-default-9hjjd"
Sep 18 17:50:29.985719 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:29.985321    2561 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 17:50:29.985719 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.985326    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2e2c3ab1-da7b-40e3-8d33-4d7ec5980634-snapshots\") pod \"insights-operator-6f9c644dd9-4hdg8\" (UID: \"2e2c3ab1-da7b-40e3-8d33-4d7ec5980634\") " pod="openshift-insights/insights-operator-6f9c644dd9-4hdg8"
Sep 18 17:50:29.985719 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:29.985368    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1286716b-4028-484e-b51d-58eb4b9f7305-samples-operator-tls podName:1286716b-4028-484e-b51d-58eb4b9f7305 nodeName:}" failed. No retries permitted until 2026-09-18 17:50:30.485352683 +0000 UTC m=+35.112484281 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1286716b-4028-484e-b51d-58eb4b9f7305-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-qd9mb" (UID: "1286716b-4028-484e-b51d-58eb4b9f7305") : secret "samples-operator-tls" not found
Sep 18 17:50:29.985719 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.985417    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e2c3ab1-da7b-40e3-8d33-4d7ec5980634-tmp\") pod \"insights-operator-6f9c644dd9-4hdg8\" (UID: \"2e2c3ab1-da7b-40e3-8d33-4d7ec5980634\") " pod="openshift-insights/insights-operator-6f9c644dd9-4hdg8"
Sep 18 17:50:29.985719 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.985471    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/75f2a1f7-95a7-462e-9ea7-23c51dd29599-trusted-ca\") pod \"console-operator-fd4bb46c-5kjlz\" (UID: \"75f2a1f7-95a7-462e-9ea7-23c51dd29599\") " pod="openshift-console-operator/console-operator-fd4bb46c-5kjlz"
Sep 18 17:50:29.986926 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.986715    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e2c3ab1-da7b-40e3-8d33-4d7ec5980634-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-4hdg8\" (UID: \"2e2c3ab1-da7b-40e3-8d33-4d7ec5980634\") " pod="openshift-insights/insights-operator-6f9c644dd9-4hdg8"
Sep 18 17:50:29.987975 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.987949    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e2c3ab1-da7b-40e3-8d33-4d7ec5980634-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-4hdg8\" (UID: \"2e2c3ab1-da7b-40e3-8d33-4d7ec5980634\") " pod="openshift-insights/insights-operator-6f9c644dd9-4hdg8"
Sep 18 17:50:29.988131 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.988095    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/75f2a1f7-95a7-462e-9ea7-23c51dd29599-config\") pod \"console-operator-fd4bb46c-5kjlz\" (UID: \"75f2a1f7-95a7-462e-9ea7-23c51dd29599\") " pod="openshift-console-operator/console-operator-fd4bb46c-5kjlz"
Sep 18 17:50:29.988197 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.988166    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2e2c3ab1-da7b-40e3-8d33-4d7ec5980634-serving-cert\") pod \"insights-operator-6f9c644dd9-4hdg8\" (UID: \"2e2c3ab1-da7b-40e3-8d33-4d7ec5980634\") " pod="openshift-insights/insights-operator-6f9c644dd9-4hdg8"
Sep 18 17:50:29.992089 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.988296    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ac028d85-f1a2-4c5a-91ad-fb7141335c07-cert\") pod \"ingress-canary-kw7df\" (UID: \"ac028d85-f1a2-4c5a-91ad-fb7141335c07\") " pod="openshift-ingress-canary/ingress-canary-kw7df"
Sep 18 17:50:29.992089 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.988594    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/60a031b6-31f6-47a2-859c-bfd960f0b2c7-config-volume\") pod \"dns-default-9hjjd\" (UID: \"60a031b6-31f6-47a2-859c-bfd960f0b2c7\") " pod="openshift-dns/dns-default-9hjjd"
Sep 18 17:50:29.992089 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.988660    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-vzcw7"
Sep 18 17:50:29.992089 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.991719    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/75f2a1f7-95a7-462e-9ea7-23c51dd29599-serving-cert\") pod \"console-operator-fd4bb46c-5kjlz\" (UID: \"75f2a1f7-95a7-462e-9ea7-23c51dd29599\") " pod="openshift-console-operator/console-operator-fd4bb46c-5kjlz"
Sep 18 17:50:29.996317 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.996274    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn24x\" (UniqueName: \"kubernetes.io/projected/60a031b6-31f6-47a2-859c-bfd960f0b2c7-kube-api-access-fn24x\") pod \"dns-default-9hjjd\" (UID: \"60a031b6-31f6-47a2-859c-bfd960f0b2c7\") " pod="openshift-dns/dns-default-9hjjd"
Sep 18 17:50:29.996449 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.996425    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6n6f\" (UniqueName: \"kubernetes.io/projected/1286716b-4028-484e-b51d-58eb4b9f7305-kube-api-access-g6n6f\") pod \"cluster-samples-operator-685d7d68b5-qd9mb\" (UID: \"1286716b-4028-484e-b51d-58eb4b9f7305\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-qd9mb"
Sep 18 17:50:29.997578 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.997558    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgmbt\" (UniqueName: \"kubernetes.io/projected/2e2c3ab1-da7b-40e3-8d33-4d7ec5980634-kube-api-access-pgmbt\") pod \"insights-operator-6f9c644dd9-4hdg8\" (UID: \"2e2c3ab1-da7b-40e3-8d33-4d7ec5980634\") " pod="openshift-insights/insights-operator-6f9c644dd9-4hdg8"
Sep 18 17:50:29.997965 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.997943    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvfvh\" (UniqueName: \"kubernetes.io/projected/6586ff5a-f123-4692-a672-1844f9ce5a11-kube-api-access-rvfvh\") pod \"network-check-source-7cbc45fddc-9ttqt\" (UID: \"6586ff5a-f123-4692-a672-1844f9ce5a11\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-9ttqt"
Sep 18 17:50:29.998904 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.998726    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-qsqqj"
Sep 18 17:50:29.999334 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.999252    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7pc2\" (UniqueName: \"kubernetes.io/projected/75f2a1f7-95a7-462e-9ea7-23c51dd29599-kube-api-access-k7pc2\") pod \"console-operator-fd4bb46c-5kjlz\" (UID: \"75f2a1f7-95a7-462e-9ea7-23c51dd29599\") " pod="openshift-console-operator/console-operator-fd4bb46c-5kjlz"
Sep 18 17:50:29.999852 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:29.999832    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znz5k\" (UniqueName: \"kubernetes.io/projected/ac028d85-f1a2-4c5a-91ad-fb7141335c07-kube-api-access-znz5k\") pod \"ingress-canary-kw7df\" (UID: \"ac028d85-f1a2-4c5a-91ad-fb7141335c07\") " pod="openshift-ingress-canary/ingress-canary-kw7df"
Sep 18 17:50:30.032568 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:30.032532    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-dx752"
Sep 18 17:50:30.068156 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:30.067798    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-5kjlz"
Sep 18 17:50:30.075465 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:30.075271    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-9ttqt"
Sep 18 17:50:30.085818 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:30.084948    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-4hdg8"
Sep 18 17:50:30.086664 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:30.086192    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kw7df"
Sep 18 17:50:30.286226 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:30.286157    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-vzcw7"]
Sep 18 17:50:30.288728 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:30.288708    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-qsqqj"]
Sep 18 17:50:30.291037 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:50:30.291008    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d0c7ec9_9691_4341_afd6_ba415fdb1326.slice/crio-36c1004695e33e6c5fa22c37b4d69743f2c0191ee1b5382f515b5e31f7ff8e79 WatchSource:0}: Error finding container 36c1004695e33e6c5fa22c37b4d69743f2c0191ee1b5382f515b5e31f7ff8e79: Status 404 returned error can't find the container with id 36c1004695e33e6c5fa22c37b4d69743f2c0191ee1b5382f515b5e31f7ff8e79
Sep 18 17:50:30.291662 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:50:30.291636    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60520815_25c1_4a4e_a036_c3fb968adbaf.slice/crio-34cf5f578c6cc1564c46d240a7931d9728d4ec46959ff5dedd7999ca9caa17fa WatchSource:0}: Error finding container 34cf5f578c6cc1564c46d240a7931d9728d4ec46959ff5dedd7999ca9caa17fa: Status 404 returned error can't find the container with id 34cf5f578c6cc1564c46d240a7931d9728d4ec46959ff5dedd7999ca9caa17fa
Sep 18 17:50:30.324341 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:30.324307    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-dx752"]
Sep 18 17:50:30.328020 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:50:30.327996    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26ece3e7_900c_4efe_b497_653cf58b5e60.slice/crio-041676a9aba347d780e12b8d722e51318e0ef271b424746ec143a60b62ac89de WatchSource:0}: Error finding container 041676a9aba347d780e12b8d722e51318e0ef271b424746ec143a60b62ac89de: Status 404 returned error can't find the container with id 041676a9aba347d780e12b8d722e51318e0ef271b424746ec143a60b62ac89de
Sep 18 17:50:30.366533 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:30.366491    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-9ttqt"]
Sep 18 17:50:30.369370 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:50:30.369345    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6586ff5a_f123_4692_a672_1844f9ce5a11.slice/crio-2b223bc5d9d4237395c9b3a833367bc5bf8bce4bfff0efd4712af8723a865628 WatchSource:0}: Error finding container 2b223bc5d9d4237395c9b3a833367bc5bf8bce4bfff0efd4712af8723a865628: Status 404 returned error can't find the container with id 2b223bc5d9d4237395c9b3a833367bc5bf8bce4bfff0efd4712af8723a865628
Sep 18 17:50:30.378991 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:30.378950    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-5kjlz"]
Sep 18 17:50:30.382452 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:50:30.382427    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75f2a1f7_95a7_462e_9ea7_23c51dd29599.slice/crio-8630d82d35a7ddc8f28d71b095decea6fcc4ffee674eedcba1b90a1290a0ced0 WatchSource:0}: Error finding container 8630d82d35a7ddc8f28d71b095decea6fcc4ffee674eedcba1b90a1290a0ced0: Status 404 returned error can't find the container with id 8630d82d35a7ddc8f28d71b095decea6fcc4ffee674eedcba1b90a1290a0ced0
Sep 18 17:50:30.389707 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:30.389685    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a1e4f37f-b0ba-469f-8923-d0c9d38e619b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-xjzc9\" (UID: \"a1e4f37f-b0ba-469f-8923-d0c9d38e619b\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xjzc9"
Sep 18 17:50:30.389810 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:30.389745    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20117f3e-e50b-4b03-aa48-604fc42d1737-metrics-certs\") pod \"router-default-7dc96485b6-66p6f\" (UID: \"20117f3e-e50b-4b03-aa48-604fc42d1737\") " pod="openshift-ingress/router-default-7dc96485b6-66p6f"
Sep 18 17:50:30.389810 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:30.389780    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1096ace1-2554-43bc-bf2e-5e6d09e1c48f-registry-tls\") pod \"image-registry-78c879b97f-xtp8m\" (UID: \"1096ace1-2554-43bc-bf2e-5e6d09e1c48f\") " pod="openshift-image-registry/image-registry-78c879b97f-xtp8m"
Sep 18 17:50:30.389915 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:30.389821    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20117f3e-e50b-4b03-aa48-604fc42d1737-service-ca-bundle\") pod \"router-default-7dc96485b6-66p6f\" (UID: \"20117f3e-e50b-4b03-aa48-604fc42d1737\") " pod="openshift-ingress/router-default-7dc96485b6-66p6f"
Sep 18 17:50:30.389915 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:30.389853    2561 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 17:50:30.389915 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:30.389905    2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 17:50:30.390034 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:30.389914    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20117f3e-e50b-4b03-aa48-604fc42d1737-metrics-certs podName:20117f3e-e50b-4b03-aa48-604fc42d1737 nodeName:}" failed. No retries permitted until 2026-09-18 17:50:31.389894425 +0000 UTC m=+36.017026042 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/20117f3e-e50b-4b03-aa48-604fc42d1737-metrics-certs") pod "router-default-7dc96485b6-66p6f" (UID: "20117f3e-e50b-4b03-aa48-604fc42d1737") : secret "router-metrics-certs-default" not found
Sep 18 17:50:30.390034 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:30.389969    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 17:50:30.390034 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:30.389977    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/20117f3e-e50b-4b03-aa48-604fc42d1737-service-ca-bundle podName:20117f3e-e50b-4b03-aa48-604fc42d1737 nodeName:}" failed. No retries permitted until 2026-09-18 17:50:31.389960701 +0000 UTC m=+36.017092306 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/20117f3e-e50b-4b03-aa48-604fc42d1737-service-ca-bundle") pod "router-default-7dc96485b6-66p6f" (UID: "20117f3e-e50b-4b03-aa48-604fc42d1737") : configmap references non-existent config key: service-ca.crt
Sep 18 17:50:30.390034 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:30.389980    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78c879b97f-xtp8m: secret "image-registry-tls" not found
Sep 18 17:50:30.390221 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:30.390146    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1e4f37f-b0ba-469f-8923-d0c9d38e619b-cluster-monitoring-operator-tls podName:a1e4f37f-b0ba-469f-8923-d0c9d38e619b nodeName:}" failed. No retries permitted until 2026-09-18 17:50:31.390131503 +0000 UTC m=+36.017263105 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a1e4f37f-b0ba-469f-8923-d0c9d38e619b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-xjzc9" (UID: "a1e4f37f-b0ba-469f-8923-d0c9d38e619b") : secret "cluster-monitoring-operator-tls" not found
Sep 18 17:50:30.390221 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:30.390192    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1096ace1-2554-43bc-bf2e-5e6d09e1c48f-registry-tls podName:1096ace1-2554-43bc-bf2e-5e6d09e1c48f nodeName:}" failed. No retries permitted until 2026-09-18 17:50:31.390173228 +0000 UTC m=+36.017304832 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1096ace1-2554-43bc-bf2e-5e6d09e1c48f-registry-tls") pod "image-registry-78c879b97f-xtp8m" (UID: "1096ace1-2554-43bc-bf2e-5e6d09e1c48f") : secret "image-registry-tls" not found
Sep 18 17:50:30.397542 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:30.397515    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kw7df"]
Sep 18 17:50:30.400077 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:30.400054    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-4hdg8"]
Sep 18 17:50:30.400788 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:50:30.400767    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac028d85_f1a2_4c5a_91ad_fb7141335c07.slice/crio-41a0a4de12c91df3f8ee203f9913e68fda3b1d518462bf24f8247c133f4e5fce WatchSource:0}: Error finding container 41a0a4de12c91df3f8ee203f9913e68fda3b1d518462bf24f8247c133f4e5fce: Status 404 returned error can't find the container with id 41a0a4de12c91df3f8ee203f9913e68fda3b1d518462bf24f8247c133f4e5fce
Sep 18 17:50:30.403974 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:50:30.403950    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e2c3ab1_da7b_40e3_8d33_4d7ec5980634.slice/crio-86cd317327a14fb5acc57c1c1372b471dd2664e99ea7b935d9c41f28ac0cda23 WatchSource:0}: Error finding container 86cd317327a14fb5acc57c1c1372b471dd2664e99ea7b935d9c41f28ac0cda23: Status 404 returned error can't find the container with id 86cd317327a14fb5acc57c1c1372b471dd2664e99ea7b935d9c41f28ac0cda23
Sep 18 17:50:30.490666 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:30.490635    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1286716b-4028-484e-b51d-58eb4b9f7305-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-qd9mb\" (UID: \"1286716b-4028-484e-b51d-58eb4b9f7305\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-qd9mb"
Sep 18 17:50:30.491158 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:30.490760    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/60a031b6-31f6-47a2-859c-bfd960f0b2c7-metrics-tls\") pod \"dns-default-9hjjd\" (UID: \"60a031b6-31f6-47a2-859c-bfd960f0b2c7\") " pod="openshift-dns/dns-default-9hjjd"
Sep 18 17:50:30.491158 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:30.490798    2561 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 17:50:30.491158 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:30.490868    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1286716b-4028-484e-b51d-58eb4b9f7305-samples-operator-tls podName:1286716b-4028-484e-b51d-58eb4b9f7305 nodeName:}" failed. No retries permitted until 2026-09-18 17:50:31.490852341 +0000 UTC m=+36.117983962 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1286716b-4028-484e-b51d-58eb4b9f7305-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-qd9mb" (UID: "1286716b-4028-484e-b51d-58eb4b9f7305") : secret "samples-operator-tls" not found
Sep 18 17:50:30.491158 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:30.490890    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 17:50:30.491158 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:30.490933    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60a031b6-31f6-47a2-859c-bfd960f0b2c7-metrics-tls podName:60a031b6-31f6-47a2-859c-bfd960f0b2c7 nodeName:}" failed. No retries permitted until 2026-09-18 17:50:31.49092134 +0000 UTC m=+36.118052963 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/60a031b6-31f6-47a2-859c-bfd960f0b2c7-metrics-tls") pod "dns-default-9hjjd" (UID: "60a031b6-31f6-47a2-859c-bfd960f0b2c7") : secret "dns-default-metrics-tls" not found
Sep 18 17:50:30.855315 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:30.855283    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bgwtf"
Sep 18 17:50:30.855494 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:30.855283    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tmtsv"
Sep 18 17:50:30.855579 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:30.855283    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mvtdm"
Sep 18 17:50:30.859016 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:30.858991    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-r5l4c\""
Sep 18 17:50:30.859016 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:30.859006    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 17:50:30.859206 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:30.859059    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sdsc5\""
Sep 18 17:50:30.859206 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:30.859006    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 17:50:30.995272 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:30.995230    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-4hdg8" event={"ID":"2e2c3ab1-da7b-40e3-8d33-4d7ec5980634","Type":"ContainerStarted","Data":"86cd317327a14fb5acc57c1c1372b471dd2664e99ea7b935d9c41f28ac0cda23"}
Sep 18 17:50:30.996639 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:30.996606    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-9ttqt" event={"ID":"6586ff5a-f123-4692-a672-1844f9ce5a11","Type":"ContainerStarted","Data":"2b223bc5d9d4237395c9b3a833367bc5bf8bce4bfff0efd4712af8723a865628"}
Sep 18 17:50:30.997893 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:30.997867    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-5kjlz" event={"ID":"75f2a1f7-95a7-462e-9ea7-23c51dd29599","Type":"ContainerStarted","Data":"8630d82d35a7ddc8f28d71b095decea6fcc4ffee674eedcba1b90a1290a0ced0"}
Sep 18 17:50:30.999159 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:30.999136    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-vzcw7" event={"ID":"6d0c7ec9-9691-4341-afd6-ba415fdb1326","Type":"ContainerStarted","Data":"36c1004695e33e6c5fa22c37b4d69743f2c0191ee1b5382f515b5e31f7ff8e79"}
Sep 18 17:50:31.000313 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:31.000290    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-dx752" event={"ID":"26ece3e7-900c-4efe-b497-653cf58b5e60","Type":"ContainerStarted","Data":"041676a9aba347d780e12b8d722e51318e0ef271b424746ec143a60b62ac89de"}
Sep 18 17:50:31.001545 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:31.001518    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-qsqqj" event={"ID":"60520815-25c1-4a4e-a036-c3fb968adbaf","Type":"ContainerStarted","Data":"34cf5f578c6cc1564c46d240a7931d9728d4ec46959ff5dedd7999ca9caa17fa"}
Sep 18 17:50:31.002712 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:31.002690    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kw7df" event={"ID":"ac028d85-f1a2-4c5a-91ad-fb7141335c07","Type":"ContainerStarted","Data":"41a0a4de12c91df3f8ee203f9913e68fda3b1d518462bf24f8247c133f4e5fce"}
Sep 18 17:50:31.398513 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:31.398475    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a1e4f37f-b0ba-469f-8923-d0c9d38e619b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-xjzc9\" (UID: \"a1e4f37f-b0ba-469f-8923-d0c9d38e619b\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xjzc9"
Sep 18 17:50:31.398703 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:31.398543    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20117f3e-e50b-4b03-aa48-604fc42d1737-metrics-certs\") pod \"router-default-7dc96485b6-66p6f\" (UID: \"20117f3e-e50b-4b03-aa48-604fc42d1737\") " pod="openshift-ingress/router-default-7dc96485b6-66p6f"
Sep 18 17:50:31.398703 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:31.398581    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1096ace1-2554-43bc-bf2e-5e6d09e1c48f-registry-tls\") pod \"image-registry-78c879b97f-xtp8m\" (UID: \"1096ace1-2554-43bc-bf2e-5e6d09e1c48f\") " pod="openshift-image-registry/image-registry-78c879b97f-xtp8m"
Sep 18 17:50:31.398703 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:31.398616    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20117f3e-e50b-4b03-aa48-604fc42d1737-service-ca-bundle\") pod \"router-default-7dc96485b6-66p6f\" (UID: \"20117f3e-e50b-4b03-aa48-604fc42d1737\") " pod="openshift-ingress/router-default-7dc96485b6-66p6f"
Sep 18 17:50:31.398703 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:31.398617    2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 17:50:31.398906 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:31.398703    2561 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 17:50:31.398906 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:31.398741    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1e4f37f-b0ba-469f-8923-d0c9d38e619b-cluster-monitoring-operator-tls podName:a1e4f37f-b0ba-469f-8923-d0c9d38e619b nodeName:}" failed. No retries permitted until 2026-09-18 17:50:33.398724622 +0000 UTC m=+38.025856223 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a1e4f37f-b0ba-469f-8923-d0c9d38e619b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-xjzc9" (UID: "a1e4f37f-b0ba-469f-8923-d0c9d38e619b") : secret "cluster-monitoring-operator-tls" not found
Sep 18 17:50:31.398906 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:31.398707    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 17:50:31.398906 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:31.398763    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78c879b97f-xtp8m: secret "image-registry-tls" not found
Sep 18 17:50:31.398906 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:31.398768    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20117f3e-e50b-4b03-aa48-604fc42d1737-metrics-certs podName:20117f3e-e50b-4b03-aa48-604fc42d1737 nodeName:}" failed. No retries permitted until 2026-09-18 17:50:33.398748727 +0000 UTC m=+38.025880328 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/20117f3e-e50b-4b03-aa48-604fc42d1737-metrics-certs") pod "router-default-7dc96485b6-66p6f" (UID: "20117f3e-e50b-4b03-aa48-604fc42d1737") : secret "router-metrics-certs-default" not found
Sep 18 17:50:31.398906 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:31.398786    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/20117f3e-e50b-4b03-aa48-604fc42d1737-service-ca-bundle podName:20117f3e-e50b-4b03-aa48-604fc42d1737 nodeName:}" failed. No retries permitted until 2026-09-18 17:50:33.398776943 +0000 UTC m=+38.025908547 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/20117f3e-e50b-4b03-aa48-604fc42d1737-service-ca-bundle") pod "router-default-7dc96485b6-66p6f" (UID: "20117f3e-e50b-4b03-aa48-604fc42d1737") : configmap references non-existent config key: service-ca.crt
Sep 18 17:50:31.398906 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:31.398801    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1096ace1-2554-43bc-bf2e-5e6d09e1c48f-registry-tls podName:1096ace1-2554-43bc-bf2e-5e6d09e1c48f nodeName:}" failed. No retries permitted until 2026-09-18 17:50:33.398792244 +0000 UTC m=+38.025923846 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1096ace1-2554-43bc-bf2e-5e6d09e1c48f-registry-tls") pod "image-registry-78c879b97f-xtp8m" (UID: "1096ace1-2554-43bc-bf2e-5e6d09e1c48f") : secret "image-registry-tls" not found
Sep 18 17:50:31.499034 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:31.499001    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/60a031b6-31f6-47a2-859c-bfd960f0b2c7-metrics-tls\") pod \"dns-default-9hjjd\" (UID: \"60a031b6-31f6-47a2-859c-bfd960f0b2c7\") " pod="openshift-dns/dns-default-9hjjd"
Sep 18 17:50:31.499639 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:31.499162    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1286716b-4028-484e-b51d-58eb4b9f7305-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-qd9mb\" (UID: \"1286716b-4028-484e-b51d-58eb4b9f7305\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-qd9mb"
Sep 18 17:50:31.499639 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:31.499180    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 17:50:31.499639 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:31.499237    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60a031b6-31f6-47a2-859c-bfd960f0b2c7-metrics-tls podName:60a031b6-31f6-47a2-859c-bfd960f0b2c7 nodeName:}" failed. No retries permitted until 2026-09-18 17:50:33.499222059 +0000 UTC m=+38.126353678 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/60a031b6-31f6-47a2-859c-bfd960f0b2c7-metrics-tls") pod "dns-default-9hjjd" (UID: "60a031b6-31f6-47a2-859c-bfd960f0b2c7") : secret "dns-default-metrics-tls" not found
Sep 18 17:50:31.499639 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:31.499302    2561 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 17:50:31.499639 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:31.499355    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1286716b-4028-484e-b51d-58eb4b9f7305-samples-operator-tls podName:1286716b-4028-484e-b51d-58eb4b9f7305 nodeName:}" failed. No retries permitted until 2026-09-18 17:50:33.499340073 +0000 UTC m=+38.126471693 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1286716b-4028-484e-b51d-58eb4b9f7305-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-qd9mb" (UID: "1286716b-4028-484e-b51d-58eb4b9f7305") : secret "samples-operator-tls" not found
Sep 18 17:50:32.467929 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:32.467826    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:50:32.564373 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:32.564228    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:50:32.640759 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:32.640693    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:50:33.032127 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:33.032027    2561 generic.go:356] "Generic (PLEG): container finished" podID="1dc25d25-efc8-477f-9a1e-5220f1d20345" containerID="f33e1e1e98b0d9bd2de378c3d611b2338a3330d184db8eb3a096ff63e8e78ff5" exitCode=0
Sep 18 17:50:33.032276 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:33.032136    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cqzh9" event={"ID":"1dc25d25-efc8-477f-9a1e-5220f1d20345","Type":"ContainerDied","Data":"f33e1e1e98b0d9bd2de378c3d611b2338a3330d184db8eb3a096ff63e8e78ff5"}
Sep 18 17:50:33.419784 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:33.418718    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a1e4f37f-b0ba-469f-8923-d0c9d38e619b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-xjzc9\" (UID: \"a1e4f37f-b0ba-469f-8923-d0c9d38e619b\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xjzc9"
Sep 18 17:50:33.419784 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:33.418793    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20117f3e-e50b-4b03-aa48-604fc42d1737-metrics-certs\") pod \"router-default-7dc96485b6-66p6f\" (UID: \"20117f3e-e50b-4b03-aa48-604fc42d1737\") " pod="openshift-ingress/router-default-7dc96485b6-66p6f"
Sep 18 17:50:33.419784 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:33.418829    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1096ace1-2554-43bc-bf2e-5e6d09e1c48f-registry-tls\") pod \"image-registry-78c879b97f-xtp8m\" (UID: \"1096ace1-2554-43bc-bf2e-5e6d09e1c48f\") " pod="openshift-image-registry/image-registry-78c879b97f-xtp8m"
Sep 18 17:50:33.419784 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:33.418867    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20117f3e-e50b-4b03-aa48-604fc42d1737-service-ca-bundle\") pod \"router-default-7dc96485b6-66p6f\" (UID: \"20117f3e-e50b-4b03-aa48-604fc42d1737\") " pod="openshift-ingress/router-default-7dc96485b6-66p6f"
Sep 18 17:50:33.419784 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:33.419018    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/20117f3e-e50b-4b03-aa48-604fc42d1737-service-ca-bundle podName:20117f3e-e50b-4b03-aa48-604fc42d1737 nodeName:}" failed. No retries permitted until 2026-09-18 17:50:37.418999265 +0000 UTC m=+42.046130866 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/20117f3e-e50b-4b03-aa48-604fc42d1737-service-ca-bundle") pod "router-default-7dc96485b6-66p6f" (UID: "20117f3e-e50b-4b03-aa48-604fc42d1737") : configmap references non-existent config key: service-ca.crt
Sep 18 17:50:33.419784 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:33.419459    2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 17:50:33.419784 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:33.419510    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1e4f37f-b0ba-469f-8923-d0c9d38e619b-cluster-monitoring-operator-tls podName:a1e4f37f-b0ba-469f-8923-d0c9d38e619b nodeName:}" failed. No retries permitted until 2026-09-18 17:50:37.419494184 +0000 UTC m=+42.046625786 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a1e4f37f-b0ba-469f-8923-d0c9d38e619b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-xjzc9" (UID: "a1e4f37f-b0ba-469f-8923-d0c9d38e619b") : secret "cluster-monitoring-operator-tls" not found
Sep 18 17:50:33.419784 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:33.419571    2561 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 17:50:33.419784 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:33.419603    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20117f3e-e50b-4b03-aa48-604fc42d1737-metrics-certs podName:20117f3e-e50b-4b03-aa48-604fc42d1737 nodeName:}" failed. No retries permitted until 2026-09-18 17:50:37.419592491 +0000 UTC m=+42.046724094 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/20117f3e-e50b-4b03-aa48-604fc42d1737-metrics-certs") pod "router-default-7dc96485b6-66p6f" (UID: "20117f3e-e50b-4b03-aa48-604fc42d1737") : secret "router-metrics-certs-default" not found
Sep 18 17:50:33.419784 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:33.419661    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 17:50:33.419784 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:33.419673    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78c879b97f-xtp8m: secret "image-registry-tls" not found
Sep 18 17:50:33.419784 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:33.419703    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1096ace1-2554-43bc-bf2e-5e6d09e1c48f-registry-tls podName:1096ace1-2554-43bc-bf2e-5e6d09e1c48f nodeName:}" failed. No retries permitted until 2026-09-18 17:50:37.419693088 +0000 UTC m=+42.046824686 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1096ace1-2554-43bc-bf2e-5e6d09e1c48f-registry-tls") pod "image-registry-78c879b97f-xtp8m" (UID: "1096ace1-2554-43bc-bf2e-5e6d09e1c48f") : secret "image-registry-tls" not found
Sep 18 17:50:33.520264 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:33.519955    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/60a031b6-31f6-47a2-859c-bfd960f0b2c7-metrics-tls\") pod \"dns-default-9hjjd\" (UID: \"60a031b6-31f6-47a2-859c-bfd960f0b2c7\") " pod="openshift-dns/dns-default-9hjjd"
Sep 18 17:50:33.521389 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:33.520956    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1286716b-4028-484e-b51d-58eb4b9f7305-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-qd9mb\" (UID: \"1286716b-4028-484e-b51d-58eb4b9f7305\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-qd9mb"
Sep 18 17:50:33.521389 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:33.521201    2561 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 17:50:33.521389 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:33.521258    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1286716b-4028-484e-b51d-58eb4b9f7305-samples-operator-tls podName:1286716b-4028-484e-b51d-58eb4b9f7305 nodeName:}" failed. No retries permitted until 2026-09-18 17:50:37.521240521 +0000 UTC m=+42.148372124 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1286716b-4028-484e-b51d-58eb4b9f7305-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-qd9mb" (UID: "1286716b-4028-484e-b51d-58eb4b9f7305") : secret "samples-operator-tls" not found
Sep 18 17:50:33.521389 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:33.521322    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 17:50:33.521389 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:33.521359    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60a031b6-31f6-47a2-859c-bfd960f0b2c7-metrics-tls podName:60a031b6-31f6-47a2-859c-bfd960f0b2c7 nodeName:}" failed. No retries permitted until 2026-09-18 17:50:37.521340452 +0000 UTC m=+42.148472068 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/60a031b6-31f6-47a2-859c-bfd960f0b2c7-metrics-tls") pod "dns-default-9hjjd" (UID: "60a031b6-31f6-47a2-859c-bfd960f0b2c7") : secret "dns-default-metrics-tls" not found
Sep 18 17:50:34.038474 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:34.038162    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-vzcw7" event={"ID":"6d0c7ec9-9691-4341-afd6-ba415fdb1326","Type":"ContainerStarted","Data":"1fabc4627415c63ae1ff410b0acf44031cf25aa353cd07f0ac00d4e2997e5d6a"}
Sep 18 17:50:34.043740 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:34.043709    2561 generic.go:356] "Generic (PLEG): container finished" podID="1dc25d25-efc8-477f-9a1e-5220f1d20345" containerID="cb08de87235c00cc7f65e41b4ca401aaee7965b9aeccf2443ba0ea27cd15bb3a" exitCode=0
Sep 18 17:50:34.043863 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:34.043783    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cqzh9" event={"ID":"1dc25d25-efc8-477f-9a1e-5220f1d20345","Type":"ContainerDied","Data":"cb08de87235c00cc7f65e41b4ca401aaee7965b9aeccf2443ba0ea27cd15bb3a"}
Sep 18 17:50:34.060228 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:34.060154    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-vzcw7" podStartSLOduration=10.752387285 podStartE2EDuration="14.060135207s" totalImagesPullingTime="3.307747922s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:50:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:50:34.058966791 +0000 UTC m=+38.686098435" watchObservedRunningTime="2026-09-18 17:50:34.060135207 +0000 UTC m=+38.687266826"
Sep 18 17:50:34.178828 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:34.178789    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:50:34.280842 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:34.280789    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:50:34.386535 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:34.386458    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:50:34.483685 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:34.483642    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:50:35.609645 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:35.609570    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:50:35.712294 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:35.712266    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:50:36.024732 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:36.024661    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:50:36.183094 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:36.183063    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:50:37.459011 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:37.458979    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20117f3e-e50b-4b03-aa48-604fc42d1737-metrics-certs\") pod \"router-default-7dc96485b6-66p6f\" (UID: \"20117f3e-e50b-4b03-aa48-604fc42d1737\") " pod="openshift-ingress/router-default-7dc96485b6-66p6f"
Sep 18 17:50:37.459436 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:37.459020    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1096ace1-2554-43bc-bf2e-5e6d09e1c48f-registry-tls\") pod \"image-registry-78c879b97f-xtp8m\" (UID: \"1096ace1-2554-43bc-bf2e-5e6d09e1c48f\") " pod="openshift-image-registry/image-registry-78c879b97f-xtp8m"
Sep 18 17:50:37.459436 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:37.459047    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20117f3e-e50b-4b03-aa48-604fc42d1737-service-ca-bundle\") pod \"router-default-7dc96485b6-66p6f\" (UID: \"20117f3e-e50b-4b03-aa48-604fc42d1737\") " pod="openshift-ingress/router-default-7dc96485b6-66p6f"
Sep 18 17:50:37.459436 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:37.459131    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a1e4f37f-b0ba-469f-8923-d0c9d38e619b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-xjzc9\" (UID: \"a1e4f37f-b0ba-469f-8923-d0c9d38e619b\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xjzc9"
Sep 18 17:50:37.459436 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:37.459156    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 17:50:37.459436 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:37.459175    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78c879b97f-xtp8m: secret "image-registry-tls" not found
Sep 18 17:50:37.459436 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:37.459153    2561 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 17:50:37.459436 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:37.459228    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1096ace1-2554-43bc-bf2e-5e6d09e1c48f-registry-tls podName:1096ace1-2554-43bc-bf2e-5e6d09e1c48f nodeName:}" failed. No retries permitted until 2026-09-18 17:50:45.459208392 +0000 UTC m=+50.086339990 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1096ace1-2554-43bc-bf2e-5e6d09e1c48f-registry-tls") pod "image-registry-78c879b97f-xtp8m" (UID: "1096ace1-2554-43bc-bf2e-5e6d09e1c48f") : secret "image-registry-tls" not found
Sep 18 17:50:37.459436 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:37.459253    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/20117f3e-e50b-4b03-aa48-604fc42d1737-service-ca-bundle podName:20117f3e-e50b-4b03-aa48-604fc42d1737 nodeName:}" failed. No retries permitted until 2026-09-18 17:50:45.45923408 +0000 UTC m=+50.086365701 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/20117f3e-e50b-4b03-aa48-604fc42d1737-service-ca-bundle") pod "router-default-7dc96485b6-66p6f" (UID: "20117f3e-e50b-4b03-aa48-604fc42d1737") : configmap references non-existent config key: service-ca.crt
Sep 18 17:50:37.459436 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:37.459218    2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 17:50:37.459436 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:37.459288    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20117f3e-e50b-4b03-aa48-604fc42d1737-metrics-certs podName:20117f3e-e50b-4b03-aa48-604fc42d1737 nodeName:}" failed. No retries permitted until 2026-09-18 17:50:45.459272471 +0000 UTC m=+50.086404069 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/20117f3e-e50b-4b03-aa48-604fc42d1737-metrics-certs") pod "router-default-7dc96485b6-66p6f" (UID: "20117f3e-e50b-4b03-aa48-604fc42d1737") : secret "router-metrics-certs-default" not found
Sep 18 17:50:37.459436 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:37.459306    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1e4f37f-b0ba-469f-8923-d0c9d38e619b-cluster-monitoring-operator-tls podName:a1e4f37f-b0ba-469f-8923-d0c9d38e619b nodeName:}" failed. No retries permitted until 2026-09-18 17:50:45.459294789 +0000 UTC m=+50.086426391 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a1e4f37f-b0ba-469f-8923-d0c9d38e619b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-xjzc9" (UID: "a1e4f37f-b0ba-469f-8923-d0c9d38e619b") : secret "cluster-monitoring-operator-tls" not found
Sep 18 17:50:37.559540 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:37.559506    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1286716b-4028-484e-b51d-58eb4b9f7305-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-qd9mb\" (UID: \"1286716b-4028-484e-b51d-58eb4b9f7305\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-qd9mb"
Sep 18 17:50:37.559681 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:37.559598    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/60a031b6-31f6-47a2-859c-bfd960f0b2c7-metrics-tls\") pod \"dns-default-9hjjd\" (UID: \"60a031b6-31f6-47a2-859c-bfd960f0b2c7\") " pod="openshift-dns/dns-default-9hjjd"
Sep 18 17:50:37.559681 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:37.559638    2561 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 17:50:37.559755 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:37.559684    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 17:50:37.559755 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:37.559689    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1286716b-4028-484e-b51d-58eb4b9f7305-samples-operator-tls podName:1286716b-4028-484e-b51d-58eb4b9f7305 nodeName:}" failed. No retries permitted until 2026-09-18 17:50:45.559676398 +0000 UTC m=+50.186807996 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1286716b-4028-484e-b51d-58eb4b9f7305-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-qd9mb" (UID: "1286716b-4028-484e-b51d-58eb4b9f7305") : secret "samples-operator-tls" not found
Sep 18 17:50:37.559755 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:37.559717    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60a031b6-31f6-47a2-859c-bfd960f0b2c7-metrics-tls podName:60a031b6-31f6-47a2-859c-bfd960f0b2c7 nodeName:}" failed. No retries permitted until 2026-09-18 17:50:45.559707078 +0000 UTC m=+50.186838676 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/60a031b6-31f6-47a2-859c-bfd960f0b2c7-metrics-tls") pod "dns-default-9hjjd" (UID: "60a031b6-31f6-47a2-859c-bfd960f0b2c7") : secret "dns-default-metrics-tls" not found
Sep 18 17:50:40.062002 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:40.061959    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kw7df" event={"ID":"ac028d85-f1a2-4c5a-91ad-fb7141335c07","Type":"ContainerStarted","Data":"7a0e462318c141bbd44bc822da71c36d3e51ac6642e62f5d9ecbfcd317dde64f"}
Sep 18 17:50:40.063928 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:40.063898    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-4hdg8" event={"ID":"2e2c3ab1-da7b-40e3-8d33-4d7ec5980634","Type":"ContainerStarted","Data":"f35e2d551ca510ea45480b149592bde2a0757d17ca069f7f2711053a527bcffc"}
Sep 18 17:50:40.065922 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:40.065901    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-9ttqt" event={"ID":"6586ff5a-f123-4692-a672-1844f9ce5a11","Type":"ContainerStarted","Data":"692c8feb265bdf5f9d5344955c0bc565bd4580f6f5d26bf3e2cd1566e5bc87a9"}
Sep 18 17:50:40.067981 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:40.067964    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-5kjlz_75f2a1f7-95a7-462e-9ea7-23c51dd29599/console-operator/0.log"
Sep 18 17:50:40.068075 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:40.067996    2561 generic.go:356] "Generic (PLEG): container finished" podID="75f2a1f7-95a7-462e-9ea7-23c51dd29599" containerID="500d058a61b1eddab16490856f8c4260c2cd0a4f81da86168ca8ad1e434ed7d9" exitCode=255
Sep 18 17:50:40.068075 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:40.068062    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-5kjlz" event={"ID":"75f2a1f7-95a7-462e-9ea7-23c51dd29599","Type":"ContainerDied","Data":"500d058a61b1eddab16490856f8c4260c2cd0a4f81da86168ca8ad1e434ed7d9"}
Sep 18 17:50:40.068323 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:40.068304    2561 scope.go:117] "RemoveContainer" containerID="500d058a61b1eddab16490856f8c4260c2cd0a4f81da86168ca8ad1e434ed7d9"
Sep 18 17:50:40.072580 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:40.072554    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cqzh9" event={"ID":"1dc25d25-efc8-477f-9a1e-5220f1d20345","Type":"ContainerStarted","Data":"b978eb0ccabcf2b70af004bff78d667ebdd7f0fe7920b299e76b6c86949b0a1d"}
Sep 18 17:50:40.074545 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:40.074524    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-dx752" event={"ID":"26ece3e7-900c-4efe-b497-653cf58b5e60","Type":"ContainerStarted","Data":"75f337190abcf6d9045d733bab624019a8decc191b556dd202973d18bdb9eb36"}
Sep 18 17:50:40.076162 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:40.076141    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-qsqqj" event={"ID":"60520815-25c1-4a4e-a036-c3fb968adbaf","Type":"ContainerStarted","Data":"abea28a7634ae57c57ea4fe664b9af206095882e81f8e799a54cdd61c064e74c"}
Sep 18 17:50:40.103536 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:40.103489    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-kw7df" podStartSLOduration=2.248174036 podStartE2EDuration="11.103470775s" totalImagesPullingTime="8.855296739s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:50:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:50:40.080949181 +0000 UTC m=+44.708080795" watchObservedRunningTime="2026-09-18 17:50:40.103470775 +0000 UTC m=+44.730602391"
Sep 18 17:50:40.103752 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:40.103727    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-6f9c644dd9-4hdg8" podStartSLOduration=11.251382863 podStartE2EDuration="20.103718763s" totalImagesPullingTime="8.8523359s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:50:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:50:40.103411219 +0000 UTC m=+44.730542837" watchObservedRunningTime="2026-09-18 17:50:40.103718763 +0000 UTC m=+44.730850385"
Sep 18 17:50:40.128739 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:40.128682    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-cqzh9" podStartSLOduration=13.469198273 podStartE2EDuration="43.128662956s" totalImagesPullingTime="29.659464683s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 17:49:57 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:50:40.126798174 +0000 UTC m=+44.753929788" watchObservedRunningTime="2026-09-18 17:50:40.128662956 +0000 UTC m=+44.755794593"
Sep 18 17:50:40.149708 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:40.149666    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-qsqqj" podStartSLOduration=11.184324862 podStartE2EDuration="20.149631228s" totalImagesPullingTime="8.965306366s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:50:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:50:40.148815249 +0000 UTC m=+44.775946870" watchObservedRunningTime="2026-09-18 17:50:40.149631228 +0000 UTC m=+44.776762856"
Sep 18 17:50:40.189141 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:40.188918    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-9ttqt" podStartSLOduration=8.292049607 podStartE2EDuration="17.188897888s" totalImagesPullingTime="8.896848281s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:50:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:50:40.186851152 +0000 UTC m=+44.813982773" watchObservedRunningTime="2026-09-18 17:50:40.188897888 +0000 UTC m=+44.816029509"
Sep 18 17:50:40.204703 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:40.203108    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-dx752" podStartSLOduration=11.275153084 podStartE2EDuration="20.203087753s" totalImagesPullingTime="8.927934669s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:50:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:50:40.202663287 +0000 UTC m=+44.829794924" watchObservedRunningTime="2026-09-18 17:50:40.203087753 +0000 UTC m=+44.830219374"
Sep 18 17:50:40.255947 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:40.255093    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:50:40.326051 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:40.325970    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:50:40.781979 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:40.781944    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:50:40.881931 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:40.881901    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:50:40.982494 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:40.982454    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:50:41.040450 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:41.040370    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:50:41.085982 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:41.085959    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-5kjlz_75f2a1f7-95a7-462e-9ea7-23c51dd29599/console-operator/1.log"
Sep 18 17:50:41.086510 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:41.086432    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-5kjlz_75f2a1f7-95a7-462e-9ea7-23c51dd29599/console-operator/0.log"
Sep 18 17:50:41.086510 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:41.086476    2561 generic.go:356] "Generic (PLEG): container finished" podID="75f2a1f7-95a7-462e-9ea7-23c51dd29599" containerID="33b362ee52182f27fdd4b5f2ccd4b533faa50dc5e417f5cf8c1bcd11ba242883" exitCode=255
Sep 18 17:50:41.086618 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:41.086571    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-5kjlz" event={"ID":"75f2a1f7-95a7-462e-9ea7-23c51dd29599","Type":"ContainerDied","Data":"33b362ee52182f27fdd4b5f2ccd4b533faa50dc5e417f5cf8c1bcd11ba242883"}
Sep 18 17:50:41.086691 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:41.086677    2561 scope.go:117] "RemoveContainer" containerID="500d058a61b1eddab16490856f8c4260c2cd0a4f81da86168ca8ad1e434ed7d9"
Sep 18 17:50:41.086802 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:41.086768    2561 scope.go:117] "RemoveContainer" containerID="33b362ee52182f27fdd4b5f2ccd4b533faa50dc5e417f5cf8c1bcd11ba242883"
Sep 18 17:50:41.086977 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:41.086955    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-5kjlz_openshift-console-operator(75f2a1f7-95a7-462e-9ea7-23c51dd29599)\"" pod="openshift-console-operator/console-operator-fd4bb46c-5kjlz" podUID="75f2a1f7-95a7-462e-9ea7-23c51dd29599"
Sep 18 17:50:41.444500 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:41.444465    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-cjzln"]
Sep 18 17:50:41.454049 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:41.454018    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:50:41.479417 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:41.479386    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-cjzln"]
Sep 18 17:50:41.479575 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:41.479506    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cjzln"
Sep 18 17:50:41.481997 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:41.481976    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 18 17:50:41.481997 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:41.481989    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 18 17:50:41.482187 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:41.482150    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vmk8k\""
Sep 18 17:50:41.528820 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:41.528784    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:50:41.598080 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:41.598051    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6e4028c2-33ee-41f9-bb98-a4bbdffe8b24-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cjzln\" (UID: \"6e4028c2-33ee-41f9-bb98-a4bbdffe8b24\") " pod="openshift-insights/insights-runtime-extractor-cjzln"
Sep 18 17:50:41.598080 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:41.598083    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px6zg\" (UniqueName: \"kubernetes.io/projected/6e4028c2-33ee-41f9-bb98-a4bbdffe8b24-kube-api-access-px6zg\") pod \"insights-runtime-extractor-cjzln\" (UID: \"6e4028c2-33ee-41f9-bb98-a4bbdffe8b24\") " pod="openshift-insights/insights-runtime-extractor-cjzln"
Sep 18 17:50:41.598313 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:41.598261    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6e4028c2-33ee-41f9-bb98-a4bbdffe8b24-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cjzln\" (UID: \"6e4028c2-33ee-41f9-bb98-a4bbdffe8b24\") " pod="openshift-insights/insights-runtime-extractor-cjzln"
Sep 18 17:50:41.598313 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:41.598305    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6e4028c2-33ee-41f9-bb98-a4bbdffe8b24-data-volume\") pod \"insights-runtime-extractor-cjzln\" (UID: \"6e4028c2-33ee-41f9-bb98-a4bbdffe8b24\") " pod="openshift-insights/insights-runtime-extractor-cjzln"
Sep 18 17:50:41.598404 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:41.598359    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6e4028c2-33ee-41f9-bb98-a4bbdffe8b24-crio-socket\") pod \"insights-runtime-extractor-cjzln\" (UID: \"6e4028c2-33ee-41f9-bb98-a4bbdffe8b24\") " pod="openshift-insights/insights-runtime-extractor-cjzln"
Sep 18 17:50:41.699037 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:41.698959    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6e4028c2-33ee-41f9-bb98-a4bbdffe8b24-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cjzln\" (UID: \"6e4028c2-33ee-41f9-bb98-a4bbdffe8b24\") " pod="openshift-insights/insights-runtime-extractor-cjzln"
Sep 18 17:50:41.699037 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:41.699006    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6e4028c2-33ee-41f9-bb98-a4bbdffe8b24-data-volume\") pod \"insights-runtime-extractor-cjzln\" (UID: \"6e4028c2-33ee-41f9-bb98-a4bbdffe8b24\") " pod="openshift-insights/insights-runtime-extractor-cjzln"
Sep 18 17:50:41.699273 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:41.699109    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 17:50:41.699273 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:41.699166    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6e4028c2-33ee-41f9-bb98-a4bbdffe8b24-crio-socket\") pod \"insights-runtime-extractor-cjzln\" (UID: \"6e4028c2-33ee-41f9-bb98-a4bbdffe8b24\") " pod="openshift-insights/insights-runtime-extractor-cjzln"
Sep 18 17:50:41.699273 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:41.699186    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e4028c2-33ee-41f9-bb98-a4bbdffe8b24-insights-runtime-extractor-tls podName:6e4028c2-33ee-41f9-bb98-a4bbdffe8b24 nodeName:}" failed. No retries permitted until 2026-09-18 17:50:42.199170596 +0000 UTC m=+46.826302198 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6e4028c2-33ee-41f9-bb98-a4bbdffe8b24-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cjzln" (UID: "6e4028c2-33ee-41f9-bb98-a4bbdffe8b24") : secret "insights-runtime-extractor-tls" not found
Sep 18 17:50:41.699273 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:41.699247    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6e4028c2-33ee-41f9-bb98-a4bbdffe8b24-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cjzln\" (UID: \"6e4028c2-33ee-41f9-bb98-a4bbdffe8b24\") " pod="openshift-insights/insights-runtime-extractor-cjzln"
Sep 18 17:50:41.699273 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:41.699272    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px6zg\" (UniqueName: \"kubernetes.io/projected/6e4028c2-33ee-41f9-bb98-a4bbdffe8b24-kube-api-access-px6zg\") pod \"insights-runtime-extractor-cjzln\" (UID: \"6e4028c2-33ee-41f9-bb98-a4bbdffe8b24\") " pod="openshift-insights/insights-runtime-extractor-cjzln"
Sep 18 17:50:41.699562 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:41.699304    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6e4028c2-33ee-41f9-bb98-a4bbdffe8b24-crio-socket\") pod \"insights-runtime-extractor-cjzln\" (UID: \"6e4028c2-33ee-41f9-bb98-a4bbdffe8b24\") " pod="openshift-insights/insights-runtime-extractor-cjzln"
Sep 18 17:50:41.699562 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:41.699360    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6e4028c2-33ee-41f9-bb98-a4bbdffe8b24-data-volume\") pod \"insights-runtime-extractor-cjzln\" (UID: \"6e4028c2-33ee-41f9-bb98-a4bbdffe8b24\") " pod="openshift-insights/insights-runtime-extractor-cjzln"
Sep 18 17:50:41.699562 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:41.699368    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 17:50:41.699562 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:41.699489    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6e4028c2-33ee-41f9-bb98-a4bbdffe8b24-kube-rbac-proxy-cm podName:6e4028c2-33ee-41f9-bb98-a4bbdffe8b24 nodeName:}" failed. No retries permitted until 2026-09-18 17:50:42.199470516 +0000 UTC m=+46.826602129 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6e4028c2-33ee-41f9-bb98-a4bbdffe8b24-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cjzln" (UID: "6e4028c2-33ee-41f9-bb98-a4bbdffe8b24") : configmap "kube-rbac-proxy" not found
Sep 18 17:50:41.704544 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:41.704522    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:50:41.709525 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:41.709498    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-px6zg\" (UniqueName: \"kubernetes.io/projected/6e4028c2-33ee-41f9-bb98-a4bbdffe8b24-kube-api-access-px6zg\") pod \"insights-runtime-extractor-cjzln\" (UID: \"6e4028c2-33ee-41f9-bb98-a4bbdffe8b24\") " pod="openshift-insights/insights-runtime-extractor-cjzln"
Sep 18 17:50:41.806636 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:41.806607    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:50:42.090900 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:42.090826    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-5kjlz_75f2a1f7-95a7-462e-9ea7-23c51dd29599/console-operator/1.log"
Sep 18 17:50:42.091264 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:42.091143    2561 scope.go:117] "RemoveContainer" containerID="33b362ee52182f27fdd4b5f2ccd4b533faa50dc5e417f5cf8c1bcd11ba242883"
Sep 18 17:50:42.091306 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:42.091288    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-5kjlz_openshift-console-operator(75f2a1f7-95a7-462e-9ea7-23c51dd29599)\"" pod="openshift-console-operator/console-operator-fd4bb46c-5kjlz" podUID="75f2a1f7-95a7-462e-9ea7-23c51dd29599"
Sep 18 17:50:42.204700 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:42.204664    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6e4028c2-33ee-41f9-bb98-a4bbdffe8b24-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cjzln\" (UID: \"6e4028c2-33ee-41f9-bb98-a4bbdffe8b24\") " pod="openshift-insights/insights-runtime-extractor-cjzln"
Sep 18 17:50:42.204871 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:42.204780    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6e4028c2-33ee-41f9-bb98-a4bbdffe8b24-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cjzln\" (UID: \"6e4028c2-33ee-41f9-bb98-a4bbdffe8b24\") " pod="openshift-insights/insights-runtime-extractor-cjzln"
Sep 18 17:50:42.204871 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:42.204781    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 17:50:42.204871 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:42.204854    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6e4028c2-33ee-41f9-bb98-a4bbdffe8b24-kube-rbac-proxy-cm podName:6e4028c2-33ee-41f9-bb98-a4bbdffe8b24 nodeName:}" failed. No retries permitted until 2026-09-18 17:50:43.204833576 +0000 UTC m=+47.831965194 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6e4028c2-33ee-41f9-bb98-a4bbdffe8b24-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cjzln" (UID: "6e4028c2-33ee-41f9-bb98-a4bbdffe8b24") : configmap "kube-rbac-proxy" not found
Sep 18 17:50:42.205034 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:42.204885    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 17:50:42.205034 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:42.204932    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e4028c2-33ee-41f9-bb98-a4bbdffe8b24-insights-runtime-extractor-tls podName:6e4028c2-33ee-41f9-bb98-a4bbdffe8b24 nodeName:}" failed. No retries permitted until 2026-09-18 17:50:43.204917863 +0000 UTC m=+47.832049486 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6e4028c2-33ee-41f9-bb98-a4bbdffe8b24-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cjzln" (UID: "6e4028c2-33ee-41f9-bb98-a4bbdffe8b24") : secret "insights-runtime-extractor-tls" not found
Sep 18 17:50:43.213851 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:43.213815    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6e4028c2-33ee-41f9-bb98-a4bbdffe8b24-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cjzln\" (UID: \"6e4028c2-33ee-41f9-bb98-a4bbdffe8b24\") " pod="openshift-insights/insights-runtime-extractor-cjzln"
Sep 18 17:50:43.214237 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:43.213938    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 17:50:43.214237 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:43.214009    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6e4028c2-33ee-41f9-bb98-a4bbdffe8b24-kube-rbac-proxy-cm podName:6e4028c2-33ee-41f9-bb98-a4bbdffe8b24 nodeName:}" failed. No retries permitted until 2026-09-18 17:50:45.213994389 +0000 UTC m=+49.841125987 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6e4028c2-33ee-41f9-bb98-a4bbdffe8b24-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cjzln" (UID: "6e4028c2-33ee-41f9-bb98-a4bbdffe8b24") : configmap "kube-rbac-proxy" not found
Sep 18 17:50:43.214237 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:43.214045    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6e4028c2-33ee-41f9-bb98-a4bbdffe8b24-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cjzln\" (UID: \"6e4028c2-33ee-41f9-bb98-a4bbdffe8b24\") " pod="openshift-insights/insights-runtime-extractor-cjzln"
Sep 18 17:50:43.214237 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:43.214203    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 17:50:43.214396 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:43.214271    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e4028c2-33ee-41f9-bb98-a4bbdffe8b24-insights-runtime-extractor-tls podName:6e4028c2-33ee-41f9-bb98-a4bbdffe8b24 nodeName:}" failed. No retries permitted until 2026-09-18 17:50:45.214260314 +0000 UTC m=+49.841391912 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6e4028c2-33ee-41f9-bb98-a4bbdffe8b24-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cjzln" (UID: "6e4028c2-33ee-41f9-bb98-a4bbdffe8b24") : secret "insights-runtime-extractor-tls" not found
Sep 18 17:50:43.337705 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:43.337678    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:50:44.209311 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:44.209285    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-q2btm_1907e944-be36-42b2-bc90-90938a227371/dns-node-resolver/0.log"
Sep 18 17:50:44.390519 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:44.390490    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:50:45.235088 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:45.235047    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6e4028c2-33ee-41f9-bb98-a4bbdffe8b24-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cjzln\" (UID: \"6e4028c2-33ee-41f9-bb98-a4bbdffe8b24\") " pod="openshift-insights/insights-runtime-extractor-cjzln"
Sep 18 17:50:45.235303 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:45.235184    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 17:50:45.235303 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:45.235203    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6e4028c2-33ee-41f9-bb98-a4bbdffe8b24-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cjzln\" (UID: \"6e4028c2-33ee-41f9-bb98-a4bbdffe8b24\") " pod="openshift-insights/insights-runtime-extractor-cjzln"
Sep 18 17:50:45.235303 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:45.235271    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6e4028c2-33ee-41f9-bb98-a4bbdffe8b24-kube-rbac-proxy-cm podName:6e4028c2-33ee-41f9-bb98-a4bbdffe8b24 nodeName:}" failed. No retries permitted until 2026-09-18 17:50:49.235252185 +0000 UTC m=+53.862383782 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6e4028c2-33ee-41f9-bb98-a4bbdffe8b24-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cjzln" (UID: "6e4028c2-33ee-41f9-bb98-a4bbdffe8b24") : configmap "kube-rbac-proxy" not found
Sep 18 17:50:45.235422 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:45.235351    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 17:50:45.235422 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:45.235405    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e4028c2-33ee-41f9-bb98-a4bbdffe8b24-insights-runtime-extractor-tls podName:6e4028c2-33ee-41f9-bb98-a4bbdffe8b24 nodeName:}" failed. No retries permitted until 2026-09-18 17:50:49.235392697 +0000 UTC m=+53.862524295 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6e4028c2-33ee-41f9-bb98-a4bbdffe8b24-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cjzln" (UID: "6e4028c2-33ee-41f9-bb98-a4bbdffe8b24") : secret "insights-runtime-extractor-tls" not found
Sep 18 17:50:45.411420 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:45.411397    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zvmrv_4945651f-6aa7-4ce0-82e2-3f5f234b607f/node-ca/0.log"
Sep 18 17:50:45.537544 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:45.537464    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a1e4f37f-b0ba-469f-8923-d0c9d38e619b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-xjzc9\" (UID: \"a1e4f37f-b0ba-469f-8923-d0c9d38e619b\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xjzc9"
Sep 18 17:50:45.537544 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:45.537518    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20117f3e-e50b-4b03-aa48-604fc42d1737-metrics-certs\") pod \"router-default-7dc96485b6-66p6f\" (UID: \"20117f3e-e50b-4b03-aa48-604fc42d1737\") " pod="openshift-ingress/router-default-7dc96485b6-66p6f"
Sep 18 17:50:45.537544 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:45.537541    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1096ace1-2554-43bc-bf2e-5e6d09e1c48f-registry-tls\") pod \"image-registry-78c879b97f-xtp8m\" (UID: \"1096ace1-2554-43bc-bf2e-5e6d09e1c48f\") " pod="openshift-image-registry/image-registry-78c879b97f-xtp8m"
Sep 18 17:50:45.537816 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:45.537564    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20117f3e-e50b-4b03-aa48-604fc42d1737-service-ca-bundle\") pod \"router-default-7dc96485b6-66p6f\" (UID: \"20117f3e-e50b-4b03-aa48-604fc42d1737\") " pod="openshift-ingress/router-default-7dc96485b6-66p6f"
Sep 18 17:50:45.537816 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:45.537608    2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 17:50:45.537816 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:45.537654    2561 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 17:50:45.537816 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:45.537668    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1e4f37f-b0ba-469f-8923-d0c9d38e619b-cluster-monitoring-operator-tls podName:a1e4f37f-b0ba-469f-8923-d0c9d38e619b nodeName:}" failed. No retries permitted until 2026-09-18 17:51:01.537653452 +0000 UTC m=+66.164785051 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a1e4f37f-b0ba-469f-8923-d0c9d38e619b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-xjzc9" (UID: "a1e4f37f-b0ba-469f-8923-d0c9d38e619b") : secret "cluster-monitoring-operator-tls" not found
Sep 18 17:50:45.537816 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:45.537658    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 17:50:45.537816 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:45.537697    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20117f3e-e50b-4b03-aa48-604fc42d1737-metrics-certs podName:20117f3e-e50b-4b03-aa48-604fc42d1737 nodeName:}" failed. No retries permitted until 2026-09-18 17:51:01.537684856 +0000 UTC m=+66.164816453 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/20117f3e-e50b-4b03-aa48-604fc42d1737-metrics-certs") pod "router-default-7dc96485b6-66p6f" (UID: "20117f3e-e50b-4b03-aa48-604fc42d1737") : secret "router-metrics-certs-default" not found
Sep 18 17:50:45.537816 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:45.537707    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78c879b97f-xtp8m: secret "image-registry-tls" not found
Sep 18 17:50:45.537816 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:45.537745    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/20117f3e-e50b-4b03-aa48-604fc42d1737-service-ca-bundle podName:20117f3e-e50b-4b03-aa48-604fc42d1737 nodeName:}" failed. No retries permitted until 2026-09-18 17:51:01.537722975 +0000 UTC m=+66.164854604 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/20117f3e-e50b-4b03-aa48-604fc42d1737-service-ca-bundle") pod "router-default-7dc96485b6-66p6f" (UID: "20117f3e-e50b-4b03-aa48-604fc42d1737") : configmap references non-existent config key: service-ca.crt
Sep 18 17:50:45.537816 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:45.537794    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1096ace1-2554-43bc-bf2e-5e6d09e1c48f-registry-tls podName:1096ace1-2554-43bc-bf2e-5e6d09e1c48f nodeName:}" failed. No retries permitted until 2026-09-18 17:51:01.537760839 +0000 UTC m=+66.164892439 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1096ace1-2554-43bc-bf2e-5e6d09e1c48f-registry-tls") pod "image-registry-78c879b97f-xtp8m" (UID: "1096ace1-2554-43bc-bf2e-5e6d09e1c48f") : secret "image-registry-tls" not found
Sep 18 17:50:45.583592 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:45.583552    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:50:45.638875 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:45.638846    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1286716b-4028-484e-b51d-58eb4b9f7305-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-qd9mb\" (UID: \"1286716b-4028-484e-b51d-58eb4b9f7305\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-qd9mb"
Sep 18 17:50:45.639036 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:45.638936    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/60a031b6-31f6-47a2-859c-bfd960f0b2c7-metrics-tls\") pod \"dns-default-9hjjd\" (UID: \"60a031b6-31f6-47a2-859c-bfd960f0b2c7\") " pod="openshift-dns/dns-default-9hjjd"
Sep 18 17:50:45.639036 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:45.638995    2561 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 17:50:45.639036 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:45.639015    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 17:50:45.639178 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:45.639053    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60a031b6-31f6-47a2-859c-bfd960f0b2c7-metrics-tls podName:60a031b6-31f6-47a2-859c-bfd960f0b2c7 nodeName:}" failed. No retries permitted until 2026-09-18 17:51:01.639041829 +0000 UTC m=+66.266173427 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/60a031b6-31f6-47a2-859c-bfd960f0b2c7-metrics-tls") pod "dns-default-9hjjd" (UID: "60a031b6-31f6-47a2-859c-bfd960f0b2c7") : secret "dns-default-metrics-tls" not found
Sep 18 17:50:45.639178 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:45.639066    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1286716b-4028-484e-b51d-58eb4b9f7305-samples-operator-tls podName:1286716b-4028-484e-b51d-58eb4b9f7305 nodeName:}" failed. No retries permitted until 2026-09-18 17:51:01.63905945 +0000 UTC m=+66.266191047 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1286716b-4028-484e-b51d-58eb4b9f7305-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-qd9mb" (UID: "1286716b-4028-484e-b51d-58eb4b9f7305") : secret "samples-operator-tls" not found
Sep 18 17:50:46.008390 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:46.008365    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kw7df_ac028d85-f1a2-4c5a-91ad-fb7141335c07/serve-healthcheck-canary/0.log"
Sep 18 17:50:46.810531 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:46.810501    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-dx752_26ece3e7-900c-4efe-b497-653cf58b5e60/kube-storage-version-migrator-operator/0.log"
Sep 18 17:50:47.456906 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:47.456872    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:50:47.686246 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:47.686198    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:50:47.917817 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:47.917787    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:50:48.175774 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:48.175699    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:50:48.280945 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:48.280917    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:50:48.596845 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:48.596745    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:50:49.269331 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:49.269295    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6e4028c2-33ee-41f9-bb98-a4bbdffe8b24-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cjzln\" (UID: \"6e4028c2-33ee-41f9-bb98-a4bbdffe8b24\") " pod="openshift-insights/insights-runtime-extractor-cjzln"
Sep 18 17:50:49.269813 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:49.269389    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6e4028c2-33ee-41f9-bb98-a4bbdffe8b24-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cjzln\" (UID: \"6e4028c2-33ee-41f9-bb98-a4bbdffe8b24\") " pod="openshift-insights/insights-runtime-extractor-cjzln"
Sep 18 17:50:49.269813 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:49.269525    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 17:50:49.269813 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:49.269590    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e4028c2-33ee-41f9-bb98-a4bbdffe8b24-insights-runtime-extractor-tls podName:6e4028c2-33ee-41f9-bb98-a4bbdffe8b24 nodeName:}" failed. No retries permitted until 2026-09-18 17:50:57.269575827 +0000 UTC m=+61.896707424 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6e4028c2-33ee-41f9-bb98-a4bbdffe8b24-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cjzln" (UID: "6e4028c2-33ee-41f9-bb98-a4bbdffe8b24") : secret "insights-runtime-extractor-tls" not found
Sep 18 17:50:49.270006 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:49.269934    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6e4028c2-33ee-41f9-bb98-a4bbdffe8b24-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cjzln\" (UID: \"6e4028c2-33ee-41f9-bb98-a4bbdffe8b24\") " pod="openshift-insights/insights-runtime-extractor-cjzln"
Sep 18 17:50:49.387566 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:49.387525    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:50:49.718087 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:49.718058    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:50:50.016445 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:50.016347    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:50:50.068043 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:50.068018    2561 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-5kjlz"
Sep 18 17:50:50.068175 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:50.068066    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-5kjlz"
Sep 18 17:50:50.068379 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:50.068367    2561 scope.go:117] "RemoveContainer" containerID="33b362ee52182f27fdd4b5f2ccd4b533faa50dc5e417f5cf8c1bcd11ba242883"
Sep 18 17:50:50.068526 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:50:50.068507    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-5kjlz_openshift-console-operator(75f2a1f7-95a7-462e-9ea7-23c51dd29599)\"" pod="openshift-console-operator/console-operator-fd4bb46c-5kjlz" podUID="75f2a1f7-95a7-462e-9ea7-23c51dd29599"
Sep 18 17:50:50.198913 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:50.198867    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:50:50.809864 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:50.809833    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:50:50.979984 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:50.979956    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:50:51.220437 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:51.220407    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:50:51.618388 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:51.618309    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:50:51.722948 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:51.722921    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:50:51.834455 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:51.834399    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:50:52.380751 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:52.380722    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:50:52.453634 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:52.453604    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:50:52.847897 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:52.847868    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:50:52.986488 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:52.986461    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:50:53.090488 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:53.090459    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:50:53.199012 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:53.198984    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:50:53.600303 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:53.600204    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:50:54.172442 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:54.172385    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:50:54.372450 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:54.372411    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:50:54.585382 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:54.585299    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:50:54.644660 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:54.644630    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:50:54.768089 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:54.768036    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:50:55.494285 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:55.494254    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:50:56.016533 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:56.016494    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:50:56.187349 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:56.187317    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:50:56.400975 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:56.400912    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:50:56.580925 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:56.580897    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:50:56.891012 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:56.890874    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:50:56.991176 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:56.991154    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rkcgk"
Sep 18 17:50:57.341444 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:57.341397    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6e4028c2-33ee-41f9-bb98-a4bbdffe8b24-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cjzln\" (UID: \"6e4028c2-33ee-41f9-bb98-a4bbdffe8b24\") " pod="openshift-insights/insights-runtime-extractor-cjzln"
Sep 18 17:50:57.343643 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:57.343624    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6e4028c2-33ee-41f9-bb98-a4bbdffe8b24-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cjzln\" (UID: \"6e4028c2-33ee-41f9-bb98-a4bbdffe8b24\") " pod="openshift-insights/insights-runtime-extractor-cjzln"
Sep 18 17:50:57.379906 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:57.379876    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:50:57.391437 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:57.391419    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vmk8k\""
Sep 18 17:50:57.399412 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:57.399392    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cjzln"
Sep 18 17:50:57.658939 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:57.658907    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:50:57.680473 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:57.680448    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-cjzln"]
Sep 18 17:50:57.681421 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:50:57.681394    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e4028c2_33ee_41f9_bb98_a4bbdffe8b24.slice/crio-a3d022bdf2d48397683278db5d0dc44d41949a332175eed9f6863079d055ff4b WatchSource:0}: Error finding container a3d022bdf2d48397683278db5d0dc44d41949a332175eed9f6863079d055ff4b: Status 404 returned error can't find the container with id a3d022bdf2d48397683278db5d0dc44d41949a332175eed9f6863079d055ff4b
Sep 18 17:50:57.980509 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:57.980424    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:50:58.138970 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:58.138934    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cjzln" event={"ID":"6e4028c2-33ee-41f9-bb98-a4bbdffe8b24","Type":"ContainerStarted","Data":"6591cf87e3fb9967dc855880a81d0387dded0e37472e0262d7aeed8ca8d18e47"}
Sep 18 17:50:58.139145 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:58.138992    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cjzln" event={"ID":"6e4028c2-33ee-41f9-bb98-a4bbdffe8b24","Type":"ContainerStarted","Data":"a3d022bdf2d48397683278db5d0dc44d41949a332175eed9f6863079d055ff4b"}
Sep 18 17:50:59.144624 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:50:59.144587    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cjzln" event={"ID":"6e4028c2-33ee-41f9-bb98-a4bbdffe8b24","Type":"ContainerStarted","Data":"9cc34f1ba70d23026249e14e9ad390a6f35996cfe34847bc8c9da2db5b270b79"}
Sep 18 17:51:00.149911 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:00.149880    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cjzln" event={"ID":"6e4028c2-33ee-41f9-bb98-a4bbdffe8b24","Type":"ContainerStarted","Data":"533016a4fa2e2f66b3f54922c2fd84b5dc290efad2da3823124f953a34e38db9"}
Sep 18 17:51:00.169060 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:00.169016    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-cjzln" podStartSLOduration=17.214287428 podStartE2EDuration="19.169001232s" totalImagesPullingTime="1.954713804s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 17:50:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:51:00.168593782 +0000 UTC m=+64.795725403" watchObservedRunningTime="2026-09-18 17:51:00.169001232 +0000 UTC m=+64.796132852"
Sep 18 17:51:01.479212 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:01.479171    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45a7fd6c-fe96-4a1a-b3a5-2cb78c5435e8-metrics-certs\") pod \"network-metrics-daemon-tmtsv\" (UID: \"45a7fd6c-fe96-4a1a-b3a5-2cb78c5435e8\") " pod="openshift-multus/network-metrics-daemon-tmtsv"
Sep 18 17:51:01.482448 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:01.482428    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 17:51:01.491817 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:01.491793    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45a7fd6c-fe96-4a1a-b3a5-2cb78c5435e8-metrics-certs\") pod \"network-metrics-daemon-tmtsv\" (UID: \"45a7fd6c-fe96-4a1a-b3a5-2cb78c5435e8\") " pod="openshift-multus/network-metrics-daemon-tmtsv"
Sep 18 17:51:01.580381 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:01.580349    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scbjn\" (UniqueName: \"kubernetes.io/projected/234003fa-8d5d-4108-8f62-859ea05eb9aa-kube-api-access-scbjn\") pod \"network-check-target-bgwtf\" (UID: \"234003fa-8d5d-4108-8f62-859ea05eb9aa\") " pod="openshift-network-diagnostics/network-check-target-bgwtf"
Sep 18 17:51:01.580495 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:01.580386    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7a3ae2d8-a30b-4c83-a53c-634fb07621d6-original-pull-secret\") pod \"global-pull-secret-syncer-mvtdm\" (UID: \"7a3ae2d8-a30b-4c83-a53c-634fb07621d6\") " pod="kube-system/global-pull-secret-syncer-mvtdm"
Sep 18 17:51:01.580495 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:01.580417    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a1e4f37f-b0ba-469f-8923-d0c9d38e619b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-xjzc9\" (UID: \"a1e4f37f-b0ba-469f-8923-d0c9d38e619b\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xjzc9"
Sep 18 17:51:01.580495 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:01.580466    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20117f3e-e50b-4b03-aa48-604fc42d1737-metrics-certs\") pod \"router-default-7dc96485b6-66p6f\" (UID: \"20117f3e-e50b-4b03-aa48-604fc42d1737\") " pod="openshift-ingress/router-default-7dc96485b6-66p6f"
Sep 18 17:51:01.580666 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:01.580504    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1096ace1-2554-43bc-bf2e-5e6d09e1c48f-registry-tls\") pod \"image-registry-78c879b97f-xtp8m\" (UID: \"1096ace1-2554-43bc-bf2e-5e6d09e1c48f\") " pod="openshift-image-registry/image-registry-78c879b97f-xtp8m"
Sep 18 17:51:01.580666 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:01.580528    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20117f3e-e50b-4b03-aa48-604fc42d1737-service-ca-bundle\") pod \"router-default-7dc96485b6-66p6f\" (UID: \"20117f3e-e50b-4b03-aa48-604fc42d1737\") " pod="openshift-ingress/router-default-7dc96485b6-66p6f"
Sep 18 17:51:01.581257 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:01.581231    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20117f3e-e50b-4b03-aa48-604fc42d1737-service-ca-bundle\") pod \"router-default-7dc96485b6-66p6f\" (UID: \"20117f3e-e50b-4b03-aa48-604fc42d1737\") " pod="openshift-ingress/router-default-7dc96485b6-66p6f"
Sep 18 17:51:01.582955 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:01.582923    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a1e4f37f-b0ba-469f-8923-d0c9d38e619b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-xjzc9\" (UID: \"a1e4f37f-b0ba-469f-8923-d0c9d38e619b\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xjzc9"
Sep 18 17:51:01.583069 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:01.582975    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20117f3e-e50b-4b03-aa48-604fc42d1737-metrics-certs\") pod \"router-default-7dc96485b6-66p6f\" (UID: \"20117f3e-e50b-4b03-aa48-604fc42d1737\") " pod="openshift-ingress/router-default-7dc96485b6-66p6f"
Sep 18 17:51:01.583069 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:01.583025    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1096ace1-2554-43bc-bf2e-5e6d09e1c48f-registry-tls\") pod \"image-registry-78c879b97f-xtp8m\" (UID: \"1096ace1-2554-43bc-bf2e-5e6d09e1c48f\") " pod="openshift-image-registry/image-registry-78c879b97f-xtp8m"
Sep 18 17:51:01.583195 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:01.583108    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scbjn\" (UniqueName: \"kubernetes.io/projected/234003fa-8d5d-4108-8f62-859ea05eb9aa-kube-api-access-scbjn\") pod \"network-check-target-bgwtf\" (UID: \"234003fa-8d5d-4108-8f62-859ea05eb9aa\") " pod="openshift-network-diagnostics/network-check-target-bgwtf"
Sep 18 17:51:01.583718 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:01.583701    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 17:51:01.592244 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:01.592224    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7a3ae2d8-a30b-4c83-a53c-634fb07621d6-original-pull-secret\") pod \"global-pull-secret-syncer-mvtdm\" (UID: \"7a3ae2d8-a30b-4c83-a53c-634fb07621d6\") " pod="kube-system/global-pull-secret-syncer-mvtdm"
Sep 18 17:51:01.681840 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:01.681801    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/60a031b6-31f6-47a2-859c-bfd960f0b2c7-metrics-tls\") pod \"dns-default-9hjjd\" (UID: \"60a031b6-31f6-47a2-859c-bfd960f0b2c7\") " pod="openshift-dns/dns-default-9hjjd"
Sep 18 17:51:01.681980 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:01.681865    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1286716b-4028-484e-b51d-58eb4b9f7305-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-qd9mb\" (UID: \"1286716b-4028-484e-b51d-58eb4b9f7305\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-qd9mb"
Sep 18 17:51:01.683983 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:01.683965    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/60a031b6-31f6-47a2-859c-bfd960f0b2c7-metrics-tls\") pod \"dns-default-9hjjd\" (UID: \"60a031b6-31f6-47a2-859c-bfd960f0b2c7\") " pod="openshift-dns/dns-default-9hjjd"
Sep 18 17:51:01.684217 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:01.684200    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1286716b-4028-484e-b51d-58eb4b9f7305-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-qd9mb\" (UID: \"1286716b-4028-484e-b51d-58eb4b9f7305\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-qd9mb"
Sep 18 17:51:01.770235 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:01.770176    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-r5l4c\""
Sep 18 17:51:01.775860 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:01.775845    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sdsc5\""
Sep 18 17:51:01.777703 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:01.777691    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bgwtf"
Sep 18 17:51:01.778830 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:01.778815    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mvtdm"
Sep 18 17:51:01.784011 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:01.784000    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tmtsv"
Sep 18 17:51:01.816303 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:01.816285    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xssnq\""
Sep 18 17:51:01.823831 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:01.823810    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78c879b97f-xtp8m"
Sep 18 17:51:01.823923 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:01.823835    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-wffwh\""
Sep 18 17:51:01.833811 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:01.833786    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7dc96485b6-66p6f"
Sep 18 17:51:01.841861 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:01.841827    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-tzxvx\""
Sep 18 17:51:01.850300 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:01.850246    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xjzc9"
Sep 18 17:51:01.854296 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:01.853954    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-s9cmf\""
Sep 18 17:51:01.855631 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:01.855609    2561 scope.go:117] "RemoveContainer" containerID="33b362ee52182f27fdd4b5f2ccd4b533faa50dc5e417f5cf8c1bcd11ba242883"
Sep 18 17:51:01.862212 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:01.862166    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9hjjd"
Sep 18 17:51:01.872643 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:01.872611    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-qtjrw\""
Sep 18 17:51:01.880438 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:01.880375    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-qd9mb"
Sep 18 17:51:02.082104 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:02.082067    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bgwtf"]
Sep 18 17:51:02.085714 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:51:02.085687    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod234003fa_8d5d_4108_8f62_859ea05eb9aa.slice/crio-6e840b87d66d258db901d06031d0208864d26b53f9029030e4fed870820ff067 WatchSource:0}: Error finding container 6e840b87d66d258db901d06031d0208864d26b53f9029030e4fed870820ff067: Status 404 returned error can't find the container with id 6e840b87d66d258db901d06031d0208864d26b53f9029030e4fed870820ff067
Sep 18 17:51:02.118546 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:02.118521    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mvtdm"]
Sep 18 17:51:02.121299 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:51:02.121273    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a3ae2d8_a30b_4c83_a53c_634fb07621d6.slice/crio-4d7bb5eee3b8bb6981fcb03b0c62d12fdbb4cf242b0431d0c339bbbdd6c0dcbf WatchSource:0}: Error finding container 4d7bb5eee3b8bb6981fcb03b0c62d12fdbb4cf242b0431d0c339bbbdd6c0dcbf: Status 404 returned error can't find the container with id 4d7bb5eee3b8bb6981fcb03b0c62d12fdbb4cf242b0431d0c339bbbdd6c0dcbf
Sep 18 17:51:02.154095 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:02.154073    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-78c879b97f-xtp8m"]
Sep 18 17:51:02.157270 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:51:02.157249    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1096ace1_2554_43bc_bf2e_5e6d09e1c48f.slice/crio-880cf5d1aa08d2b1319e109235fd3d92e860703cffb8e5c8449c063e66c459da WatchSource:0}: Error finding container 880cf5d1aa08d2b1319e109235fd3d92e860703cffb8e5c8449c063e66c459da: Status 404 returned error can't find the container with id 880cf5d1aa08d2b1319e109235fd3d92e860703cffb8e5c8449c063e66c459da
Sep 18 17:51:02.158401 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:02.158378    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mvtdm" event={"ID":"7a3ae2d8-a30b-4c83-a53c-634fb07621d6","Type":"ContainerStarted","Data":"4d7bb5eee3b8bb6981fcb03b0c62d12fdbb4cf242b0431d0c339bbbdd6c0dcbf"}
Sep 18 17:51:02.159474 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:02.159454    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bgwtf" event={"ID":"234003fa-8d5d-4108-8f62-859ea05eb9aa","Type":"ContainerStarted","Data":"6e840b87d66d258db901d06031d0208864d26b53f9029030e4fed870820ff067"}
Sep 18 17:51:02.161306 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:02.161289    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-5kjlz_75f2a1f7-95a7-462e-9ea7-23c51dd29599/console-operator/1.log"
Sep 18 17:51:02.161373 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:02.161334    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-5kjlz" event={"ID":"75f2a1f7-95a7-462e-9ea7-23c51dd29599","Type":"ContainerStarted","Data":"66ca3b2cdd0c340fb2bd61a6492d6808febdca8c33849d51e701c6356c05b4db"}
Sep 18 17:51:02.190548 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:02.190501    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-5kjlz"
Sep 18 17:51:02.206722 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:02.206699    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tmtsv"]
Sep 18 17:51:02.214184 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:51:02.214150    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45a7fd6c_fe96_4a1a_b3a5_2cb78c5435e8.slice/crio-b18023f34e084bdd2f51f866ec742d373e7f75a9044876e1031a54ca2fe4cb19 WatchSource:0}: Error finding container b18023f34e084bdd2f51f866ec742d373e7f75a9044876e1031a54ca2fe4cb19: Status 404 returned error can't find the container with id b18023f34e084bdd2f51f866ec742d373e7f75a9044876e1031a54ca2fe4cb19
Sep 18 17:51:02.217982 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:02.217927    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7dc96485b6-66p6f"]
Sep 18 17:51:02.218829 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:02.218784    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-fd4bb46c-5kjlz" podStartSLOduration=33.345228407 podStartE2EDuration="42.218766205s" totalImagesPullingTime="8.873537798s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:50:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:51:02.2148601 +0000 UTC m=+66.841991743" watchObservedRunningTime="2026-09-18 17:51:02.218766205 +0000 UTC m=+66.845897826"
Sep 18 17:51:02.219750 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:51:02.219726    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20117f3e_e50b_4b03_aa48_604fc42d1737.slice/crio-ba8658deaec84089b1156f1230aa54e7877d23d1fcfa612ff72ae9702b447f9e WatchSource:0}: Error finding container ba8658deaec84089b1156f1230aa54e7877d23d1fcfa612ff72ae9702b447f9e: Status 404 returned error can't find the container with id ba8658deaec84089b1156f1230aa54e7877d23d1fcfa612ff72ae9702b447f9e
Sep 18 17:51:02.253806 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:02.253788    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xjzc9"]
Sep 18 17:51:02.258514 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:51:02.258493    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1e4f37f_b0ba_469f_8923_d0c9d38e619b.slice/crio-be59c20b8be9646fe305a3791ea1fdd1e144f321093a1f5346b2247bb7f75151 WatchSource:0}: Error finding container be59c20b8be9646fe305a3791ea1fdd1e144f321093a1f5346b2247bb7f75151: Status 404 returned error can't find the container with id be59c20b8be9646fe305a3791ea1fdd1e144f321093a1f5346b2247bb7f75151
Sep 18 17:51:02.270784 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:02.270733    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-qd9mb"]
Sep 18 17:51:02.282493 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:02.282468    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9hjjd"]
Sep 18 17:51:02.286312 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:51:02.286278    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60a031b6_31f6_47a2_859c_bfd960f0b2c7.slice/crio-4cc6142fc257e9080632f077edda3d62120ea4cf37ec4619df8f5eae7ba7ea1c WatchSource:0}: Error finding container 4cc6142fc257e9080632f077edda3d62120ea4cf37ec4619df8f5eae7ba7ea1c: Status 404 returned error can't find the container with id 4cc6142fc257e9080632f077edda3d62120ea4cf37ec4619df8f5eae7ba7ea1c
Sep 18 17:51:02.450059 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:02.450022    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:02.548402 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:02.548323    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:02.771363 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:02.771330    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-fd4bb46c-5kjlz"
Sep 18 17:51:03.051587 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:03.051554    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:03.171418 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:03.171302    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9hjjd" event={"ID":"60a031b6-31f6-47a2-859c-bfd960f0b2c7","Type":"ContainerStarted","Data":"4cc6142fc257e9080632f077edda3d62120ea4cf37ec4619df8f5eae7ba7ea1c"}
Sep 18 17:51:03.174890 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:03.174827    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:03.175956 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:03.175422    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bgwtf" event={"ID":"234003fa-8d5d-4108-8f62-859ea05eb9aa","Type":"ContainerStarted","Data":"217516b27236b2026bad6ced347c47c01e73e709fa085c811414c0919d98712e"}
Sep 18 17:51:03.175956 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:03.175730    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-bgwtf"
Sep 18 17:51:03.177937 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:03.177911    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xjzc9" event={"ID":"a1e4f37f-b0ba-469f-8923-d0c9d38e619b","Type":"ContainerStarted","Data":"be59c20b8be9646fe305a3791ea1fdd1e144f321093a1f5346b2247bb7f75151"}
Sep 18 17:51:03.180496 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:03.180466    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-qd9mb" event={"ID":"1286716b-4028-484e-b51d-58eb4b9f7305","Type":"ContainerStarted","Data":"40d26c86d8efd264c6f0c3dce3d7b3cb86a557f8ee637b3a3f5c5a89abc451ad"}
Sep 18 17:51:03.182540 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:03.182515    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7dc96485b6-66p6f" event={"ID":"20117f3e-e50b-4b03-aa48-604fc42d1737","Type":"ContainerStarted","Data":"45f036aec24f11bb5178b3058c5bade98be84b85815552caad2017fee8de5eb7"}
Sep 18 17:51:03.182633 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:03.182557    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7dc96485b6-66p6f" event={"ID":"20117f3e-e50b-4b03-aa48-604fc42d1737","Type":"ContainerStarted","Data":"ba8658deaec84089b1156f1230aa54e7877d23d1fcfa612ff72ae9702b447f9e"}
Sep 18 17:51:03.185416 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:03.185393    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tmtsv" event={"ID":"45a7fd6c-fe96-4a1a-b3a5-2cb78c5435e8","Type":"ContainerStarted","Data":"b18023f34e084bdd2f51f866ec742d373e7f75a9044876e1031a54ca2fe4cb19"}
Sep 18 17:51:03.188655 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:03.188632    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-78c879b97f-xtp8m" event={"ID":"1096ace1-2554-43bc-bf2e-5e6d09e1c48f","Type":"ContainerStarted","Data":"c82be5f9159a1a35c7b7f1681894fd2ec9465c9b5e6b4a127bbb43639fb27a79"}
Sep 18 17:51:03.188746 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:03.188673    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-78c879b97f-xtp8m" event={"ID":"1096ace1-2554-43bc-bf2e-5e6d09e1c48f","Type":"ContainerStarted","Data":"880cf5d1aa08d2b1319e109235fd3d92e860703cffb8e5c8449c063e66c459da"}
Sep 18 17:51:03.188746 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:03.188696    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-78c879b97f-xtp8m"
Sep 18 17:51:03.216801 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:03.216166    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-bgwtf" podStartSLOduration=67.216146548 podStartE2EDuration="1m7.216146548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:49:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:51:03.215097576 +0000 UTC m=+67.842229196" watchObservedRunningTime="2026-09-18 17:51:03.216146548 +0000 UTC m=+67.843278184"
Sep 18 17:51:03.307775 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:03.307671    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-7dc96485b6-66p6f" podStartSLOduration=43.307649757 podStartE2EDuration="43.307649757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:50:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:51:03.272397076 +0000 UTC m=+67.899528698" watchObservedRunningTime="2026-09-18 17:51:03.307649757 +0000 UTC m=+67.934781379"
Sep 18 17:51:03.331993 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:03.331960    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:03.431661 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:03.431586    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:03.532139 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:03.532060    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:03.834388 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:03.834363    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7dc96485b6-66p6f"
Sep 18 17:51:03.838533 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:03.838337    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7dc96485b6-66p6f"
Sep 18 17:51:03.863951 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:03.861626    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-78c879b97f-xtp8m" podStartSLOduration=66.861606611 podStartE2EDuration="1m6.861606611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:49:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:51:03.307148737 +0000 UTC m=+67.934280351" watchObservedRunningTime="2026-09-18 17:51:03.861606611 +0000 UTC m=+68.488738232"
Sep 18 17:51:03.864467 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:03.864393    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:03.933307 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:03.933236    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:04.039932 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:04.039900    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:04.144573 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:04.144535    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:04.197246 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:04.195874    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tmtsv" event={"ID":"45a7fd6c-fe96-4a1a-b3a5-2cb78c5435e8","Type":"ContainerStarted","Data":"8c1db0515224de4d5ca2315f0aad7b2474d157fcdc96afcf6c3c4c9590bf5f74"}
Sep 18 17:51:04.197246 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:04.195935    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tmtsv" event={"ID":"45a7fd6c-fe96-4a1a-b3a5-2cb78c5435e8","Type":"ContainerStarted","Data":"5f78cdd96a1d2e1e8797349b5d3dfdc5c1d1543fd8e9dd1c62fa488fc4bb5aa3"}
Sep 18 17:51:04.197246 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:04.196871    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7dc96485b6-66p6f"
Sep 18 17:51:04.197804 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:04.197757    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7dc96485b6-66p6f"
Sep 18 17:51:04.225433 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:04.225393    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-tmtsv" podStartSLOduration=66.665960847 podStartE2EDuration="1m8.225378218s" totalImagesPullingTime="1.559417371s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:49:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:51:04.223409633 +0000 UTC m=+68.850541255" watchObservedRunningTime="2026-09-18 17:51:04.225378218 +0000 UTC m=+68.852509837"
Sep 18 17:51:04.718693 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:04.718629    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:04.844604 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:04.844552    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:04.965334 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:04.965285    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:05.070874 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:05.070783    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:05.196060 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:05.196030    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:05.883547 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:05.883466    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:05.984886 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:05.984854    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:06.645766 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:06.645723    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:06.744796 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:06.744763    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:07.212332 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:07.212078    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xjzc9" event={"ID":"a1e4f37f-b0ba-469f-8923-d0c9d38e619b","Type":"ContainerStarted","Data":"59294a976403ae8e6fb9a545b80a28246900ca426a8a856338054693105edcc6"}
Sep 18 17:51:07.214930 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:07.214903    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-qd9mb" event={"ID":"1286716b-4028-484e-b51d-58eb4b9f7305","Type":"ContainerStarted","Data":"38da53e70ab0b1c5ce8f875222d2db5af169e431d6645dbdb8d2c694ef144828"}
Sep 18 17:51:07.216902 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:07.216878    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mvtdm" event={"ID":"7a3ae2d8-a30b-4c83-a53c-634fb07621d6","Type":"ContainerStarted","Data":"60dea62c12e2cd9fc67651ee02526b3081c06be2d50e63f0829f46e7b6462206"}
Sep 18 17:51:07.218767 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:07.218746    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9hjjd" event={"ID":"60a031b6-31f6-47a2-859c-bfd960f0b2c7","Type":"ContainerStarted","Data":"9e544755c7a76d1f46f079d2d31a8e6c9bb6b4ea0bf1acb3484f94443cb1a504"}
Sep 18 17:51:07.240091 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:07.240037    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xjzc9" podStartSLOduration=42.503746679 podStartE2EDuration="47.240017486s" totalImagesPullingTime="4.736270807s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:50:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:51:07.239597517 +0000 UTC m=+71.866729140" watchObservedRunningTime="2026-09-18 17:51:07.240017486 +0000 UTC m=+71.867149106"
Sep 18 17:51:07.308337 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:07.308285    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-mvtdm" podStartSLOduration=65.398840593 podStartE2EDuration="1m10.308266448s" totalImagesPullingTime="4.909425855s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 17:49:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:51:07.306854411 +0000 UTC m=+71.933986033" watchObservedRunningTime="2026-09-18 17:51:07.308266448 +0000 UTC m=+71.935398122"
Sep 18 17:51:07.508692 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:07.508621    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:07.742635 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:07.742596    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:07.938981 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:07.938953    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-qwhfg"]
Sep 18 17:51:07.941658 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:07.941643    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-qwhfg"
Sep 18 17:51:07.946379 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:07.946354    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 18 17:51:07.946494 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:07.946409    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-vlgcb\""
Sep 18 17:51:07.985592 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:07.985567    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-qwhfg"]
Sep 18 17:51:08.039718 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:08.039687    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4bfe00a4-fcbf-4cd2-a1f2-44cbe3af7c13-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-qwhfg\" (UID: \"4bfe00a4-fcbf-4cd2-a1f2-44cbe3af7c13\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-qwhfg"
Sep 18 17:51:08.140470 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:08.140441    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4bfe00a4-fcbf-4cd2-a1f2-44cbe3af7c13-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-qwhfg\" (UID: \"4bfe00a4-fcbf-4cd2-a1f2-44cbe3af7c13\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-qwhfg"
Sep 18 17:51:08.142738 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:08.142710    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4bfe00a4-fcbf-4cd2-a1f2-44cbe3af7c13-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-qwhfg\" (UID: \"4bfe00a4-fcbf-4cd2-a1f2-44cbe3af7c13\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-qwhfg"
Sep 18 17:51:08.150428 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:08.150408    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:08.224533 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:08.224459    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9hjjd" event={"ID":"60a031b6-31f6-47a2-859c-bfd960f0b2c7","Type":"ContainerStarted","Data":"887edeb7aa2fc9e5233e214078ec85836b35ff8e0f2cb82426cf2cfc7225d310"}
Sep 18 17:51:08.224533 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:08.224531    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-9hjjd"
Sep 18 17:51:08.226365 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:08.226340    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-qd9mb" event={"ID":"1286716b-4028-484e-b51d-58eb4b9f7305","Type":"ContainerStarted","Data":"da4c1775ed26f771baf4a8530922cd8a5bb144af492246423fc6d6b409cde87f"}
Sep 18 17:51:08.249728 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:08.249708    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-qwhfg"
Sep 18 17:51:08.275724 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:08.275685    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-9hjjd" podStartSLOduration=34.563537012 podStartE2EDuration="39.275669605s" totalImagesPullingTime="4.712132593s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:50:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:51:08.26898695 +0000 UTC m=+72.896118573" watchObservedRunningTime="2026-09-18 17:51:08.275669605 +0000 UTC m=+72.902801225"
Sep 18 17:51:08.309725 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:08.309669    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-qd9mb" podStartSLOduration=43.624830891 podStartE2EDuration="48.309645751s" totalImagesPullingTime="4.68481486s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:50:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:51:08.308822911 +0000 UTC m=+72.935954531" watchObservedRunningTime="2026-09-18 17:51:08.309645751 +0000 UTC m=+72.936777387"
Sep 18 17:51:08.434645 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:08.434613    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:08.526188 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:08.526082    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-qwhfg"]
Sep 18 17:51:08.529577 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:51:08.529545    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bfe00a4_fcbf_4cd2_a1f2_44cbe3af7c13.slice/crio-f3b64957c35deeb1be7f98ab3e9aa5cb0d81f0c9722e6d2c7a2589c8569b57be WatchSource:0}: Error finding container f3b64957c35deeb1be7f98ab3e9aa5cb0d81f0c9722e6d2c7a2589c8569b57be: Status 404 returned error can't find the container with id f3b64957c35deeb1be7f98ab3e9aa5cb0d81f0c9722e6d2c7a2589c8569b57be
Sep 18 17:51:09.232951 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:09.232892    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-qwhfg" event={"ID":"4bfe00a4-fcbf-4cd2-a1f2-44cbe3af7c13","Type":"ContainerStarted","Data":"f3b64957c35deeb1be7f98ab3e9aa5cb0d81f0c9722e6d2c7a2589c8569b57be"}
Sep 18 17:51:10.238026 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:10.238001    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-qwhfg" event={"ID":"4bfe00a4-fcbf-4cd2-a1f2-44cbe3af7c13","Type":"ContainerStarted","Data":"9784e46aba4a7a66df14ecda73f93e4305278b9b8736496e7e100c4a34d727bf"}
Sep 18 17:51:11.241059 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:11.241025    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-qwhfg"
Sep 18 17:51:11.245799 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:11.245772    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-qwhfg"
Sep 18 17:51:11.273530 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:11.273489    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-qwhfg" podStartSLOduration=2.663922466 podStartE2EDuration="4.273474199s" totalImagesPullingTime="1.609551733s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:51:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:51:11.270933906 +0000 UTC m=+75.898065525" watchObservedRunningTime="2026-09-18 17:51:11.273474199 +0000 UTC m=+75.900605856"
Sep 18 17:51:11.625476 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:11.625403    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:11.866068 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:11.866042    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:12.053623 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:12.053591    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:12.961003 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:12.960970    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:14.093611 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:14.093570    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-65c58fbbcc-l699c"]
Sep 18 17:51:14.101572 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:14.101541    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:14.127541 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:14.127511    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65c58fbbcc-l699c"]
Sep 18 17:51:14.127690 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:14.127617    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65c58fbbcc-l699c"
Sep 18 17:51:14.130172 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:14.130144    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 18 17:51:14.130336 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:14.130315    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 18 17:51:14.130624 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:14.130603    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 18 17:51:14.131199 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:14.131182    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-5j4br\""
Sep 18 17:51:14.131564 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:14.131543    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 18 17:51:14.131681 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:14.131590    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 18 17:51:14.131681 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:14.131593    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 18 17:51:14.131681 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:14.131596    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 18 17:51:14.135739 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:14.135708    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 18 17:51:14.286008 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:14.285975    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/24d70c78-d64e-4aa0-9d34-e1a23d5996af-service-ca\") pod \"console-65c58fbbcc-l699c\" (UID: \"24d70c78-d64e-4aa0-9d34-e1a23d5996af\") " pod="openshift-console/console-65c58fbbcc-l699c"
Sep 18 17:51:14.286181 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:14.286018    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrwqx\" (UniqueName: \"kubernetes.io/projected/24d70c78-d64e-4aa0-9d34-e1a23d5996af-kube-api-access-rrwqx\") pod \"console-65c58fbbcc-l699c\" (UID: \"24d70c78-d64e-4aa0-9d34-e1a23d5996af\") " pod="openshift-console/console-65c58fbbcc-l699c"
Sep 18 17:51:14.286181 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:14.286049    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/24d70c78-d64e-4aa0-9d34-e1a23d5996af-oauth-serving-cert\") pod \"console-65c58fbbcc-l699c\" (UID: \"24d70c78-d64e-4aa0-9d34-e1a23d5996af\") " pod="openshift-console/console-65c58fbbcc-l699c"
Sep 18 17:51:14.286181 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:14.286089    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/24d70c78-d64e-4aa0-9d34-e1a23d5996af-console-oauth-config\") pod \"console-65c58fbbcc-l699c\" (UID: \"24d70c78-d64e-4aa0-9d34-e1a23d5996af\") " pod="openshift-console/console-65c58fbbcc-l699c"
Sep 18 17:51:14.286181 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:14.286110    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/24d70c78-d64e-4aa0-9d34-e1a23d5996af-console-serving-cert\") pod \"console-65c58fbbcc-l699c\" (UID: \"24d70c78-d64e-4aa0-9d34-e1a23d5996af\") " pod="openshift-console/console-65c58fbbcc-l699c"
Sep 18 17:51:14.286181 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:14.286145    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24d70c78-d64e-4aa0-9d34-e1a23d5996af-trusted-ca-bundle\") pod \"console-65c58fbbcc-l699c\" (UID: \"24d70c78-d64e-4aa0-9d34-e1a23d5996af\") " pod="openshift-console/console-65c58fbbcc-l699c"
Sep 18 17:51:14.286181 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:14.286174    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/24d70c78-d64e-4aa0-9d34-e1a23d5996af-console-config\") pod \"console-65c58fbbcc-l699c\" (UID: \"24d70c78-d64e-4aa0-9d34-e1a23d5996af\") " pod="openshift-console/console-65c58fbbcc-l699c"
Sep 18 17:51:14.387325 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:14.387259    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/24d70c78-d64e-4aa0-9d34-e1a23d5996af-oauth-serving-cert\") pod \"console-65c58fbbcc-l699c\" (UID: \"24d70c78-d64e-4aa0-9d34-e1a23d5996af\") " pod="openshift-console/console-65c58fbbcc-l699c"
Sep 18 17:51:14.387325 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:14.387314    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/24d70c78-d64e-4aa0-9d34-e1a23d5996af-console-oauth-config\") pod \"console-65c58fbbcc-l699c\" (UID: \"24d70c78-d64e-4aa0-9d34-e1a23d5996af\") " pod="openshift-console/console-65c58fbbcc-l699c"
Sep 18 17:51:14.387493 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:14.387342    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/24d70c78-d64e-4aa0-9d34-e1a23d5996af-console-serving-cert\") pod \"console-65c58fbbcc-l699c\" (UID: \"24d70c78-d64e-4aa0-9d34-e1a23d5996af\") " pod="openshift-console/console-65c58fbbcc-l699c"
Sep 18 17:51:14.387493 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:14.387367    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24d70c78-d64e-4aa0-9d34-e1a23d5996af-trusted-ca-bundle\") pod \"console-65c58fbbcc-l699c\" (UID: \"24d70c78-d64e-4aa0-9d34-e1a23d5996af\") " pod="openshift-console/console-65c58fbbcc-l699c"
Sep 18 17:51:14.387493 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:14.387395    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/24d70c78-d64e-4aa0-9d34-e1a23d5996af-console-config\") pod \"console-65c58fbbcc-l699c\" (UID: \"24d70c78-d64e-4aa0-9d34-e1a23d5996af\") " pod="openshift-console/console-65c58fbbcc-l699c"
Sep 18 17:51:14.387493 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:14.387439    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/24d70c78-d64e-4aa0-9d34-e1a23d5996af-service-ca\") pod \"console-65c58fbbcc-l699c\" (UID: \"24d70c78-d64e-4aa0-9d34-e1a23d5996af\") " pod="openshift-console/console-65c58fbbcc-l699c"
Sep 18 17:51:14.387493 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:14.387482    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrwqx\" (UniqueName: \"kubernetes.io/projected/24d70c78-d64e-4aa0-9d34-e1a23d5996af-kube-api-access-rrwqx\") pod \"console-65c58fbbcc-l699c\" (UID: \"24d70c78-d64e-4aa0-9d34-e1a23d5996af\") " pod="openshift-console/console-65c58fbbcc-l699c"
Sep 18 17:51:14.388097 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:14.388076    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/24d70c78-d64e-4aa0-9d34-e1a23d5996af-oauth-serving-cert\") pod \"console-65c58fbbcc-l699c\" (UID: \"24d70c78-d64e-4aa0-9d34-e1a23d5996af\") " pod="openshift-console/console-65c58fbbcc-l699c"
Sep 18 17:51:14.388210 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:14.388076    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/24d70c78-d64e-4aa0-9d34-e1a23d5996af-console-config\") pod \"console-65c58fbbcc-l699c\" (UID: \"24d70c78-d64e-4aa0-9d34-e1a23d5996af\") " pod="openshift-console/console-65c58fbbcc-l699c"
Sep 18 17:51:14.388252 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:14.388207    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/24d70c78-d64e-4aa0-9d34-e1a23d5996af-service-ca\") pod \"console-65c58fbbcc-l699c\" (UID: \"24d70c78-d64e-4aa0-9d34-e1a23d5996af\") " pod="openshift-console/console-65c58fbbcc-l699c"
Sep 18 17:51:14.388308 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:14.388282    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24d70c78-d64e-4aa0-9d34-e1a23d5996af-trusted-ca-bundle\") pod \"console-65c58fbbcc-l699c\" (UID: \"24d70c78-d64e-4aa0-9d34-e1a23d5996af\") " pod="openshift-console/console-65c58fbbcc-l699c"
Sep 18 17:51:14.389778 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:14.389743    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/24d70c78-d64e-4aa0-9d34-e1a23d5996af-console-oauth-config\") pod \"console-65c58fbbcc-l699c\" (UID: \"24d70c78-d64e-4aa0-9d34-e1a23d5996af\") " pod="openshift-console/console-65c58fbbcc-l699c"
Sep 18 17:51:14.389884 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:14.389868    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/24d70c78-d64e-4aa0-9d34-e1a23d5996af-console-serving-cert\") pod \"console-65c58fbbcc-l699c\" (UID: \"24d70c78-d64e-4aa0-9d34-e1a23d5996af\") " pod="openshift-console/console-65c58fbbcc-l699c"
Sep 18 17:51:14.397593 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:14.397570    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrwqx\" (UniqueName: \"kubernetes.io/projected/24d70c78-d64e-4aa0-9d34-e1a23d5996af-kube-api-access-rrwqx\") pod \"console-65c58fbbcc-l699c\" (UID: \"24d70c78-d64e-4aa0-9d34-e1a23d5996af\") " pod="openshift-console/console-65c58fbbcc-l699c"
Sep 18 17:51:14.437400 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:14.437372    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65c58fbbcc-l699c"
Sep 18 17:51:14.737782 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:14.737742    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65c58fbbcc-l699c"]
Sep 18 17:51:14.741199 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:51:14.741169    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24d70c78_d64e_4aa0_9d34_e1a23d5996af.slice/crio-312eb86d42942fa7a123665b23314a21f6c6a0a0a85130f908b79def37591e41 WatchSource:0}: Error finding container 312eb86d42942fa7a123665b23314a21f6c6a0a0a85130f908b79def37591e41: Status 404 returned error can't find the container with id 312eb86d42942fa7a123665b23314a21f6c6a0a0a85130f908b79def37591e41
Sep 18 17:51:15.261860 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:15.261820    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65c58fbbcc-l699c" event={"ID":"24d70c78-d64e-4aa0-9d34-e1a23d5996af","Type":"ContainerStarted","Data":"312eb86d42942fa7a123665b23314a21f6c6a0a0a85130f908b79def37591e41"}
Sep 18 17:51:17.398401 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:17.398368    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:17.621499 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:17.621466    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:17.753602 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:17.753525    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:18.236245 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:18.236212    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-9hjjd"
Sep 18 17:51:19.280489 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:19.280449    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65c58fbbcc-l699c" event={"ID":"24d70c78-d64e-4aa0-9d34-e1a23d5996af","Type":"ContainerStarted","Data":"2e987a96b2951bf3c2ad06288cfb9fe2d516802ed824dccc7f22db41dbdb334a"}
Sep 18 17:51:19.304282 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:19.304236    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-65c58fbbcc-l699c" podStartSLOduration=1.421436275 podStartE2EDuration="5.304220577s" totalImagesPullingTime="3.882784302s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:51:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:51:19.30353416 +0000 UTC m=+83.930665792" watchObservedRunningTime="2026-09-18 17:51:19.304220577 +0000 UTC m=+83.931352196"
Sep 18 17:51:21.112502 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:21.112466    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:21.183387 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:21.183351    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:21.828001 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:21.827962    2561 patch_prober.go:28] interesting pod/image-registry-78c879b97f-xtp8m container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 18 17:51:21.828179 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:21.828014    2561 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-78c879b97f-xtp8m" podUID="1096ace1-2554-43bc-bf2e-5e6d09e1c48f" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 17:51:24.200924 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:24.200900    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-78c879b97f-xtp8m"
Sep 18 17:51:24.386876 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:24.386850    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:24.437899 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:24.437868    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-65c58fbbcc-l699c"
Sep 18 17:51:24.438062 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:24.437920    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-65c58fbbcc-l699c"
Sep 18 17:51:24.443163 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:24.443140    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-65c58fbbcc-l699c"
Sep 18 17:51:24.466914 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:24.466825    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:25.306636 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:25.306608    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-65c58fbbcc-l699c"
Sep 18 17:51:25.573483 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:25.573402    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:27.162931 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.162904    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:27.291630 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.291598    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:27.454769 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.454696    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:27.575087 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.575045    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-kcr2n"]
Sep 18 17:51:27.577957 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.577938    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-kcr2n"
Sep 18 17:51:27.581374 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.581347    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-xzbds\""
Sep 18 17:51:27.581502 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.581355    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 18 17:51:27.581502 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.581470    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 18 17:51:27.581767 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.581753    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 18 17:51:27.596131 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.596095    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-kcr2n"]
Sep 18 17:51:27.618309 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.618285    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-knpq6"]
Sep 18 17:51:27.620508 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.620489    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-drc7s"]
Sep 18 17:51:27.620624 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.620608    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-knpq6"
Sep 18 17:51:27.622323 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.622304    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-drc7s"
Sep 18 17:51:27.625402 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.625383    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 18 17:51:27.625818 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.625788    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-blndk\""
Sep 18 17:51:27.625915 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.625897    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 18 17:51:27.626068 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.626049    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 18 17:51:27.626595 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.626555    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 18 17:51:27.627474 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.627455    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 18 17:51:27.627571 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.627471    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-gg4sf\""
Sep 18 17:51:27.627571 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.627518    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 18 17:51:27.635862 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.635833    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:27.642692 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.642675    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-knpq6"]
Sep 18 17:51:27.676527 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.676444    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-78c879b97f-xtp8m"]
Sep 18 17:51:27.687905 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.687880    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/36b0382b-a6dd-4a2b-b5d3-77c6219666ec-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-knpq6\" (UID: \"36b0382b-a6dd-4a2b-b5d3-77c6219666ec\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-knpq6"
Sep 18 17:51:27.688008 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.687921    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l2vt\" (UniqueName: \"kubernetes.io/projected/fbb000f1-48a6-44b2-929e-0efba2b6dada-kube-api-access-5l2vt\") pod \"openshift-state-metrics-86c5bc79d6-kcr2n\" (UID: \"fbb000f1-48a6-44b2-929e-0efba2b6dada\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-kcr2n"
Sep 18 17:51:27.688103 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.687998    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/36b0382b-a6dd-4a2b-b5d3-77c6219666ec-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-knpq6\" (UID: \"36b0382b-a6dd-4a2b-b5d3-77c6219666ec\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-knpq6"
Sep 18 17:51:27.688103 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.688045    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ae03e141-49ed-4b4b-9ec3-1fd32d2dacff-node-exporter-tls\") pod \"node-exporter-drc7s\" (UID: \"ae03e141-49ed-4b4b-9ec3-1fd32d2dacff\") " pod="openshift-monitoring/node-exporter-drc7s"
Sep 18 17:51:27.688103 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.688076    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/36b0382b-a6dd-4a2b-b5d3-77c6219666ec-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-knpq6\" (UID: \"36b0382b-a6dd-4a2b-b5d3-77c6219666ec\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-knpq6"
Sep 18 17:51:27.688253 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.688105    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fbb000f1-48a6-44b2-929e-0efba2b6dada-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-kcr2n\" (UID: \"fbb000f1-48a6-44b2-929e-0efba2b6dada\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-kcr2n"
Sep 18 17:51:27.688253 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.688185    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fbb000f1-48a6-44b2-929e-0efba2b6dada-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-kcr2n\" (UID: \"fbb000f1-48a6-44b2-929e-0efba2b6dada\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-kcr2n"
Sep 18 17:51:27.688253 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.688211    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ae03e141-49ed-4b4b-9ec3-1fd32d2dacff-sys\") pod \"node-exporter-drc7s\" (UID: \"ae03e141-49ed-4b4b-9ec3-1fd32d2dacff\") " pod="openshift-monitoring/node-exporter-drc7s"
Sep 18 17:51:27.688253 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.688233    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ae03e141-49ed-4b4b-9ec3-1fd32d2dacff-metrics-client-ca\") pod \"node-exporter-drc7s\" (UID: \"ae03e141-49ed-4b4b-9ec3-1fd32d2dacff\") " pod="openshift-monitoring/node-exporter-drc7s"
Sep 18 17:51:27.688446 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.688259    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ae03e141-49ed-4b4b-9ec3-1fd32d2dacff-root\") pod \"node-exporter-drc7s\" (UID: \"ae03e141-49ed-4b4b-9ec3-1fd32d2dacff\") " pod="openshift-monitoring/node-exporter-drc7s"
Sep 18 17:51:27.688446 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.688284    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ae03e141-49ed-4b4b-9ec3-1fd32d2dacff-node-exporter-wtmp\") pod \"node-exporter-drc7s\" (UID: \"ae03e141-49ed-4b4b-9ec3-1fd32d2dacff\") " pod="openshift-monitoring/node-exporter-drc7s"
Sep 18 17:51:27.688446 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.688313    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/36b0382b-a6dd-4a2b-b5d3-77c6219666ec-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-knpq6\" (UID: \"36b0382b-a6dd-4a2b-b5d3-77c6219666ec\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-knpq6"
Sep 18 17:51:27.688446 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.688373    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/36b0382b-a6dd-4a2b-b5d3-77c6219666ec-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-knpq6\" (UID: \"36b0382b-a6dd-4a2b-b5d3-77c6219666ec\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-knpq6"
Sep 18 17:51:27.688446 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.688419    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ae03e141-49ed-4b4b-9ec3-1fd32d2dacff-node-exporter-textfile\") pod \"node-exporter-drc7s\" (UID: \"ae03e141-49ed-4b4b-9ec3-1fd32d2dacff\") " pod="openshift-monitoring/node-exporter-drc7s"
Sep 18 17:51:27.688677 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.688445    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b54mc\" (UniqueName: \"kubernetes.io/projected/36b0382b-a6dd-4a2b-b5d3-77c6219666ec-kube-api-access-b54mc\") pod \"kube-state-metrics-84c885658c-knpq6\" (UID: \"36b0382b-a6dd-4a2b-b5d3-77c6219666ec\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-knpq6"
Sep 18 17:51:27.688677 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.688477    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ae03e141-49ed-4b4b-9ec3-1fd32d2dacff-node-exporter-accelerators-collector-config\") pod \"node-exporter-drc7s\" (UID: \"ae03e141-49ed-4b4b-9ec3-1fd32d2dacff\") " pod="openshift-monitoring/node-exporter-drc7s"
Sep 18 17:51:27.688677 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.688519    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fbb000f1-48a6-44b2-929e-0efba2b6dada-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-kcr2n\" (UID: \"fbb000f1-48a6-44b2-929e-0efba2b6dada\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-kcr2n"
Sep 18 17:51:27.688677 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.688543    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvbmv\" (UniqueName: \"kubernetes.io/projected/ae03e141-49ed-4b4b-9ec3-1fd32d2dacff-kube-api-access-xvbmv\") pod \"node-exporter-drc7s\" (UID: \"ae03e141-49ed-4b4b-9ec3-1fd32d2dacff\") " pod="openshift-monitoring/node-exporter-drc7s"
Sep 18 17:51:27.688677 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.688578    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ae03e141-49ed-4b4b-9ec3-1fd32d2dacff-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-drc7s\" (UID: \"ae03e141-49ed-4b4b-9ec3-1fd32d2dacff\") " pod="openshift-monitoring/node-exporter-drc7s"
Sep 18 17:51:27.751542 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.751461    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:27.789762 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.789726    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvbmv\" (UniqueName: \"kubernetes.io/projected/ae03e141-49ed-4b4b-9ec3-1fd32d2dacff-kube-api-access-xvbmv\") pod \"node-exporter-drc7s\" (UID: \"ae03e141-49ed-4b4b-9ec3-1fd32d2dacff\") " pod="openshift-monitoring/node-exporter-drc7s"
Sep 18 17:51:27.789762 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.789767    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ae03e141-49ed-4b4b-9ec3-1fd32d2dacff-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-drc7s\" (UID: \"ae03e141-49ed-4b4b-9ec3-1fd32d2dacff\") " pod="openshift-monitoring/node-exporter-drc7s"
Sep 18 17:51:27.789962 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.789794    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/36b0382b-a6dd-4a2b-b5d3-77c6219666ec-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-knpq6\" (UID: \"36b0382b-a6dd-4a2b-b5d3-77c6219666ec\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-knpq6"
Sep 18 17:51:27.789962 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.789815    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l2vt\" (UniqueName: \"kubernetes.io/projected/fbb000f1-48a6-44b2-929e-0efba2b6dada-kube-api-access-5l2vt\") pod \"openshift-state-metrics-86c5bc79d6-kcr2n\" (UID: \"fbb000f1-48a6-44b2-929e-0efba2b6dada\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-kcr2n"
Sep 18 17:51:27.789962 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.789842    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/36b0382b-a6dd-4a2b-b5d3-77c6219666ec-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-knpq6\" (UID: \"36b0382b-a6dd-4a2b-b5d3-77c6219666ec\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-knpq6"
Sep 18 17:51:27.790100 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.789969    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ae03e141-49ed-4b4b-9ec3-1fd32d2dacff-node-exporter-tls\") pod \"node-exporter-drc7s\" (UID: \"ae03e141-49ed-4b4b-9ec3-1fd32d2dacff\") " pod="openshift-monitoring/node-exporter-drc7s"
Sep 18 17:51:27.790100 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.790010    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/36b0382b-a6dd-4a2b-b5d3-77c6219666ec-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-knpq6\" (UID: \"36b0382b-a6dd-4a2b-b5d3-77c6219666ec\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-knpq6"
Sep 18 17:51:27.790100 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.790038    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fbb000f1-48a6-44b2-929e-0efba2b6dada-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-kcr2n\" (UID: \"fbb000f1-48a6-44b2-929e-0efba2b6dada\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-kcr2n"
Sep 18 17:51:27.790286 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.790140    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fbb000f1-48a6-44b2-929e-0efba2b6dada-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-kcr2n\" (UID: \"fbb000f1-48a6-44b2-929e-0efba2b6dada\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-kcr2n"
Sep 18 17:51:27.790286 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:51:27.790167    2561 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 18 17:51:27.790286 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:51:27.790255    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae03e141-49ed-4b4b-9ec3-1fd32d2dacff-node-exporter-tls podName:ae03e141-49ed-4b4b-9ec3-1fd32d2dacff nodeName:}" failed. No retries permitted until 2026-09-18 17:51:28.290233769 +0000 UTC m=+92.917365379 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/ae03e141-49ed-4b4b-9ec3-1fd32d2dacff-node-exporter-tls") pod "node-exporter-drc7s" (UID: "ae03e141-49ed-4b4b-9ec3-1fd32d2dacff") : secret "node-exporter-tls" not found
Sep 18 17:51:27.790286 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.790251    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/36b0382b-a6dd-4a2b-b5d3-77c6219666ec-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-knpq6\" (UID: \"36b0382b-a6dd-4a2b-b5d3-77c6219666ec\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-knpq6"
Sep 18 17:51:27.790286 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.790172    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ae03e141-49ed-4b4b-9ec3-1fd32d2dacff-sys\") pod \"node-exporter-drc7s\" (UID: \"ae03e141-49ed-4b4b-9ec3-1fd32d2dacff\") " pod="openshift-monitoring/node-exporter-drc7s"
Sep 18 17:51:27.790601 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.790320    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ae03e141-49ed-4b4b-9ec3-1fd32d2dacff-metrics-client-ca\") pod \"node-exporter-drc7s\" (UID: \"ae03e141-49ed-4b4b-9ec3-1fd32d2dacff\") " pod="openshift-monitoring/node-exporter-drc7s"
Sep 18 17:51:27.790601 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.790351    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ae03e141-49ed-4b4b-9ec3-1fd32d2dacff-root\") pod \"node-exporter-drc7s\" (UID: \"ae03e141-49ed-4b4b-9ec3-1fd32d2dacff\") " pod="openshift-monitoring/node-exporter-drc7s"
Sep 18 17:51:27.790601 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.790380    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ae03e141-49ed-4b4b-9ec3-1fd32d2dacff-node-exporter-wtmp\") pod \"node-exporter-drc7s\" (UID: \"ae03e141-49ed-4b4b-9ec3-1fd32d2dacff\") " pod="openshift-monitoring/node-exporter-drc7s"
Sep 18 17:51:27.790601 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.790416    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/36b0382b-a6dd-4a2b-b5d3-77c6219666ec-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-knpq6\" (UID: \"36b0382b-a6dd-4a2b-b5d3-77c6219666ec\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-knpq6"
Sep 18 17:51:27.790601 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.790457    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/36b0382b-a6dd-4a2b-b5d3-77c6219666ec-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-knpq6\" (UID: \"36b0382b-a6dd-4a2b-b5d3-77c6219666ec\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-knpq6"
Sep 18 17:51:27.790601 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.790487    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ae03e141-49ed-4b4b-9ec3-1fd32d2dacff-node-exporter-textfile\") pod \"node-exporter-drc7s\" (UID: \"ae03e141-49ed-4b4b-9ec3-1fd32d2dacff\") " pod="openshift-monitoring/node-exporter-drc7s"
Sep 18 17:51:27.790601 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.790500    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ae03e141-49ed-4b4b-9ec3-1fd32d2dacff-root\") pod \"node-exporter-drc7s\" (UID: \"ae03e141-49ed-4b4b-9ec3-1fd32d2dacff\") " pod="openshift-monitoring/node-exporter-drc7s"
Sep 18 17:51:27.790601 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.790511    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b54mc\" (UniqueName: \"kubernetes.io/projected/36b0382b-a6dd-4a2b-b5d3-77c6219666ec-kube-api-access-b54mc\") pod \"kube-state-metrics-84c885658c-knpq6\" (UID: \"36b0382b-a6dd-4a2b-b5d3-77c6219666ec\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-knpq6"
Sep 18 17:51:27.790601 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.790602    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ae03e141-49ed-4b4b-9ec3-1fd32d2dacff-node-exporter-accelerators-collector-config\") pod \"node-exporter-drc7s\" (UID: \"ae03e141-49ed-4b4b-9ec3-1fd32d2dacff\") " pod="openshift-monitoring/node-exporter-drc7s"
Sep 18 17:51:27.791047 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.790637    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fbb000f1-48a6-44b2-929e-0efba2b6dada-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-kcr2n\" (UID: \"fbb000f1-48a6-44b2-929e-0efba2b6dada\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-kcr2n"
Sep 18 17:51:27.791047 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.790902    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/36b0382b-a6dd-4a2b-b5d3-77c6219666ec-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-knpq6\" (UID: \"36b0382b-a6dd-4a2b-b5d3-77c6219666ec\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-knpq6"
Sep 18 17:51:27.791047 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.790966    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ae03e141-49ed-4b4b-9ec3-1fd32d2dacff-sys\") pod \"node-exporter-drc7s\" (UID: \"ae03e141-49ed-4b4b-9ec3-1fd32d2dacff\") " pod="openshift-monitoring/node-exporter-drc7s"
Sep 18 17:51:27.791047 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.791037    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ae03e141-49ed-4b4b-9ec3-1fd32d2dacff-metrics-client-ca\") pod \"node-exporter-drc7s\" (UID: \"ae03e141-49ed-4b4b-9ec3-1fd32d2dacff\") " pod="openshift-monitoring/node-exporter-drc7s"
Sep 18 17:51:27.792230 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.791422    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fbb000f1-48a6-44b2-929e-0efba2b6dada-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-kcr2n\" (UID: \"fbb000f1-48a6-44b2-929e-0efba2b6dada\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-kcr2n"
Sep 18 17:51:27.792230 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.791530    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ae03e141-49ed-4b4b-9ec3-1fd32d2dacff-node-exporter-textfile\") pod \"node-exporter-drc7s\" (UID: \"ae03e141-49ed-4b4b-9ec3-1fd32d2dacff\") " pod="openshift-monitoring/node-exporter-drc7s"
Sep 18 17:51:27.792230 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.791570    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ae03e141-49ed-4b4b-9ec3-1fd32d2dacff-node-exporter-wtmp\") pod \"node-exporter-drc7s\" (UID: \"ae03e141-49ed-4b4b-9ec3-1fd32d2dacff\") " pod="openshift-monitoring/node-exporter-drc7s"
Sep 18 17:51:27.792230 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.792181    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ae03e141-49ed-4b4b-9ec3-1fd32d2dacff-node-exporter-accelerators-collector-config\") pod \"node-exporter-drc7s\" (UID: \"ae03e141-49ed-4b4b-9ec3-1fd32d2dacff\") " pod="openshift-monitoring/node-exporter-drc7s"
Sep 18 17:51:27.792538 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.792510    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/36b0382b-a6dd-4a2b-b5d3-77c6219666ec-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-knpq6\" (UID: \"36b0382b-a6dd-4a2b-b5d3-77c6219666ec\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-knpq6"
Sep 18 17:51:27.792836 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.792813    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ae03e141-49ed-4b4b-9ec3-1fd32d2dacff-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-drc7s\" (UID: \"ae03e141-49ed-4b4b-9ec3-1fd32d2dacff\") " pod="openshift-monitoring/node-exporter-drc7s"
Sep 18 17:51:27.792917 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.792880    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/36b0382b-a6dd-4a2b-b5d3-77c6219666ec-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-knpq6\" (UID: \"36b0382b-a6dd-4a2b-b5d3-77c6219666ec\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-knpq6"
Sep 18 17:51:27.793051 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.793028    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fbb000f1-48a6-44b2-929e-0efba2b6dada-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-kcr2n\" (UID: \"fbb000f1-48a6-44b2-929e-0efba2b6dada\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-kcr2n"
Sep 18 17:51:27.793107 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.793084    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fbb000f1-48a6-44b2-929e-0efba2b6dada-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-kcr2n\" (UID: \"fbb000f1-48a6-44b2-929e-0efba2b6dada\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-kcr2n"
Sep 18 17:51:27.793544 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.793522    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/36b0382b-a6dd-4a2b-b5d3-77c6219666ec-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-knpq6\" (UID: \"36b0382b-a6dd-4a2b-b5d3-77c6219666ec\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-knpq6"
Sep 18 17:51:27.801867 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.801822    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvbmv\" (UniqueName: \"kubernetes.io/projected/ae03e141-49ed-4b4b-9ec3-1fd32d2dacff-kube-api-access-xvbmv\") pod \"node-exporter-drc7s\" (UID: \"ae03e141-49ed-4b4b-9ec3-1fd32d2dacff\") " pod="openshift-monitoring/node-exporter-drc7s"
Sep 18 17:51:27.801990 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.801969    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b54mc\" (UniqueName: \"kubernetes.io/projected/36b0382b-a6dd-4a2b-b5d3-77c6219666ec-kube-api-access-b54mc\") pod \"kube-state-metrics-84c885658c-knpq6\" (UID: \"36b0382b-a6dd-4a2b-b5d3-77c6219666ec\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-knpq6"
Sep 18 17:51:27.802180 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.802161    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l2vt\" (UniqueName: \"kubernetes.io/projected/fbb000f1-48a6-44b2-929e-0efba2b6dada-kube-api-access-5l2vt\") pod \"openshift-state-metrics-86c5bc79d6-kcr2n\" (UID: \"fbb000f1-48a6-44b2-929e-0efba2b6dada\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-kcr2n"
Sep 18 17:51:27.852730 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.852686    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:27.887026 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.887005    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-kcr2n"
Sep 18 17:51:27.930914 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.930875    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-knpq6"
Sep 18 17:51:27.961737 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:27.961703    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:28.199924 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:28.199888    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-kcr2n"]
Sep 18 17:51:28.204594 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:51:28.204571    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbb000f1_48a6_44b2_929e_0efba2b6dada.slice/crio-1dd64f717f7708c5b26807cbd84304962b239df86ffb58f70812b80d81f892cb WatchSource:0}: Error finding container 1dd64f717f7708c5b26807cbd84304962b239df86ffb58f70812b80d81f892cb: Status 404 returned error can't find the container with id 1dd64f717f7708c5b26807cbd84304962b239df86ffb58f70812b80d81f892cb
Sep 18 17:51:28.239664 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:28.239638    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-knpq6"]
Sep 18 17:51:28.244486 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:51:28.244461    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36b0382b_a6dd_4a2b_b5d3_77c6219666ec.slice/crio-484a93d58bbb787085a67cc74444f23efd8bf2d4282b188c09251d209216dd2f WatchSource:0}: Error finding container 484a93d58bbb787085a67cc74444f23efd8bf2d4282b188c09251d209216dd2f: Status 404 returned error can't find the container with id 484a93d58bbb787085a67cc74444f23efd8bf2d4282b188c09251d209216dd2f
Sep 18 17:51:28.257261 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:28.257242    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:28.293980 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:28.293948    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ae03e141-49ed-4b4b-9ec3-1fd32d2dacff-node-exporter-tls\") pod \"node-exporter-drc7s\" (UID: \"ae03e141-49ed-4b4b-9ec3-1fd32d2dacff\") " pod="openshift-monitoring/node-exporter-drc7s"
Sep 18 17:51:28.296434 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:28.296412    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ae03e141-49ed-4b4b-9ec3-1fd32d2dacff-node-exporter-tls\") pod \"node-exporter-drc7s\" (UID: \"ae03e141-49ed-4b4b-9ec3-1fd32d2dacff\") " pod="openshift-monitoring/node-exporter-drc7s"
Sep 18 17:51:28.315902 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:28.315873    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-knpq6" event={"ID":"36b0382b-a6dd-4a2b-b5d3-77c6219666ec","Type":"ContainerStarted","Data":"484a93d58bbb787085a67cc74444f23efd8bf2d4282b188c09251d209216dd2f"}
Sep 18 17:51:28.317563 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:28.317532    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-kcr2n" event={"ID":"fbb000f1-48a6-44b2-929e-0efba2b6dada","Type":"ContainerStarted","Data":"4342960b6ca97f5ea5cf86e6d1046fcf88363277c6bf1472ef6ad14a22abb9f7"}
Sep 18 17:51:28.317643 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:28.317593    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-kcr2n" event={"ID":"fbb000f1-48a6-44b2-929e-0efba2b6dada","Type":"ContainerStarted","Data":"1dd64f717f7708c5b26807cbd84304962b239df86ffb58f70812b80d81f892cb"}
Sep 18 17:51:28.350600 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:28.350577    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:28.506884 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:28.506804    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:28.535966 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:28.535924    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-drc7s"
Sep 18 17:51:28.545481 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:51:28.545457    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae03e141_49ed_4b4b_9ec3_1fd32d2dacff.slice/crio-cb602053e926cfec12c91efcb852f0f7dcf46851c93ac434844eabca6a237a77 WatchSource:0}: Error finding container cb602053e926cfec12c91efcb852f0f7dcf46851c93ac434844eabca6a237a77: Status 404 returned error can't find the container with id cb602053e926cfec12c91efcb852f0f7dcf46851c93ac434844eabca6a237a77
Sep 18 17:51:28.612220 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:28.612189    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:29.006934 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:29.006869    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:29.325363 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:29.325279    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-kcr2n" event={"ID":"fbb000f1-48a6-44b2-929e-0efba2b6dada","Type":"ContainerStarted","Data":"006a061664749d39ffdb2a45f993c8016e4b46c07dd0a73e0026eb7c05b19579"}
Sep 18 17:51:29.326867 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:29.326823    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-drc7s" event={"ID":"ae03e141-49ed-4b4b-9ec3-1fd32d2dacff","Type":"ContainerStarted","Data":"cb602053e926cfec12c91efcb852f0f7dcf46851c93ac434844eabca6a237a77"}
Sep 18 17:51:29.512805 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:29.512771    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:29.712451 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:29.712418    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:30.131689 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:30.130927    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:30.229796 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:30.229762    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:30.332510 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:30.332476    2561 generic.go:356] "Generic (PLEG): container finished" podID="ae03e141-49ed-4b4b-9ec3-1fd32d2dacff" containerID="7750cd175465f7ece4a3a77f6e38c27cb3dc2ff0507ce8e17a901b91fbd62689" exitCode=0
Sep 18 17:51:30.332939 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:30.332566    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-drc7s" event={"ID":"ae03e141-49ed-4b4b-9ec3-1fd32d2dacff","Type":"ContainerDied","Data":"7750cd175465f7ece4a3a77f6e38c27cb3dc2ff0507ce8e17a901b91fbd62689"}
Sep 18 17:51:30.335025 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:30.334994    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-knpq6" event={"ID":"36b0382b-a6dd-4a2b-b5d3-77c6219666ec","Type":"ContainerStarted","Data":"8ce13601870f9435176733841fbf02466d47772955e1b6d3500eb0a3ff6ed38c"}
Sep 18 17:51:30.335164 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:30.335056    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-knpq6" event={"ID":"36b0382b-a6dd-4a2b-b5d3-77c6219666ec","Type":"ContainerStarted","Data":"78b1864021cc505c07560af2141ca316a12b8f5adbf47652f71f3c241f32cad1"}
Sep 18 17:51:30.335164 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:30.335078    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-knpq6" event={"ID":"36b0382b-a6dd-4a2b-b5d3-77c6219666ec","Type":"ContainerStarted","Data":"9459862737b6bfe7f6e84a1c89210a612c33089c2f54a46bef1181d7b1edd522"}
Sep 18 17:51:30.337267 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:30.337238    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-kcr2n" event={"ID":"fbb000f1-48a6-44b2-929e-0efba2b6dada","Type":"ContainerStarted","Data":"d10529fe42cf31474924e0ce7f6e3dbb36006807f16e530ab2250430080a1340"}
Sep 18 17:51:30.390864 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:30.390813    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-kcr2n" podStartSLOduration=1.758054964 podStartE2EDuration="3.390792961s" totalImagesPullingTime="1.632737997s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:51:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:51:30.390265984 +0000 UTC m=+95.017397643" watchObservedRunningTime="2026-09-18 17:51:30.390792961 +0000 UTC m=+95.017924582"
Sep 18 17:51:30.425542 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:30.425500    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-84c885658c-knpq6" podStartSLOduration=1.710367105 podStartE2EDuration="3.425478305s" totalImagesPullingTime="1.7151112s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:51:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:51:30.424204266 +0000 UTC m=+95.051335887" watchObservedRunningTime="2026-09-18 17:51:30.425478305 +0000 UTC m=+95.052609929"
Sep 18 17:51:30.568178 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:30.568096    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:30.693486 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:30.693451    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:30.889481 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:30.889397    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:31.029875 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:31.029844    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:31.187670 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:31.187636    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:31.343625 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:31.343587    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-drc7s" event={"ID":"ae03e141-49ed-4b4b-9ec3-1fd32d2dacff","Type":"ContainerStarted","Data":"23905222e71830a9b7157d97dabff14b1f1faff1e4eae8f07b93a1799dff5ad0"}
Sep 18 17:51:31.344063 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:31.343642    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-drc7s" event={"ID":"ae03e141-49ed-4b4b-9ec3-1fd32d2dacff","Type":"ContainerStarted","Data":"efc09aa2a51a069df7ac7c6f69c70805a64d3c36c62e29dff4ed068e8fdb95bc"}
Sep 18 17:51:31.369012 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:31.368960    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-drc7s" podStartSLOduration=2.8597190980000002 podStartE2EDuration="4.368944796s" totalImagesPullingTime="1.410832881s" totalInitContainerRuntime="98.392817ms" isStatefulPod=true podCreationTimestamp="2026-09-18 17:51:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:51:31.367522872 +0000 UTC m=+95.994654493" watchObservedRunningTime="2026-09-18 17:51:31.368944796 +0000 UTC m=+95.996076416"
Sep 18 17:51:31.956914 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:31.956883    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:31.973056 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:31.973026    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-74bb5f98ff-qbjk5"]
Sep 18 17:51:31.976077 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:31.976060    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-74bb5f98ff-qbjk5"
Sep 18 17:51:31.978541 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:31.978512    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 18 17:51:31.978654 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:31.978533    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 18 17:51:31.978654 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:31.978624    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-rbm5v\""
Sep 18 17:51:31.978907 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:31.978887    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 18 17:51:31.979378 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:31.979363    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-66rp5ufiooovd\""
Sep 18 17:51:31.979436 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:31.979405    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 18 17:51:31.988497 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:31.988478    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-74bb5f98ff-qbjk5"]
Sep 18 17:51:32.026113 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:32.026090    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/4eee484c-2603-426b-9e46-936a81e86906-metrics-server-audit-profiles\") pod \"metrics-server-74bb5f98ff-qbjk5\" (UID: \"4eee484c-2603-426b-9e46-936a81e86906\") " pod="openshift-monitoring/metrics-server-74bb5f98ff-qbjk5"
Sep 18 17:51:32.026234 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:32.026169    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/4eee484c-2603-426b-9e46-936a81e86906-secret-metrics-server-tls\") pod \"metrics-server-74bb5f98ff-qbjk5\" (UID: \"4eee484c-2603-426b-9e46-936a81e86906\") " pod="openshift-monitoring/metrics-server-74bb5f98ff-qbjk5"
Sep 18 17:51:32.026234 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:32.026191    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4eee484c-2603-426b-9e46-936a81e86906-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-74bb5f98ff-qbjk5\" (UID: \"4eee484c-2603-426b-9e46-936a81e86906\") " pod="openshift-monitoring/metrics-server-74bb5f98ff-qbjk5"
Sep 18 17:51:32.026234 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:32.026212    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/4eee484c-2603-426b-9e46-936a81e86906-audit-log\") pod \"metrics-server-74bb5f98ff-qbjk5\" (UID: \"4eee484c-2603-426b-9e46-936a81e86906\") " pod="openshift-monitoring/metrics-server-74bb5f98ff-qbjk5"
Sep 18 17:51:32.026328 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:32.026234    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wfxr\" (UniqueName: \"kubernetes.io/projected/4eee484c-2603-426b-9e46-936a81e86906-kube-api-access-2wfxr\") pod \"metrics-server-74bb5f98ff-qbjk5\" (UID: \"4eee484c-2603-426b-9e46-936a81e86906\") " pod="openshift-monitoring/metrics-server-74bb5f98ff-qbjk5"
Sep 18 17:51:32.026328 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:32.026284    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/4eee484c-2603-426b-9e46-936a81e86906-secret-metrics-server-client-certs\") pod \"metrics-server-74bb5f98ff-qbjk5\" (UID: \"4eee484c-2603-426b-9e46-936a81e86906\") " pod="openshift-monitoring/metrics-server-74bb5f98ff-qbjk5"
Sep 18 17:51:32.026328 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:32.026315    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/4eee484c-2603-426b-9e46-936a81e86906-client-ca-bundle\") pod \"metrics-server-74bb5f98ff-qbjk5\" (UID: \"4eee484c-2603-426b-9e46-936a81e86906\") " pod="openshift-monitoring/metrics-server-74bb5f98ff-qbjk5"
Sep 18 17:51:32.129466 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:32.127550    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/4eee484c-2603-426b-9e46-936a81e86906-secret-metrics-server-tls\") pod \"metrics-server-74bb5f98ff-qbjk5\" (UID: \"4eee484c-2603-426b-9e46-936a81e86906\") " pod="openshift-monitoring/metrics-server-74bb5f98ff-qbjk5"
Sep 18 17:51:32.129466 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:32.127604    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4eee484c-2603-426b-9e46-936a81e86906-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-74bb5f98ff-qbjk5\" (UID: \"4eee484c-2603-426b-9e46-936a81e86906\") " pod="openshift-monitoring/metrics-server-74bb5f98ff-qbjk5"
Sep 18 17:51:32.129466 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:32.127635    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/4eee484c-2603-426b-9e46-936a81e86906-audit-log\") pod \"metrics-server-74bb5f98ff-qbjk5\" (UID: \"4eee484c-2603-426b-9e46-936a81e86906\") " pod="openshift-monitoring/metrics-server-74bb5f98ff-qbjk5"
Sep 18 17:51:32.129466 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:32.127663    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wfxr\" (UniqueName: \"kubernetes.io/projected/4eee484c-2603-426b-9e46-936a81e86906-kube-api-access-2wfxr\") pod \"metrics-server-74bb5f98ff-qbjk5\" (UID: \"4eee484c-2603-426b-9e46-936a81e86906\") " pod="openshift-monitoring/metrics-server-74bb5f98ff-qbjk5"
Sep 18 17:51:32.129466 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:32.127691    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/4eee484c-2603-426b-9e46-936a81e86906-secret-metrics-server-client-certs\") pod \"metrics-server-74bb5f98ff-qbjk5\" (UID: \"4eee484c-2603-426b-9e46-936a81e86906\") " pod="openshift-monitoring/metrics-server-74bb5f98ff-qbjk5"
Sep 18 17:51:32.129466 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:32.127714    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/4eee484c-2603-426b-9e46-936a81e86906-client-ca-bundle\") pod \"metrics-server-74bb5f98ff-qbjk5\" (UID: \"4eee484c-2603-426b-9e46-936a81e86906\") " pod="openshift-monitoring/metrics-server-74bb5f98ff-qbjk5"
Sep 18 17:51:32.129466 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:32.127755    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/4eee484c-2603-426b-9e46-936a81e86906-metrics-server-audit-profiles\") pod \"metrics-server-74bb5f98ff-qbjk5\" (UID: \"4eee484c-2603-426b-9e46-936a81e86906\") " pod="openshift-monitoring/metrics-server-74bb5f98ff-qbjk5"
Sep 18 17:51:32.129466 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:32.128994    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/4eee484c-2603-426b-9e46-936a81e86906-metrics-server-audit-profiles\") pod \"metrics-server-74bb5f98ff-qbjk5\" (UID: \"4eee484c-2603-426b-9e46-936a81e86906\") " pod="openshift-monitoring/metrics-server-74bb5f98ff-qbjk5"
Sep 18 17:51:32.129466 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:32.129053    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/4eee484c-2603-426b-9e46-936a81e86906-audit-log\") pod \"metrics-server-74bb5f98ff-qbjk5\" (UID: \"4eee484c-2603-426b-9e46-936a81e86906\") " pod="openshift-monitoring/metrics-server-74bb5f98ff-qbjk5"
Sep 18 17:51:32.129466 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:32.129366    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4eee484c-2603-426b-9e46-936a81e86906-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-74bb5f98ff-qbjk5\" (UID: \"4eee484c-2603-426b-9e46-936a81e86906\") " pod="openshift-monitoring/metrics-server-74bb5f98ff-qbjk5"
Sep 18 17:51:32.131632 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:32.131607    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/4eee484c-2603-426b-9e46-936a81e86906-secret-metrics-server-client-certs\") pod \"metrics-server-74bb5f98ff-qbjk5\" (UID: \"4eee484c-2603-426b-9e46-936a81e86906\") " pod="openshift-monitoring/metrics-server-74bb5f98ff-qbjk5"
Sep 18 17:51:32.132187 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:32.132163    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/4eee484c-2603-426b-9e46-936a81e86906-client-ca-bundle\") pod \"metrics-server-74bb5f98ff-qbjk5\" (UID: \"4eee484c-2603-426b-9e46-936a81e86906\") " pod="openshift-monitoring/metrics-server-74bb5f98ff-qbjk5"
Sep 18 17:51:32.132318 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:32.132297    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/4eee484c-2603-426b-9e46-936a81e86906-secret-metrics-server-tls\") pod \"metrics-server-74bb5f98ff-qbjk5\" (UID: \"4eee484c-2603-426b-9e46-936a81e86906\") " pod="openshift-monitoring/metrics-server-74bb5f98ff-qbjk5"
Sep 18 17:51:32.145055 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:32.145031    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wfxr\" (UniqueName: \"kubernetes.io/projected/4eee484c-2603-426b-9e46-936a81e86906-kube-api-access-2wfxr\") pod \"metrics-server-74bb5f98ff-qbjk5\" (UID: \"4eee484c-2603-426b-9e46-936a81e86906\") " pod="openshift-monitoring/metrics-server-74bb5f98ff-qbjk5"
Sep 18 17:51:32.200453 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:32.200429    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:32.206290 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:32.206268    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-5mwwt"]
Sep 18 17:51:32.208992 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:32.208952    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-5mwwt"
Sep 18 17:51:32.211756 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:32.211734    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-gm84x\""
Sep 18 17:51:32.211863 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:32.211734    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 18 17:51:32.220330 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:32.220310    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-5mwwt"]
Sep 18 17:51:32.271812 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:32.271785    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:32.284738 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:32.284715    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-74bb5f98ff-qbjk5"
Sep 18 17:51:32.328978 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:32.328941    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/025ce4f7-ca1b-44fc-924a-fb32d1afca45-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-5mwwt\" (UID: \"025ce4f7-ca1b-44fc-924a-fb32d1afca45\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-5mwwt"
Sep 18 17:51:32.329178 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:32.329042    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6t6m\" (UniqueName: \"kubernetes.io/projected/025ce4f7-ca1b-44fc-924a-fb32d1afca45-kube-api-access-l6t6m\") pod \"monitoring-plugin-5984459ff8-5mwwt\" (UID: \"025ce4f7-ca1b-44fc-924a-fb32d1afca45\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-5mwwt"
Sep 18 17:51:32.394210 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:32.394179    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:32.429471 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:32.429438    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/025ce4f7-ca1b-44fc-924a-fb32d1afca45-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-5mwwt\" (UID: \"025ce4f7-ca1b-44fc-924a-fb32d1afca45\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-5mwwt"
Sep 18 17:51:32.429598 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:32.429560    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6t6m\" (UniqueName: \"kubernetes.io/projected/025ce4f7-ca1b-44fc-924a-fb32d1afca45-kube-api-access-l6t6m\") pod \"monitoring-plugin-5984459ff8-5mwwt\" (UID: \"025ce4f7-ca1b-44fc-924a-fb32d1afca45\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-5mwwt"
Sep 18 17:51:32.431848 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:32.431823    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/025ce4f7-ca1b-44fc-924a-fb32d1afca45-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-5mwwt\" (UID: \"025ce4f7-ca1b-44fc-924a-fb32d1afca45\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-5mwwt"
Sep 18 17:51:32.443964 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:32.443940    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6t6m\" (UniqueName: \"kubernetes.io/projected/025ce4f7-ca1b-44fc-924a-fb32d1afca45-kube-api-access-l6t6m\") pod \"monitoring-plugin-5984459ff8-5mwwt\" (UID: \"025ce4f7-ca1b-44fc-924a-fb32d1afca45\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-5mwwt"
Sep 18 17:51:32.495696 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:32.495626    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:32.518698 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:32.518680    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-5mwwt"
Sep 18 17:51:32.561229 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:32.561194    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-74bb5f98ff-qbjk5"]
Sep 18 17:51:32.566258 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:51:32.565549    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eee484c_2603_426b_9e46_936a81e86906.slice/crio-4be699da06d2ef34a6bf086196e96d333ae0876460438b60e4eeca8f6971e5ab WatchSource:0}: Error finding container 4be699da06d2ef34a6bf086196e96d333ae0876460438b60e4eeca8f6971e5ab: Status 404 returned error can't find the container with id 4be699da06d2ef34a6bf086196e96d333ae0876460438b60e4eeca8f6971e5ab
Sep 18 17:51:32.569016 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:32.568965    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:32.796033 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:32.796009    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-5mwwt"]
Sep 18 17:51:32.798515 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:51:32.798486    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod025ce4f7_ca1b_44fc_924a_fb32d1afca45.slice/crio-e0d2b8049a3f213312b7c4d56a039fd96ab719c22c7aac3912c389416805d805 WatchSource:0}: Error finding container e0d2b8049a3f213312b7c4d56a039fd96ab719c22c7aac3912c389416805d805: Status 404 returned error can't find the container with id e0d2b8049a3f213312b7c4d56a039fd96ab719c22c7aac3912c389416805d805
Sep 18 17:51:33.355226 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:33.355188    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-74bb5f98ff-qbjk5" event={"ID":"4eee484c-2603-426b-9e46-936a81e86906","Type":"ContainerStarted","Data":"4be699da06d2ef34a6bf086196e96d333ae0876460438b60e4eeca8f6971e5ab"}
Sep 18 17:51:33.356601 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:33.356573    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-5mwwt" event={"ID":"025ce4f7-ca1b-44fc-924a-fb32d1afca45","Type":"ContainerStarted","Data":"e0d2b8049a3f213312b7c4d56a039fd96ab719c22c7aac3912c389416805d805"}
Sep 18 17:51:33.724808 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:33.724758    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:33.824662 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:33.824629    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:34.199620 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.199591    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-bgwtf"
Sep 18 17:51:34.288475 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.288385    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 17:51:34.289489 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.289448    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:34.292745 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.292722    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:51:34.298976 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.298956    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 18 17:51:34.299359 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.299054    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 18 17:51:34.299359 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.299111    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 18 17:51:34.299359 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.299187    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 18 17:51:34.299645 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.299421    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-7fsdhaco8c34s\""
Sep 18 17:51:34.299645 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.299482    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 18 17:51:34.299739 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.299697    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 18 17:51:34.300092 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.299883    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 18 17:51:34.300092 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.299977    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 18 17:51:34.300375 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.300342    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 18 17:51:34.300514 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.300497    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 18 17:51:34.300582 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.300558    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 18 17:51:34.305612 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.305250    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-v2f4q\""
Sep 18 17:51:34.305612 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.305414    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 18 17:51:34.305908 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.305887    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 18 17:51:34.308731 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.308595    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 18 17:51:34.339715 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.339686    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 17:51:34.345764 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.345716    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/02e201f9-5c67-4fc3-8822-8c39e516dddb-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:51:34.345904 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.345767    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/02e201f9-5c67-4fc3-8822-8c39e516dddb-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:51:34.345904 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.345799    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/02e201f9-5c67-4fc3-8822-8c39e516dddb-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:51:34.345904 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.345827    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/02e201f9-5c67-4fc3-8822-8c39e516dddb-web-config\") pod \"prometheus-k8s-0\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:51:34.345904 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.345867    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02e201f9-5c67-4fc3-8822-8c39e516dddb-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:51:34.346110 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.345925    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/02e201f9-5c67-4fc3-8822-8c39e516dddb-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:51:34.346110 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.345969    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/02e201f9-5c67-4fc3-8822-8c39e516dddb-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:51:34.346110 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.345994    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/02e201f9-5c67-4fc3-8822-8c39e516dddb-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:51:34.346110 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.346061    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/02e201f9-5c67-4fc3-8822-8c39e516dddb-config-out\") pod \"prometheus-k8s-0\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:51:34.346110 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.346093    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/02e201f9-5c67-4fc3-8822-8c39e516dddb-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:51:34.346437 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.346157    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/02e201f9-5c67-4fc3-8822-8c39e516dddb-config\") pod \"prometheus-k8s-0\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:51:34.346437 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.346186    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp2sp\" (UniqueName: \"kubernetes.io/projected/02e201f9-5c67-4fc3-8822-8c39e516dddb-kube-api-access-qp2sp\") pod \"prometheus-k8s-0\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:51:34.346437 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.346245    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/02e201f9-5c67-4fc3-8822-8c39e516dddb-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:51:34.346437 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.346265    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/02e201f9-5c67-4fc3-8822-8c39e516dddb-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:51:34.346437 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.346294    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/02e201f9-5c67-4fc3-8822-8c39e516dddb-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:51:34.346437 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.346321    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/02e201f9-5c67-4fc3-8822-8c39e516dddb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:51:34.346437 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.346346    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02e201f9-5c67-4fc3-8822-8c39e516dddb-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:51:34.346437 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.346382    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/02e201f9-5c67-4fc3-8822-8c39e516dddb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:51:34.346437 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.346401    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/02e201f9-5c67-4fc3-8822-8c39e516dddb-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:51:34.346874 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.346446    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02e201f9-5c67-4fc3-8822-8c39e516dddb-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:51:34.363383 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.363345    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-74bb5f98ff-qbjk5" event={"ID":"4eee484c-2603-426b-9e46-936a81e86906","Type":"ContainerStarted","Data":"35665010f5e6bb98d46007014a1555c2dcc5fee950215fe894a05e9a6bb4afe1"}
Sep 18 17:51:34.447030 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.446987    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02e201f9-5c67-4fc3-8822-8c39e516dddb-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:51:34.447211 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.447062    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/02e201f9-5c67-4fc3-8822-8c39e516dddb-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:51:34.447211 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.447131    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/02e201f9-5c67-4fc3-8822-8c39e516dddb-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:51:34.447211 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.447157    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/02e201f9-5c67-4fc3-8822-8c39e516dddb-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:51:34.447211 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.447183    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/02e201f9-5c67-4fc3-8822-8c39e516dddb-web-config\") pod \"prometheus-k8s-0\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:51:34.447482 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.447220    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02e201f9-5c67-4fc3-8822-8c39e516dddb-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:51:34.447482 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.447252    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/02e201f9-5c67-4fc3-8822-8c39e516dddb-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:51:34.447482 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.447293    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/02e201f9-5c67-4fc3-8822-8c39e516dddb-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:51:34.447482 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.447315    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/02e201f9-5c67-4fc3-8822-8c39e516dddb-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:51:34.447482 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.447372    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/02e201f9-5c67-4fc3-8822-8c39e516dddb-config-out\") pod \"prometheus-k8s-0\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:51:34.447482 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.447406    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/02e201f9-5c67-4fc3-8822-8c39e516dddb-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:51:34.447482 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.447440    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/02e201f9-5c67-4fc3-8822-8c39e516dddb-config\") pod \"prometheus-k8s-0\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:51:34.447482 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.447470    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp2sp\" (UniqueName: \"kubernetes.io/projected/02e201f9-5c67-4fc3-8822-8c39e516dddb-kube-api-access-qp2sp\") pod \"prometheus-k8s-0\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:51:34.447843 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.447761    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/02e201f9-5c67-4fc3-8822-8c39e516dddb-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:51:34.447843 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.447800    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/02e201f9-5c67-4fc3-8822-8c39e516dddb-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:51:34.447993 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.447867    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/02e201f9-5c67-4fc3-8822-8c39e516dddb-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:51:34.447993 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.447869    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/02e201f9-5c67-4fc3-8822-8c39e516dddb-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:51:34.447993 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.447900    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/02e201f9-5c67-4fc3-8822-8c39e516dddb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:51:34.447993 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.447908    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/02e201f9-5c67-4fc3-8822-8c39e516dddb-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:51:34.447993 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.447925    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02e201f9-5c67-4fc3-8822-8c39e516dddb-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:51:34.448269 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.447997    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/02e201f9-5c67-4fc3-8822-8c39e516dddb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:51:34.448269 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.448026    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/02e201f9-5c67-4fc3-8822-8c39e516dddb-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:51:34.448448 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.448405    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02e201f9-5c67-4fc3-8822-8c39e516dddb-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:51:34.448793 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.448566    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02e201f9-5c67-4fc3-8822-8c39e516dddb-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:51:34.453472 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.449066    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/02e201f9-5c67-4fc3-8822-8c39e516dddb-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:51:34.453472 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.451899    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/02e201f9-5c67-4fc3-8822-8c39e516dddb-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:51:34.453472 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.452606    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/02e201f9-5c67-4fc3-8822-8c39e516dddb-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:51:34.453472 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.452733    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/02e201f9-5c67-4fc3-8822-8c39e516dddb-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:51:34.453472 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.453023    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:34.457895 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.457425    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02e201f9-5c67-4fc3-8822-8c39e516dddb-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:51:34.465624 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.459379    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/02e201f9-5c67-4fc3-8822-8c39e516dddb-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:51:34.465624 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.459484    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/02e201f9-5c67-4fc3-8822-8c39e516dddb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:51:34.465624 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.459680    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/02e201f9-5c67-4fc3-8822-8c39e516dddb-config\") pod \"prometheus-k8s-0\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:51:34.465624 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.459776    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/02e201f9-5c67-4fc3-8822-8c39e516dddb-config-out\") pod \"prometheus-k8s-0\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:51:34.465624 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.460059    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/02e201f9-5c67-4fc3-8822-8c39e516dddb-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:51:34.465624 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.460266    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/02e201f9-5c67-4fc3-8822-8c39e516dddb-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:51:34.465624 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.461769    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/02e201f9-5c67-4fc3-8822-8c39e516dddb-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:51:34.465624 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.462041    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/02e201f9-5c67-4fc3-8822-8c39e516dddb-web-config\") pod \"prometheus-k8s-0\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:51:34.465624 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.462370    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/02e201f9-5c67-4fc3-8822-8c39e516dddb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:51:34.465624 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.462579    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/02e201f9-5c67-4fc3-8822-8c39e516dddb-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:51:34.474273 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.474251    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp2sp\" (UniqueName: \"kubernetes.io/projected/02e201f9-5c67-4fc3-8822-8c39e516dddb-kube-api-access-qp2sp\") pod \"prometheus-k8s-0\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:51:34.607404 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:34.607321    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:51:35.152145 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:35.152075    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-74bb5f98ff-qbjk5" podStartSLOduration=2.7907516230000002 podStartE2EDuration="4.152051365s" totalImagesPullingTime="1.361299742s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:51:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:51:34.434661121 +0000 UTC m=+99.061792752" watchObservedRunningTime="2026-09-18 17:51:35.152051365 +0000 UTC m=+99.779182987"
Sep 18 17:51:35.153029 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:35.153006    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 17:51:35.154595 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:51:35.154572    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02e201f9_5c67_4fc3_8822_8c39e516dddb.slice/crio-d1e1be8d9f7b9e6c99c96ce23552c191ff551ce8655c7db8fd77ed5aa6d04d8c WatchSource:0}: Error finding container d1e1be8d9f7b9e6c99c96ce23552c191ff551ce8655c7db8fd77ed5aa6d04d8c: Status 404 returned error can't find the container with id d1e1be8d9f7b9e6c99c96ce23552c191ff551ce8655c7db8fd77ed5aa6d04d8c
Sep 18 17:51:35.368967 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:35.368935    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"02e201f9-5c67-4fc3-8822-8c39e516dddb","Type":"ContainerStarted","Data":"d1e1be8d9f7b9e6c99c96ce23552c191ff551ce8655c7db8fd77ed5aa6d04d8c"}
Sep 18 17:51:35.370630 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:35.370598    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-5mwwt" event={"ID":"025ce4f7-ca1b-44fc-924a-fb32d1afca45","Type":"ContainerStarted","Data":"3d945aa40afac9ed91da060a071585bb7f9ab235a515212c1fa1dd8d171eeef5"}
Sep 18 17:51:35.402339 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:35.402252    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5984459ff8-5mwwt" podStartSLOduration=1.302786067 podStartE2EDuration="3.402230716s" totalImagesPullingTime="2.099444649s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:51:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:51:35.400639301 +0000 UTC m=+100.027770920" watchObservedRunningTime="2026-09-18 17:51:35.402230716 +0000 UTC m=+100.029362338"
Sep 18 17:51:36.376796 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:36.376765    2561 generic.go:356] "Generic (PLEG): container finished" podID="02e201f9-5c67-4fc3-8822-8c39e516dddb" containerID="6a70cb9a75cc5f915d386219b8c5ab68bd8ffd36a1a018530b26cc0a20c7ef62" exitCode=0
Sep 18 17:51:36.377222 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:36.376892    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"02e201f9-5c67-4fc3-8822-8c39e516dddb","Type":"ContainerDied","Data":"6a70cb9a75cc5f915d386219b8c5ab68bd8ffd36a1a018530b26cc0a20c7ef62"}
Sep 18 17:51:36.377222 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:36.377141    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-5mwwt"
Sep 18 17:51:36.382533 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:36.382515    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-5mwwt"
Sep 18 17:51:40.391971 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:40.391935    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:40.397413 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:40.397382    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"02e201f9-5c67-4fc3-8822-8c39e516dddb","Type":"ContainerStarted","Data":"cae77bd82a6938cf314fffa5dbeb4ed4f67024f1d5e28447c519697f39c9bb8c"}
Sep 18 17:51:40.397551 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:40.397452    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"02e201f9-5c67-4fc3-8822-8c39e516dddb","Type":"ContainerStarted","Data":"5d47cee9475e7b69a9c0e95a5d58e85210157a0a019820b878bb75a0f376afb7"}
Sep 18 17:51:40.527254 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:40.527218    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:41.007066 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:41.007016    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:41.108434 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:41.108386    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:41.184790 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:41.184760    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:42.408973 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:42.408930    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"02e201f9-5c67-4fc3-8822-8c39e516dddb","Type":"ContainerStarted","Data":"cac905f0d7e26eff0b121e5dac51d2dbbc047ff087d7c85991c1ad3270370cf6"}
Sep 18 17:51:42.409372 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:42.409049    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"02e201f9-5c67-4fc3-8822-8c39e516dddb","Type":"ContainerStarted","Data":"5ee666f651430a2827e3e851debe8643e8ac394b23660b97351850784d1698da"}
Sep 18 17:51:42.409372 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:42.409075    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"02e201f9-5c67-4fc3-8822-8c39e516dddb","Type":"ContainerStarted","Data":"11d68315b27586f2006e2848127343625491c24b02b71ca0dbe7049a5de8bc82"}
Sep 18 17:51:42.409372 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:42.409101    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"02e201f9-5c67-4fc3-8822-8c39e516dddb","Type":"ContainerStarted","Data":"4fa431a5325cc8da90d0a10f85db0a7e546046597b2bab50093fd69874855b04"}
Sep 18 17:51:42.442185 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:42.442112    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=1.981911501 podStartE2EDuration="8.442090298s" totalImagesPullingTime="6.441114587s" totalInitContainerRuntime="19.06421ms" isStatefulPod=false podCreationTimestamp="2026-09-18 17:51:34 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:51:42.440228303 +0000 UTC m=+107.067359924" watchObservedRunningTime="2026-09-18 17:51:42.442090298 +0000 UTC m=+107.069221920"
Sep 18 17:51:44.607722 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:44.607692    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:51:44.810467 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:44.810436    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65c58fbbcc-l699c"]
Sep 18 17:51:45.188307 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:45.188278    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:45.310970 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:45.310936    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:45.427058 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:45.427022    2561 generic.go:356] "Generic (PLEG): container finished" podID="60520815-25c1-4a4e-a036-c3fb968adbaf" containerID="abea28a7634ae57c57ea4fe664b9af206095882e81f8e799a54cdd61c064e74c" exitCode=0
Sep 18 17:51:45.427248 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:45.427086    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-qsqqj" event={"ID":"60520815-25c1-4a4e-a036-c3fb968adbaf","Type":"ContainerDied","Data":"abea28a7634ae57c57ea4fe664b9af206095882e81f8e799a54cdd61c064e74c"}
Sep 18 17:51:45.427458 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:45.427444    2561 scope.go:117] "RemoveContainer" containerID="abea28a7634ae57c57ea4fe664b9af206095882e81f8e799a54cdd61c064e74c"
Sep 18 17:51:46.433490 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:46.433449    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-qsqqj" event={"ID":"60520815-25c1-4a4e-a036-c3fb968adbaf","Type":"ContainerStarted","Data":"31f2681f30d3bab4ce1383d90cea1b7bc0c23abf65dba5b7909f16971c4b0cee"}
Sep 18 17:51:47.493105 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:47.493074    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:47.658550 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:47.658519    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:50.451037 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:50.451008    2561 generic.go:356] "Generic (PLEG): container finished" podID="2e2c3ab1-da7b-40e3-8d33-4d7ec5980634" containerID="f35e2d551ca510ea45480b149592bde2a0757d17ca069f7f2711053a527bcffc" exitCode=0
Sep 18 17:51:50.451445 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:50.451085    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-4hdg8" event={"ID":"2e2c3ab1-da7b-40e3-8d33-4d7ec5980634","Type":"ContainerDied","Data":"f35e2d551ca510ea45480b149592bde2a0757d17ca069f7f2711053a527bcffc"}
Sep 18 17:51:50.451523 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:50.451506    2561 scope.go:117] "RemoveContainer" containerID="f35e2d551ca510ea45480b149592bde2a0757d17ca069f7f2711053a527bcffc"
Sep 18 17:51:50.452760 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:50.452737    2561 generic.go:356] "Generic (PLEG): container finished" podID="26ece3e7-900c-4efe-b497-653cf58b5e60" containerID="75f337190abcf6d9045d733bab624019a8decc191b556dd202973d18bdb9eb36" exitCode=0
Sep 18 17:51:50.452847 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:50.452822    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-dx752" event={"ID":"26ece3e7-900c-4efe-b497-653cf58b5e60","Type":"ContainerDied","Data":"75f337190abcf6d9045d733bab624019a8decc191b556dd202973d18bdb9eb36"}
Sep 18 17:51:50.453109 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:50.453094    2561 scope.go:117] "RemoveContainer" containerID="75f337190abcf6d9045d733bab624019a8decc191b556dd202973d18bdb9eb36"
Sep 18 17:51:51.459290 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:51.459251    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-4hdg8" event={"ID":"2e2c3ab1-da7b-40e3-8d33-4d7ec5980634","Type":"ContainerStarted","Data":"c46e6456f3e1fd55ec8d723b2c3b791c0bd1ec6e2cf7e6ce33705de3960743b9"}
Sep 18 17:51:51.461054 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:51.461031    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-dx752" event={"ID":"26ece3e7-900c-4efe-b497-653cf58b5e60","Type":"ContainerStarted","Data":"2a07810f3cba46989185a10cec4ef668d31fab16899c421a41d365e79feffa2b"}
Sep 18 17:51:52.285633 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:52.285583    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-74bb5f98ff-qbjk5"
Sep 18 17:51:52.285798 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:52.285667    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-74bb5f98ff-qbjk5"
Sep 18 17:51:52.696519 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:52.696482    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-78c879b97f-xtp8m" podUID="1096ace1-2554-43bc-bf2e-5e6d09e1c48f" containerName="registry" containerID="cri-o://c82be5f9159a1a35c7b7f1681894fd2ec9465c9b5e6b4a127bbb43639fb27a79" gracePeriod=30
Sep 18 17:51:52.987424 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:52.987395    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78c879b97f-xtp8m"
Sep 18 17:51:53.128688 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:53.128650    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1096ace1-2554-43bc-bf2e-5e6d09e1c48f-bound-sa-token\") pod \"1096ace1-2554-43bc-bf2e-5e6d09e1c48f\" (UID: \"1096ace1-2554-43bc-bf2e-5e6d09e1c48f\") "
Sep 18 17:51:53.128855 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:53.128694    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1096ace1-2554-43bc-bf2e-5e6d09e1c48f-image-registry-private-configuration\") pod \"1096ace1-2554-43bc-bf2e-5e6d09e1c48f\" (UID: \"1096ace1-2554-43bc-bf2e-5e6d09e1c48f\") "
Sep 18 17:51:53.128855 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:53.128740    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1096ace1-2554-43bc-bf2e-5e6d09e1c48f-installation-pull-secrets\") pod \"1096ace1-2554-43bc-bf2e-5e6d09e1c48f\" (UID: \"1096ace1-2554-43bc-bf2e-5e6d09e1c48f\") "
Sep 18 17:51:53.128855 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:53.128767    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1096ace1-2554-43bc-bf2e-5e6d09e1c48f-registry-certificates\") pod \"1096ace1-2554-43bc-bf2e-5e6d09e1c48f\" (UID: \"1096ace1-2554-43bc-bf2e-5e6d09e1c48f\") "
Sep 18 17:51:53.128855 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:53.128823    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1096ace1-2554-43bc-bf2e-5e6d09e1c48f-ca-trust-extracted\") pod \"1096ace1-2554-43bc-bf2e-5e6d09e1c48f\" (UID: \"1096ace1-2554-43bc-bf2e-5e6d09e1c48f\") "
Sep 18 17:51:53.129164 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:53.128858    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1096ace1-2554-43bc-bf2e-5e6d09e1c48f-trusted-ca\") pod \"1096ace1-2554-43bc-bf2e-5e6d09e1c48f\" (UID: \"1096ace1-2554-43bc-bf2e-5e6d09e1c48f\") "
Sep 18 17:51:53.129164 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:53.128894    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1096ace1-2554-43bc-bf2e-5e6d09e1c48f-registry-tls\") pod \"1096ace1-2554-43bc-bf2e-5e6d09e1c48f\" (UID: \"1096ace1-2554-43bc-bf2e-5e6d09e1c48f\") "
Sep 18 17:51:53.129164 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:53.128947    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8ltk8\" (UniqueName: \"kubernetes.io/projected/1096ace1-2554-43bc-bf2e-5e6d09e1c48f-kube-api-access-8ltk8\") pod \"1096ace1-2554-43bc-bf2e-5e6d09e1c48f\" (UID: \"1096ace1-2554-43bc-bf2e-5e6d09e1c48f\") "
Sep 18 17:51:53.129363 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:53.129263    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1096ace1-2554-43bc-bf2e-5e6d09e1c48f-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "1096ace1-2554-43bc-bf2e-5e6d09e1c48f" (UID: "1096ace1-2554-43bc-bf2e-5e6d09e1c48f"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:51:53.129687 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:53.129659    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1096ace1-2554-43bc-bf2e-5e6d09e1c48f-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "1096ace1-2554-43bc-bf2e-5e6d09e1c48f" (UID: "1096ace1-2554-43bc-bf2e-5e6d09e1c48f"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:51:53.131579 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:53.131551    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1096ace1-2554-43bc-bf2e-5e6d09e1c48f-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "1096ace1-2554-43bc-bf2e-5e6d09e1c48f" (UID: "1096ace1-2554-43bc-bf2e-5e6d09e1c48f"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:51:53.131579 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:53.131564    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1096ace1-2554-43bc-bf2e-5e6d09e1c48f-kube-api-access-8ltk8" (OuterVolumeSpecName: "kube-api-access-8ltk8") pod "1096ace1-2554-43bc-bf2e-5e6d09e1c48f" (UID: "1096ace1-2554-43bc-bf2e-5e6d09e1c48f"). InnerVolumeSpecName "kube-api-access-8ltk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:51:53.131757 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:53.131645    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1096ace1-2554-43bc-bf2e-5e6d09e1c48f-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "1096ace1-2554-43bc-bf2e-5e6d09e1c48f" (UID: "1096ace1-2554-43bc-bf2e-5e6d09e1c48f"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:51:53.131757 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:53.131713    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1096ace1-2554-43bc-bf2e-5e6d09e1c48f-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "1096ace1-2554-43bc-bf2e-5e6d09e1c48f" (UID: "1096ace1-2554-43bc-bf2e-5e6d09e1c48f"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:51:53.132038 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:53.132010    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1096ace1-2554-43bc-bf2e-5e6d09e1c48f-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "1096ace1-2554-43bc-bf2e-5e6d09e1c48f" (UID: "1096ace1-2554-43bc-bf2e-5e6d09e1c48f"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:51:53.137537 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:53.137513    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1096ace1-2554-43bc-bf2e-5e6d09e1c48f-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "1096ace1-2554-43bc-bf2e-5e6d09e1c48f" (UID: "1096ace1-2554-43bc-bf2e-5e6d09e1c48f"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:51:53.229831 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:53.229747    2561 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1096ace1-2554-43bc-bf2e-5e6d09e1c48f-ca-trust-extracted\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 17:51:53.229831 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:53.229778    2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1096ace1-2554-43bc-bf2e-5e6d09e1c48f-trusted-ca\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 17:51:53.229831 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:53.229786    2561 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1096ace1-2554-43bc-bf2e-5e6d09e1c48f-registry-tls\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 17:51:53.229831 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:53.229794    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ltk8\" (UniqueName: \"kubernetes.io/projected/1096ace1-2554-43bc-bf2e-5e6d09e1c48f-kube-api-access-8ltk8\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 17:51:53.229831 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:53.229803    2561 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1096ace1-2554-43bc-bf2e-5e6d09e1c48f-bound-sa-token\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 17:51:53.229831 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:53.229812    2561 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1096ace1-2554-43bc-bf2e-5e6d09e1c48f-image-registry-private-configuration\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 17:51:53.229831 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:53.229820    2561 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1096ace1-2554-43bc-bf2e-5e6d09e1c48f-installation-pull-secrets\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 17:51:53.229831 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:53.229828    2561 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1096ace1-2554-43bc-bf2e-5e6d09e1c48f-registry-certificates\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 17:51:53.470913 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:53.470877    2561 generic.go:356] "Generic (PLEG): container finished" podID="1096ace1-2554-43bc-bf2e-5e6d09e1c48f" containerID="c82be5f9159a1a35c7b7f1681894fd2ec9465c9b5e6b4a127bbb43639fb27a79" exitCode=0
Sep 18 17:51:53.471099 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:53.470926    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78c879b97f-xtp8m"
Sep 18 17:51:53.471099 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:53.470968    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-78c879b97f-xtp8m" event={"ID":"1096ace1-2554-43bc-bf2e-5e6d09e1c48f","Type":"ContainerDied","Data":"c82be5f9159a1a35c7b7f1681894fd2ec9465c9b5e6b4a127bbb43639fb27a79"}
Sep 18 17:51:53.471099 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:53.471023    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-78c879b97f-xtp8m" event={"ID":"1096ace1-2554-43bc-bf2e-5e6d09e1c48f","Type":"ContainerDied","Data":"880cf5d1aa08d2b1319e109235fd3d92e860703cffb8e5c8449c063e66c459da"}
Sep 18 17:51:53.471099 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:53.471045    2561 scope.go:117] "RemoveContainer" containerID="c82be5f9159a1a35c7b7f1681894fd2ec9465c9b5e6b4a127bbb43639fb27a79"
Sep 18 17:51:53.479815 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:53.479797    2561 scope.go:117] "RemoveContainer" containerID="c82be5f9159a1a35c7b7f1681894fd2ec9465c9b5e6b4a127bbb43639fb27a79"
Sep 18 17:51:53.480099 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:51:53.480051    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c82be5f9159a1a35c7b7f1681894fd2ec9465c9b5e6b4a127bbb43639fb27a79\": container with ID starting with c82be5f9159a1a35c7b7f1681894fd2ec9465c9b5e6b4a127bbb43639fb27a79 not found: ID does not exist" containerID="c82be5f9159a1a35c7b7f1681894fd2ec9465c9b5e6b4a127bbb43639fb27a79"
Sep 18 17:51:53.480205 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:53.480084    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c82be5f9159a1a35c7b7f1681894fd2ec9465c9b5e6b4a127bbb43639fb27a79"} err="failed to get container status \"c82be5f9159a1a35c7b7f1681894fd2ec9465c9b5e6b4a127bbb43639fb27a79\": rpc error: code = NotFound desc = could not find container \"c82be5f9159a1a35c7b7f1681894fd2ec9465c9b5e6b4a127bbb43639fb27a79\": container with ID starting with c82be5f9159a1a35c7b7f1681894fd2ec9465c9b5e6b4a127bbb43639fb27a79 not found: ID does not exist"
Sep 18 17:51:53.506225 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:53.506201    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-78c879b97f-xtp8m"]
Sep 18 17:51:53.514298 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:53.514279    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-78c879b97f-xtp8m"]
Sep 18 17:51:53.860449 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:53.860376    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1096ace1-2554-43bc-bf2e-5e6d09e1c48f" path="/var/lib/kubelet/pods/1096ace1-2554-43bc-bf2e-5e6d09e1c48f/volumes"
Sep 18 17:51:55.208238 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:55.208207    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:51:55.308467 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:51:55.308441    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:52:09.829752 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:09.829693    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-65c58fbbcc-l699c" podUID="24d70c78-d64e-4aa0-9d34-e1a23d5996af" containerName="console" containerID="cri-o://2e987a96b2951bf3c2ad06288cfb9fe2d516802ed824dccc7f22db41dbdb334a" gracePeriod=15
Sep 18 17:52:10.072152 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:10.072125    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65c58fbbcc-l699c"
Sep 18 17:52:10.171715 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:10.171691    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/24d70c78-d64e-4aa0-9d34-e1a23d5996af-console-config\") pod \"24d70c78-d64e-4aa0-9d34-e1a23d5996af\" (UID: \"24d70c78-d64e-4aa0-9d34-e1a23d5996af\") "
Sep 18 17:52:10.171899 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:10.171730    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/24d70c78-d64e-4aa0-9d34-e1a23d5996af-service-ca\") pod \"24d70c78-d64e-4aa0-9d34-e1a23d5996af\" (UID: \"24d70c78-d64e-4aa0-9d34-e1a23d5996af\") "
Sep 18 17:52:10.171899 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:10.171758    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/24d70c78-d64e-4aa0-9d34-e1a23d5996af-console-serving-cert\") pod \"24d70c78-d64e-4aa0-9d34-e1a23d5996af\" (UID: \"24d70c78-d64e-4aa0-9d34-e1a23d5996af\") "
Sep 18 17:52:10.171899 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:10.171810    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/24d70c78-d64e-4aa0-9d34-e1a23d5996af-console-oauth-config\") pod \"24d70c78-d64e-4aa0-9d34-e1a23d5996af\" (UID: \"24d70c78-d64e-4aa0-9d34-e1a23d5996af\") "
Sep 18 17:52:10.171899 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:10.171861    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/24d70c78-d64e-4aa0-9d34-e1a23d5996af-oauth-serving-cert\") pod \"24d70c78-d64e-4aa0-9d34-e1a23d5996af\" (UID: \"24d70c78-d64e-4aa0-9d34-e1a23d5996af\") "
Sep 18 17:52:10.171899 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:10.171888    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24d70c78-d64e-4aa0-9d34-e1a23d5996af-trusted-ca-bundle\") pod \"24d70c78-d64e-4aa0-9d34-e1a23d5996af\" (UID: \"24d70c78-d64e-4aa0-9d34-e1a23d5996af\") "
Sep 18 17:52:10.172171 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:10.171921    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rrwqx\" (UniqueName: \"kubernetes.io/projected/24d70c78-d64e-4aa0-9d34-e1a23d5996af-kube-api-access-rrwqx\") pod \"24d70c78-d64e-4aa0-9d34-e1a23d5996af\" (UID: \"24d70c78-d64e-4aa0-9d34-e1a23d5996af\") "
Sep 18 17:52:10.172234 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:10.172089    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24d70c78-d64e-4aa0-9d34-e1a23d5996af-console-config" (OuterVolumeSpecName: "console-config") pod "24d70c78-d64e-4aa0-9d34-e1a23d5996af" (UID: "24d70c78-d64e-4aa0-9d34-e1a23d5996af"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:52:10.172234 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:10.172218    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24d70c78-d64e-4aa0-9d34-e1a23d5996af-service-ca" (OuterVolumeSpecName: "service-ca") pod "24d70c78-d64e-4aa0-9d34-e1a23d5996af" (UID: "24d70c78-d64e-4aa0-9d34-e1a23d5996af"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:52:10.172345 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:10.172269    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24d70c78-d64e-4aa0-9d34-e1a23d5996af-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "24d70c78-d64e-4aa0-9d34-e1a23d5996af" (UID: "24d70c78-d64e-4aa0-9d34-e1a23d5996af"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:52:10.172407 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:10.172362    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24d70c78-d64e-4aa0-9d34-e1a23d5996af-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "24d70c78-d64e-4aa0-9d34-e1a23d5996af" (UID: "24d70c78-d64e-4aa0-9d34-e1a23d5996af"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:52:10.174073 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:10.174051    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24d70c78-d64e-4aa0-9d34-e1a23d5996af-kube-api-access-rrwqx" (OuterVolumeSpecName: "kube-api-access-rrwqx") pod "24d70c78-d64e-4aa0-9d34-e1a23d5996af" (UID: "24d70c78-d64e-4aa0-9d34-e1a23d5996af"). InnerVolumeSpecName "kube-api-access-rrwqx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:52:10.174160 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:10.174074    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24d70c78-d64e-4aa0-9d34-e1a23d5996af-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "24d70c78-d64e-4aa0-9d34-e1a23d5996af" (UID: "24d70c78-d64e-4aa0-9d34-e1a23d5996af"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:52:10.174160 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:10.174139    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24d70c78-d64e-4aa0-9d34-e1a23d5996af-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "24d70c78-d64e-4aa0-9d34-e1a23d5996af" (UID: "24d70c78-d64e-4aa0-9d34-e1a23d5996af"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:52:10.273435 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:10.273390    2561 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/24d70c78-d64e-4aa0-9d34-e1a23d5996af-oauth-serving-cert\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 17:52:10.273435 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:10.273431    2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24d70c78-d64e-4aa0-9d34-e1a23d5996af-trusted-ca-bundle\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 17:52:10.273435 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:10.273441    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrwqx\" (UniqueName: \"kubernetes.io/projected/24d70c78-d64e-4aa0-9d34-e1a23d5996af-kube-api-access-rrwqx\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 17:52:10.273435 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:10.273450    2561 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/24d70c78-d64e-4aa0-9d34-e1a23d5996af-console-config\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 17:52:10.273685 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:10.273459    2561 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/24d70c78-d64e-4aa0-9d34-e1a23d5996af-service-ca\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 17:52:10.273685 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:10.273467    2561 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/24d70c78-d64e-4aa0-9d34-e1a23d5996af-console-serving-cert\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 17:52:10.273685 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:10.273475    2561 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/24d70c78-d64e-4aa0-9d34-e1a23d5996af-console-oauth-config\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 17:52:10.546773 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:10.546692    2561 generic.go:356] "Generic (PLEG): container finished" podID="24d70c78-d64e-4aa0-9d34-e1a23d5996af" containerID="2e987a96b2951bf3c2ad06288cfb9fe2d516802ed824dccc7f22db41dbdb334a" exitCode=0
Sep 18 17:52:10.546773 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:10.546744    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65c58fbbcc-l699c"
Sep 18 17:52:10.546957 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:10.546782    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65c58fbbcc-l699c" event={"ID":"24d70c78-d64e-4aa0-9d34-e1a23d5996af","Type":"ContainerDied","Data":"2e987a96b2951bf3c2ad06288cfb9fe2d516802ed824dccc7f22db41dbdb334a"}
Sep 18 17:52:10.546957 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:10.546843    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65c58fbbcc-l699c" event={"ID":"24d70c78-d64e-4aa0-9d34-e1a23d5996af","Type":"ContainerDied","Data":"312eb86d42942fa7a123665b23314a21f6c6a0a0a85130f908b79def37591e41"}
Sep 18 17:52:10.546957 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:10.546867    2561 scope.go:117] "RemoveContainer" containerID="2e987a96b2951bf3c2ad06288cfb9fe2d516802ed824dccc7f22db41dbdb334a"
Sep 18 17:52:10.555931 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:10.555909    2561 scope.go:117] "RemoveContainer" containerID="2e987a96b2951bf3c2ad06288cfb9fe2d516802ed824dccc7f22db41dbdb334a"
Sep 18 17:52:10.556255 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:52:10.556222    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e987a96b2951bf3c2ad06288cfb9fe2d516802ed824dccc7f22db41dbdb334a\": container with ID starting with 2e987a96b2951bf3c2ad06288cfb9fe2d516802ed824dccc7f22db41dbdb334a not found: ID does not exist" containerID="2e987a96b2951bf3c2ad06288cfb9fe2d516802ed824dccc7f22db41dbdb334a"
Sep 18 17:52:10.556340 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:10.556257    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e987a96b2951bf3c2ad06288cfb9fe2d516802ed824dccc7f22db41dbdb334a"} err="failed to get container status \"2e987a96b2951bf3c2ad06288cfb9fe2d516802ed824dccc7f22db41dbdb334a\": rpc error: code = NotFound desc = could not find container \"2e987a96b2951bf3c2ad06288cfb9fe2d516802ed824dccc7f22db41dbdb334a\": container with ID starting with 2e987a96b2951bf3c2ad06288cfb9fe2d516802ed824dccc7f22db41dbdb334a not found: ID does not exist"
Sep 18 17:52:10.573137 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:10.573101    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65c58fbbcc-l699c"]
Sep 18 17:52:10.582045 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:10.582023    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-65c58fbbcc-l699c"]
Sep 18 17:52:11.860036 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:11.860004    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24d70c78-d64e-4aa0-9d34-e1a23d5996af" path="/var/lib/kubelet/pods/24d70c78-d64e-4aa0-9d34-e1a23d5996af/volumes"
Sep 18 17:52:12.290214 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:12.290181    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-74bb5f98ff-qbjk5"
Sep 18 17:52:12.293905 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:12.293880    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-74bb5f98ff-qbjk5"
Sep 18 17:52:12.346402 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:12.346372    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:52:12.431402 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:12.431369    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:52:34.608072 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:34.608033    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:52:34.624378 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:34.624352    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:52:34.664324 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:34.664296    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:52:47.351335 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:47.351299    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:52:47.697699 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:47.697643    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:52:52.766130 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:52.766079    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 17:52:52.766643 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:52.766406    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="02e201f9-5c67-4fc3-8822-8c39e516dddb" containerName="prometheus" containerID="cri-o://5d47cee9475e7b69a9c0e95a5d58e85210157a0a019820b878bb75a0f376afb7" gracePeriod=600
Sep 18 17:52:52.766643 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:52.766422    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="02e201f9-5c67-4fc3-8822-8c39e516dddb" containerName="kube-rbac-proxy" containerID="cri-o://5ee666f651430a2827e3e851debe8643e8ac394b23660b97351850784d1698da" gracePeriod=600
Sep 18 17:52:52.766643 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:52.766451    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="02e201f9-5c67-4fc3-8822-8c39e516dddb" containerName="kube-rbac-proxy-thanos" containerID="cri-o://cac905f0d7e26eff0b121e5dac51d2dbbc047ff087d7c85991c1ad3270370cf6" gracePeriod=600
Sep 18 17:52:52.766643 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:52.766482    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="02e201f9-5c67-4fc3-8822-8c39e516dddb" containerName="kube-rbac-proxy-web" containerID="cri-o://11d68315b27586f2006e2848127343625491c24b02b71ca0dbe7049a5de8bc82" gracePeriod=600
Sep 18 17:52:52.766643 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:52.766489    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="02e201f9-5c67-4fc3-8822-8c39e516dddb" containerName="config-reloader" containerID="cri-o://cae77bd82a6938cf314fffa5dbeb4ed4f67024f1d5e28447c519697f39c9bb8c" gracePeriod=600
Sep 18 17:52:52.766928 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:52.766422    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="02e201f9-5c67-4fc3-8822-8c39e516dddb" containerName="thanos-sidecar" containerID="cri-o://4fa431a5325cc8da90d0a10f85db0a7e546046597b2bab50093fd69874855b04" gracePeriod=600
Sep 18 17:52:53.005219 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.005196    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:52:53.025629 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.025577    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/02e201f9-5c67-4fc3-8822-8c39e516dddb-configmap-metrics-client-ca\") pod \"02e201f9-5c67-4fc3-8822-8c39e516dddb\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") "
Sep 18 17:52:53.025629 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.025607    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/02e201f9-5c67-4fc3-8822-8c39e516dddb-prometheus-k8s-db\") pod \"02e201f9-5c67-4fc3-8822-8c39e516dddb\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") "
Sep 18 17:52:53.025629 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.025625    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02e201f9-5c67-4fc3-8822-8c39e516dddb-configmap-serving-certs-ca-bundle\") pod \"02e201f9-5c67-4fc3-8822-8c39e516dddb\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") "
Sep 18 17:52:53.025859 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.025665    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qp2sp\" (UniqueName: \"kubernetes.io/projected/02e201f9-5c67-4fc3-8822-8c39e516dddb-kube-api-access-qp2sp\") pod \"02e201f9-5c67-4fc3-8822-8c39e516dddb\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") "
Sep 18 17:52:53.025859 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.025686    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02e201f9-5c67-4fc3-8822-8c39e516dddb-configmap-kubelet-serving-ca-bundle\") pod \"02e201f9-5c67-4fc3-8822-8c39e516dddb\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") "
Sep 18 17:52:53.025859 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.025700    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/02e201f9-5c67-4fc3-8822-8c39e516dddb-prometheus-k8s-rulefiles-0\") pod \"02e201f9-5c67-4fc3-8822-8c39e516dddb\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") "
Sep 18 17:52:53.025859 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.025717    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/02e201f9-5c67-4fc3-8822-8c39e516dddb-secret-grpc-tls\") pod \"02e201f9-5c67-4fc3-8822-8c39e516dddb\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") "
Sep 18 17:52:53.025859 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.025733    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/02e201f9-5c67-4fc3-8822-8c39e516dddb-secret-prometheus-k8s-tls\") pod \"02e201f9-5c67-4fc3-8822-8c39e516dddb\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") "
Sep 18 17:52:53.025859 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.025750    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/02e201f9-5c67-4fc3-8822-8c39e516dddb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"02e201f9-5c67-4fc3-8822-8c39e516dddb\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") "
Sep 18 17:52:53.025859 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.025778    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/02e201f9-5c67-4fc3-8822-8c39e516dddb-secret-metrics-client-certs\") pod \"02e201f9-5c67-4fc3-8822-8c39e516dddb\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") "
Sep 18 17:52:53.025859 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.025815    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/02e201f9-5c67-4fc3-8822-8c39e516dddb-config-out\") pod \"02e201f9-5c67-4fc3-8822-8c39e516dddb\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") "
Sep 18 17:52:53.025859 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.025842    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/02e201f9-5c67-4fc3-8822-8c39e516dddb-secret-kube-rbac-proxy\") pod \"02e201f9-5c67-4fc3-8822-8c39e516dddb\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") "
Sep 18 17:52:53.026327 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.025891    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/02e201f9-5c67-4fc3-8822-8c39e516dddb-config\") pod \"02e201f9-5c67-4fc3-8822-8c39e516dddb\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") "
Sep 18 17:52:53.026327 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.025935    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02e201f9-5c67-4fc3-8822-8c39e516dddb-prometheus-trusted-ca-bundle\") pod \"02e201f9-5c67-4fc3-8822-8c39e516dddb\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") "
Sep 18 17:52:53.026327 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.025964    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/02e201f9-5c67-4fc3-8822-8c39e516dddb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"02e201f9-5c67-4fc3-8822-8c39e516dddb\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") "
Sep 18 17:52:53.026327 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.026004    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/02e201f9-5c67-4fc3-8822-8c39e516dddb-thanos-prometheus-http-client-file\") pod \"02e201f9-5c67-4fc3-8822-8c39e516dddb\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") "
Sep 18 17:52:53.026327 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.026032    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/02e201f9-5c67-4fc3-8822-8c39e516dddb-tls-assets\") pod \"02e201f9-5c67-4fc3-8822-8c39e516dddb\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") "
Sep 18 17:52:53.026327 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.026058    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/02e201f9-5c67-4fc3-8822-8c39e516dddb-web-config\") pod \"02e201f9-5c67-4fc3-8822-8c39e516dddb\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") "
Sep 18 17:52:53.026327 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.025975    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02e201f9-5c67-4fc3-8822-8c39e516dddb-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "02e201f9-5c67-4fc3-8822-8c39e516dddb" (UID: "02e201f9-5c67-4fc3-8822-8c39e516dddb"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:52:53.026327 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.026084    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/02e201f9-5c67-4fc3-8822-8c39e516dddb-prometheus-k8s-rulefiles-2\") pod \"02e201f9-5c67-4fc3-8822-8c39e516dddb\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") "
Sep 18 17:52:53.026327 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.026156    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/02e201f9-5c67-4fc3-8822-8c39e516dddb-prometheus-k8s-rulefiles-1\") pod \"02e201f9-5c67-4fc3-8822-8c39e516dddb\" (UID: \"02e201f9-5c67-4fc3-8822-8c39e516dddb\") "
Sep 18 17:52:53.026743 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.026425    2561 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/02e201f9-5c67-4fc3-8822-8c39e516dddb-configmap-metrics-client-ca\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 17:52:53.026743 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.026468    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02e201f9-5c67-4fc3-8822-8c39e516dddb-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "02e201f9-5c67-4fc3-8822-8c39e516dddb" (UID: "02e201f9-5c67-4fc3-8822-8c39e516dddb"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:52:53.026743 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.026479    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02e201f9-5c67-4fc3-8822-8c39e516dddb-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "02e201f9-5c67-4fc3-8822-8c39e516dddb" (UID: "02e201f9-5c67-4fc3-8822-8c39e516dddb"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:52:53.026743 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.026718    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02e201f9-5c67-4fc3-8822-8c39e516dddb-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "02e201f9-5c67-4fc3-8822-8c39e516dddb" (UID: "02e201f9-5c67-4fc3-8822-8c39e516dddb"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:52:53.027280 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.026990    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02e201f9-5c67-4fc3-8822-8c39e516dddb-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "02e201f9-5c67-4fc3-8822-8c39e516dddb" (UID: "02e201f9-5c67-4fc3-8822-8c39e516dddb"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:52:53.027280 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.027229    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02e201f9-5c67-4fc3-8822-8c39e516dddb-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "02e201f9-5c67-4fc3-8822-8c39e516dddb" (UID: "02e201f9-5c67-4fc3-8822-8c39e516dddb"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:52:53.028143 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.027859    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02e201f9-5c67-4fc3-8822-8c39e516dddb-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "02e201f9-5c67-4fc3-8822-8c39e516dddb" (UID: "02e201f9-5c67-4fc3-8822-8c39e516dddb"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:52:53.028615 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.028525    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02e201f9-5c67-4fc3-8822-8c39e516dddb-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "02e201f9-5c67-4fc3-8822-8c39e516dddb" (UID: "02e201f9-5c67-4fc3-8822-8c39e516dddb"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:52:53.028789 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.028756    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02e201f9-5c67-4fc3-8822-8c39e516dddb-kube-api-access-qp2sp" (OuterVolumeSpecName: "kube-api-access-qp2sp") pod "02e201f9-5c67-4fc3-8822-8c39e516dddb" (UID: "02e201f9-5c67-4fc3-8822-8c39e516dddb"). InnerVolumeSpecName "kube-api-access-qp2sp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:52:53.028886 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.028858    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02e201f9-5c67-4fc3-8822-8c39e516dddb-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "02e201f9-5c67-4fc3-8822-8c39e516dddb" (UID: "02e201f9-5c67-4fc3-8822-8c39e516dddb"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:52:53.029367 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.029331    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02e201f9-5c67-4fc3-8822-8c39e516dddb-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "02e201f9-5c67-4fc3-8822-8c39e516dddb" (UID: "02e201f9-5c67-4fc3-8822-8c39e516dddb"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:52:53.029805 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.029608    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02e201f9-5c67-4fc3-8822-8c39e516dddb-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "02e201f9-5c67-4fc3-8822-8c39e516dddb" (UID: "02e201f9-5c67-4fc3-8822-8c39e516dddb"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:52:53.029805 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.029671    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02e201f9-5c67-4fc3-8822-8c39e516dddb-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "02e201f9-5c67-4fc3-8822-8c39e516dddb" (UID: "02e201f9-5c67-4fc3-8822-8c39e516dddb"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:52:53.030102 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.030079    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02e201f9-5c67-4fc3-8822-8c39e516dddb-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "02e201f9-5c67-4fc3-8822-8c39e516dddb" (UID: "02e201f9-5c67-4fc3-8822-8c39e516dddb"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:52:53.030591 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.030560    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02e201f9-5c67-4fc3-8822-8c39e516dddb-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "02e201f9-5c67-4fc3-8822-8c39e516dddb" (UID: "02e201f9-5c67-4fc3-8822-8c39e516dddb"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:52:53.030797 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.030780    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02e201f9-5c67-4fc3-8822-8c39e516dddb-config-out" (OuterVolumeSpecName: "config-out") pod "02e201f9-5c67-4fc3-8822-8c39e516dddb" (UID: "02e201f9-5c67-4fc3-8822-8c39e516dddb"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:52:53.031022 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.031004    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02e201f9-5c67-4fc3-8822-8c39e516dddb-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "02e201f9-5c67-4fc3-8822-8c39e516dddb" (UID: "02e201f9-5c67-4fc3-8822-8c39e516dddb"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:52:53.031096 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.031082    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02e201f9-5c67-4fc3-8822-8c39e516dddb-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "02e201f9-5c67-4fc3-8822-8c39e516dddb" (UID: "02e201f9-5c67-4fc3-8822-8c39e516dddb"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:52:53.031479 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.031461    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02e201f9-5c67-4fc3-8822-8c39e516dddb-config" (OuterVolumeSpecName: "config") pod "02e201f9-5c67-4fc3-8822-8c39e516dddb" (UID: "02e201f9-5c67-4fc3-8822-8c39e516dddb"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:52:53.041441 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.041417    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02e201f9-5c67-4fc3-8822-8c39e516dddb-web-config" (OuterVolumeSpecName: "web-config") pod "02e201f9-5c67-4fc3-8822-8c39e516dddb" (UID: "02e201f9-5c67-4fc3-8822-8c39e516dddb"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:52:53.127672 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.127639    2561 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/02e201f9-5c67-4fc3-8822-8c39e516dddb-thanos-prometheus-http-client-file\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 17:52:53.127672 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.127666    2561 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/02e201f9-5c67-4fc3-8822-8c39e516dddb-tls-assets\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 17:52:53.127849 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.127696    2561 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/02e201f9-5c67-4fc3-8822-8c39e516dddb-web-config\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 17:52:53.127849 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.127704    2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/02e201f9-5c67-4fc3-8822-8c39e516dddb-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 17:52:53.127849 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.127713    2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/02e201f9-5c67-4fc3-8822-8c39e516dddb-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 17:52:53.127849 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.127723    2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/02e201f9-5c67-4fc3-8822-8c39e516dddb-prometheus-k8s-db\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 17:52:53.127849 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.127732    2561 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02e201f9-5c67-4fc3-8822-8c39e516dddb-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 17:52:53.127849 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.127741    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp2sp\" (UniqueName: \"kubernetes.io/projected/02e201f9-5c67-4fc3-8822-8c39e516dddb-kube-api-access-qp2sp\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 17:52:53.127849 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.127750    2561 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02e201f9-5c67-4fc3-8822-8c39e516dddb-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 17:52:53.127849 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.127759    2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/02e201f9-5c67-4fc3-8822-8c39e516dddb-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 17:52:53.127849 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.127767    2561 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/02e201f9-5c67-4fc3-8822-8c39e516dddb-secret-grpc-tls\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 17:52:53.127849 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.127775    2561 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/02e201f9-5c67-4fc3-8822-8c39e516dddb-secret-prometheus-k8s-tls\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 17:52:53.127849 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.127783    2561 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/02e201f9-5c67-4fc3-8822-8c39e516dddb-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 17:52:53.127849 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.127792    2561 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/02e201f9-5c67-4fc3-8822-8c39e516dddb-secret-metrics-client-certs\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 17:52:53.127849 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.127800    2561 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/02e201f9-5c67-4fc3-8822-8c39e516dddb-config-out\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 17:52:53.127849 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.127808    2561 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/02e201f9-5c67-4fc3-8822-8c39e516dddb-secret-kube-rbac-proxy\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 17:52:53.127849 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.127816    2561 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/02e201f9-5c67-4fc3-8822-8c39e516dddb-config\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 17:52:53.127849 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.127824    2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02e201f9-5c67-4fc3-8822-8c39e516dddb-prometheus-trusted-ca-bundle\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 17:52:53.127849 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.127833    2561 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/02e201f9-5c67-4fc3-8822-8c39e516dddb-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 17:52:53.729106 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.729070    2561 generic.go:356] "Generic (PLEG): container finished" podID="02e201f9-5c67-4fc3-8822-8c39e516dddb" containerID="cac905f0d7e26eff0b121e5dac51d2dbbc047ff087d7c85991c1ad3270370cf6" exitCode=0
Sep 18 17:52:53.729106 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.729094    2561 generic.go:356] "Generic (PLEG): container finished" podID="02e201f9-5c67-4fc3-8822-8c39e516dddb" containerID="5ee666f651430a2827e3e851debe8643e8ac394b23660b97351850784d1698da" exitCode=0
Sep 18 17:52:53.729106 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.729101    2561 generic.go:356] "Generic (PLEG): container finished" podID="02e201f9-5c67-4fc3-8822-8c39e516dddb" containerID="11d68315b27586f2006e2848127343625491c24b02b71ca0dbe7049a5de8bc82" exitCode=0
Sep 18 17:52:53.729106 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.729107    2561 generic.go:356] "Generic (PLEG): container finished" podID="02e201f9-5c67-4fc3-8822-8c39e516dddb" containerID="4fa431a5325cc8da90d0a10f85db0a7e546046597b2bab50093fd69874855b04" exitCode=0
Sep 18 17:52:53.729485 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.729113    2561 generic.go:356] "Generic (PLEG): container finished" podID="02e201f9-5c67-4fc3-8822-8c39e516dddb" containerID="cae77bd82a6938cf314fffa5dbeb4ed4f67024f1d5e28447c519697f39c9bb8c" exitCode=0
Sep 18 17:52:53.729485 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.729138    2561 generic.go:356] "Generic (PLEG): container finished" podID="02e201f9-5c67-4fc3-8822-8c39e516dddb" containerID="5d47cee9475e7b69a9c0e95a5d58e85210157a0a019820b878bb75a0f376afb7" exitCode=0
Sep 18 17:52:53.729485 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.729163    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:52:53.729485 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.729157    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"02e201f9-5c67-4fc3-8822-8c39e516dddb","Type":"ContainerDied","Data":"cac905f0d7e26eff0b121e5dac51d2dbbc047ff087d7c85991c1ad3270370cf6"}
Sep 18 17:52:53.729485 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.729251    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"02e201f9-5c67-4fc3-8822-8c39e516dddb","Type":"ContainerDied","Data":"5ee666f651430a2827e3e851debe8643e8ac394b23660b97351850784d1698da"}
Sep 18 17:52:53.729485 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.729293    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"02e201f9-5c67-4fc3-8822-8c39e516dddb","Type":"ContainerDied","Data":"11d68315b27586f2006e2848127343625491c24b02b71ca0dbe7049a5de8bc82"}
Sep 18 17:52:53.729485 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.729316    2561 scope.go:117] "RemoveContainer" containerID="cac905f0d7e26eff0b121e5dac51d2dbbc047ff087d7c85991c1ad3270370cf6"
Sep 18 17:52:53.729485 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.729330    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"02e201f9-5c67-4fc3-8822-8c39e516dddb","Type":"ContainerDied","Data":"4fa431a5325cc8da90d0a10f85db0a7e546046597b2bab50093fd69874855b04"}
Sep 18 17:52:53.729485 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.729374    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"02e201f9-5c67-4fc3-8822-8c39e516dddb","Type":"ContainerDied","Data":"cae77bd82a6938cf314fffa5dbeb4ed4f67024f1d5e28447c519697f39c9bb8c"}
Sep 18 17:52:53.729485 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.729415    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"02e201f9-5c67-4fc3-8822-8c39e516dddb","Type":"ContainerDied","Data":"5d47cee9475e7b69a9c0e95a5d58e85210157a0a019820b878bb75a0f376afb7"}
Sep 18 17:52:53.729485 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.729457    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"02e201f9-5c67-4fc3-8822-8c39e516dddb","Type":"ContainerDied","Data":"d1e1be8d9f7b9e6c99c96ce23552c191ff551ce8655c7db8fd77ed5aa6d04d8c"}
Sep 18 17:52:53.740589 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.739232    2561 scope.go:117] "RemoveContainer" containerID="5ee666f651430a2827e3e851debe8643e8ac394b23660b97351850784d1698da"
Sep 18 17:52:53.748052 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.748037    2561 scope.go:117] "RemoveContainer" containerID="11d68315b27586f2006e2848127343625491c24b02b71ca0dbe7049a5de8bc82"
Sep 18 17:52:53.754558 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.754539    2561 scope.go:117] "RemoveContainer" containerID="4fa431a5325cc8da90d0a10f85db0a7e546046597b2bab50093fd69874855b04"
Sep 18 17:52:53.757766 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.757745    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 17:52:53.761933 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.761916    2561 scope.go:117] "RemoveContainer" containerID="cae77bd82a6938cf314fffa5dbeb4ed4f67024f1d5e28447c519697f39c9bb8c"
Sep 18 17:52:53.767025 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.767005    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 17:52:53.770855 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.770833    2561 scope.go:117] "RemoveContainer" containerID="5d47cee9475e7b69a9c0e95a5d58e85210157a0a019820b878bb75a0f376afb7"
Sep 18 17:52:53.778370 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.778353    2561 scope.go:117] "RemoveContainer" containerID="6a70cb9a75cc5f915d386219b8c5ab68bd8ffd36a1a018530b26cc0a20c7ef62"
Sep 18 17:52:53.791970 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.791955    2561 scope.go:117] "RemoveContainer" containerID="cac905f0d7e26eff0b121e5dac51d2dbbc047ff087d7c85991c1ad3270370cf6"
Sep 18 17:52:53.792271 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:52:53.792247    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cac905f0d7e26eff0b121e5dac51d2dbbc047ff087d7c85991c1ad3270370cf6\": container with ID starting with cac905f0d7e26eff0b121e5dac51d2dbbc047ff087d7c85991c1ad3270370cf6 not found: ID does not exist" containerID="cac905f0d7e26eff0b121e5dac51d2dbbc047ff087d7c85991c1ad3270370cf6"
Sep 18 17:52:53.792353 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.792278    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cac905f0d7e26eff0b121e5dac51d2dbbc047ff087d7c85991c1ad3270370cf6"} err="failed to get container status \"cac905f0d7e26eff0b121e5dac51d2dbbc047ff087d7c85991c1ad3270370cf6\": rpc error: code = NotFound desc = could not find container \"cac905f0d7e26eff0b121e5dac51d2dbbc047ff087d7c85991c1ad3270370cf6\": container with ID starting with cac905f0d7e26eff0b121e5dac51d2dbbc047ff087d7c85991c1ad3270370cf6 not found: ID does not exist"
Sep 18 17:52:53.792353 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.792300    2561 scope.go:117] "RemoveContainer" containerID="5ee666f651430a2827e3e851debe8643e8ac394b23660b97351850784d1698da"
Sep 18 17:52:53.792557 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:52:53.792539    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ee666f651430a2827e3e851debe8643e8ac394b23660b97351850784d1698da\": container with ID starting with 5ee666f651430a2827e3e851debe8643e8ac394b23660b97351850784d1698da not found: ID does not exist" containerID="5ee666f651430a2827e3e851debe8643e8ac394b23660b97351850784d1698da"
Sep 18 17:52:53.792594 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.792564    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ee666f651430a2827e3e851debe8643e8ac394b23660b97351850784d1698da"} err="failed to get container status \"5ee666f651430a2827e3e851debe8643e8ac394b23660b97351850784d1698da\": rpc error: code = NotFound desc = could not find container \"5ee666f651430a2827e3e851debe8643e8ac394b23660b97351850784d1698da\": container with ID starting with 5ee666f651430a2827e3e851debe8643e8ac394b23660b97351850784d1698da not found: ID does not exist"
Sep 18 17:52:53.792594 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.792580    2561 scope.go:117] "RemoveContainer" containerID="11d68315b27586f2006e2848127343625491c24b02b71ca0dbe7049a5de8bc82"
Sep 18 17:52:53.792845 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:52:53.792826    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11d68315b27586f2006e2848127343625491c24b02b71ca0dbe7049a5de8bc82\": container with ID starting with 11d68315b27586f2006e2848127343625491c24b02b71ca0dbe7049a5de8bc82 not found: ID does not exist" containerID="11d68315b27586f2006e2848127343625491c24b02b71ca0dbe7049a5de8bc82"
Sep 18 17:52:53.792880 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.792851    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11d68315b27586f2006e2848127343625491c24b02b71ca0dbe7049a5de8bc82"} err="failed to get container status \"11d68315b27586f2006e2848127343625491c24b02b71ca0dbe7049a5de8bc82\": rpc error: code = NotFound desc = could not find container \"11d68315b27586f2006e2848127343625491c24b02b71ca0dbe7049a5de8bc82\": container with ID starting with 11d68315b27586f2006e2848127343625491c24b02b71ca0dbe7049a5de8bc82 not found: ID does not exist"
Sep 18 17:52:53.792880 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.792866    2561 scope.go:117] "RemoveContainer" containerID="4fa431a5325cc8da90d0a10f85db0a7e546046597b2bab50093fd69874855b04"
Sep 18 17:52:53.793159 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:52:53.793109    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fa431a5325cc8da90d0a10f85db0a7e546046597b2bab50093fd69874855b04\": container with ID starting with 4fa431a5325cc8da90d0a10f85db0a7e546046597b2bab50093fd69874855b04 not found: ID does not exist" containerID="4fa431a5325cc8da90d0a10f85db0a7e546046597b2bab50093fd69874855b04"
Sep 18 17:52:53.793207 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.793165    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fa431a5325cc8da90d0a10f85db0a7e546046597b2bab50093fd69874855b04"} err="failed to get container status \"4fa431a5325cc8da90d0a10f85db0a7e546046597b2bab50093fd69874855b04\": rpc error: code = NotFound desc = could not find container \"4fa431a5325cc8da90d0a10f85db0a7e546046597b2bab50093fd69874855b04\": container with ID starting with 4fa431a5325cc8da90d0a10f85db0a7e546046597b2bab50093fd69874855b04 not found: ID does not exist"
Sep 18 17:52:53.793207 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.793179    2561 scope.go:117] "RemoveContainer" containerID="cae77bd82a6938cf314fffa5dbeb4ed4f67024f1d5e28447c519697f39c9bb8c"
Sep 18 17:52:53.793415 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:52:53.793399    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cae77bd82a6938cf314fffa5dbeb4ed4f67024f1d5e28447c519697f39c9bb8c\": container with ID starting with cae77bd82a6938cf314fffa5dbeb4ed4f67024f1d5e28447c519697f39c9bb8c not found: ID does not exist" containerID="cae77bd82a6938cf314fffa5dbeb4ed4f67024f1d5e28447c519697f39c9bb8c"
Sep 18 17:52:53.793455 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.793420    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cae77bd82a6938cf314fffa5dbeb4ed4f67024f1d5e28447c519697f39c9bb8c"} err="failed to get container status \"cae77bd82a6938cf314fffa5dbeb4ed4f67024f1d5e28447c519697f39c9bb8c\": rpc error: code = NotFound desc = could not find container \"cae77bd82a6938cf314fffa5dbeb4ed4f67024f1d5e28447c519697f39c9bb8c\": container with ID starting with cae77bd82a6938cf314fffa5dbeb4ed4f67024f1d5e28447c519697f39c9bb8c not found: ID does not exist"
Sep 18 17:52:53.793455 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.793433    2561 scope.go:117] "RemoveContainer" containerID="5d47cee9475e7b69a9c0e95a5d58e85210157a0a019820b878bb75a0f376afb7"
Sep 18 17:52:53.793662 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:52:53.793645    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d47cee9475e7b69a9c0e95a5d58e85210157a0a019820b878bb75a0f376afb7\": container with ID starting with 5d47cee9475e7b69a9c0e95a5d58e85210157a0a019820b878bb75a0f376afb7 not found: ID does not exist" containerID="5d47cee9475e7b69a9c0e95a5d58e85210157a0a019820b878bb75a0f376afb7"
Sep 18 17:52:53.793705 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.793667    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d47cee9475e7b69a9c0e95a5d58e85210157a0a019820b878bb75a0f376afb7"} err="failed to get container status \"5d47cee9475e7b69a9c0e95a5d58e85210157a0a019820b878bb75a0f376afb7\": rpc error: code = NotFound desc = could not find container \"5d47cee9475e7b69a9c0e95a5d58e85210157a0a019820b878bb75a0f376afb7\": container with ID starting with 5d47cee9475e7b69a9c0e95a5d58e85210157a0a019820b878bb75a0f376afb7 not found: ID does not exist"
Sep 18 17:52:53.793705 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.793681    2561 scope.go:117] "RemoveContainer" containerID="6a70cb9a75cc5f915d386219b8c5ab68bd8ffd36a1a018530b26cc0a20c7ef62"
Sep 18 17:52:53.793917 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:52:53.793894    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a70cb9a75cc5f915d386219b8c5ab68bd8ffd36a1a018530b26cc0a20c7ef62\": container with ID starting with 6a70cb9a75cc5f915d386219b8c5ab68bd8ffd36a1a018530b26cc0a20c7ef62 not found: ID does not exist" containerID="6a70cb9a75cc5f915d386219b8c5ab68bd8ffd36a1a018530b26cc0a20c7ef62"
Sep 18 17:52:53.793958 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.793921    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a70cb9a75cc5f915d386219b8c5ab68bd8ffd36a1a018530b26cc0a20c7ef62"} err="failed to get container status \"6a70cb9a75cc5f915d386219b8c5ab68bd8ffd36a1a018530b26cc0a20c7ef62\": rpc error: code = NotFound desc = could not find container \"6a70cb9a75cc5f915d386219b8c5ab68bd8ffd36a1a018530b26cc0a20c7ef62\": container with ID starting with 6a70cb9a75cc5f915d386219b8c5ab68bd8ffd36a1a018530b26cc0a20c7ef62 not found: ID does not exist"
Sep 18 17:52:53.793958 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.793935    2561 scope.go:117] "RemoveContainer" containerID="cac905f0d7e26eff0b121e5dac51d2dbbc047ff087d7c85991c1ad3270370cf6"
Sep 18 17:52:53.794156 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.794141    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cac905f0d7e26eff0b121e5dac51d2dbbc047ff087d7c85991c1ad3270370cf6"} err="failed to get container status \"cac905f0d7e26eff0b121e5dac51d2dbbc047ff087d7c85991c1ad3270370cf6\": rpc error: code = NotFound desc = could not find container \"cac905f0d7e26eff0b121e5dac51d2dbbc047ff087d7c85991c1ad3270370cf6\": container with ID starting with cac905f0d7e26eff0b121e5dac51d2dbbc047ff087d7c85991c1ad3270370cf6 not found: ID does not exist"
Sep 18 17:52:53.794200 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.794156    2561 scope.go:117] "RemoveContainer" containerID="5ee666f651430a2827e3e851debe8643e8ac394b23660b97351850784d1698da"
Sep 18 17:52:53.794381 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.794359    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ee666f651430a2827e3e851debe8643e8ac394b23660b97351850784d1698da"} err="failed to get container status \"5ee666f651430a2827e3e851debe8643e8ac394b23660b97351850784d1698da\": rpc error: code = NotFound desc = could not find container \"5ee666f651430a2827e3e851debe8643e8ac394b23660b97351850784d1698da\": container with ID starting with 5ee666f651430a2827e3e851debe8643e8ac394b23660b97351850784d1698da not found: ID does not exist"
Sep 18 17:52:53.794381 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.794382    2561 scope.go:117] "RemoveContainer" containerID="11d68315b27586f2006e2848127343625491c24b02b71ca0dbe7049a5de8bc82"
Sep 18 17:52:53.794619 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.794602    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11d68315b27586f2006e2848127343625491c24b02b71ca0dbe7049a5de8bc82"} err="failed to get container status \"11d68315b27586f2006e2848127343625491c24b02b71ca0dbe7049a5de8bc82\": rpc error: code = NotFound desc = could not find container \"11d68315b27586f2006e2848127343625491c24b02b71ca0dbe7049a5de8bc82\": container with ID starting with 11d68315b27586f2006e2848127343625491c24b02b71ca0dbe7049a5de8bc82 not found: ID does not exist"
Sep 18 17:52:53.794661 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.794622    2561 scope.go:117] "RemoveContainer" containerID="4fa431a5325cc8da90d0a10f85db0a7e546046597b2bab50093fd69874855b04"
Sep 18 17:52:53.794831 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.794814    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fa431a5325cc8da90d0a10f85db0a7e546046597b2bab50093fd69874855b04"} err="failed to get container status \"4fa431a5325cc8da90d0a10f85db0a7e546046597b2bab50093fd69874855b04\": rpc error: code = NotFound desc = could not find container \"4fa431a5325cc8da90d0a10f85db0a7e546046597b2bab50093fd69874855b04\": container with ID starting with 4fa431a5325cc8da90d0a10f85db0a7e546046597b2bab50093fd69874855b04 not found: ID does not exist"
Sep 18 17:52:53.794902 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.794833    2561 scope.go:117] "RemoveContainer" containerID="cae77bd82a6938cf314fffa5dbeb4ed4f67024f1d5e28447c519697f39c9bb8c"
Sep 18 17:52:53.795068 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.795053    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cae77bd82a6938cf314fffa5dbeb4ed4f67024f1d5e28447c519697f39c9bb8c"} err="failed to get container status \"cae77bd82a6938cf314fffa5dbeb4ed4f67024f1d5e28447c519697f39c9bb8c\": rpc error: code = NotFound desc = could not find container \"cae77bd82a6938cf314fffa5dbeb4ed4f67024f1d5e28447c519697f39c9bb8c\": container with ID starting with cae77bd82a6938cf314fffa5dbeb4ed4f67024f1d5e28447c519697f39c9bb8c not found: ID does not exist"
Sep 18 17:52:53.795112 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.795068    2561 scope.go:117] "RemoveContainer" containerID="5d47cee9475e7b69a9c0e95a5d58e85210157a0a019820b878bb75a0f376afb7"
Sep 18 17:52:53.795284 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.795267    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d47cee9475e7b69a9c0e95a5d58e85210157a0a019820b878bb75a0f376afb7"} err="failed to get container status \"5d47cee9475e7b69a9c0e95a5d58e85210157a0a019820b878bb75a0f376afb7\": rpc error: code = NotFound desc = could not find container \"5d47cee9475e7b69a9c0e95a5d58e85210157a0a019820b878bb75a0f376afb7\": container with ID starting with 5d47cee9475e7b69a9c0e95a5d58e85210157a0a019820b878bb75a0f376afb7 not found: ID does not exist"
Sep 18 17:52:53.795351 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.795287    2561 scope.go:117] "RemoveContainer" containerID="6a70cb9a75cc5f915d386219b8c5ab68bd8ffd36a1a018530b26cc0a20c7ef62"
Sep 18 17:52:53.795508 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.795493    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a70cb9a75cc5f915d386219b8c5ab68bd8ffd36a1a018530b26cc0a20c7ef62"} err="failed to get container status \"6a70cb9a75cc5f915d386219b8c5ab68bd8ffd36a1a018530b26cc0a20c7ef62\": rpc error: code = NotFound desc = could not find container \"6a70cb9a75cc5f915d386219b8c5ab68bd8ffd36a1a018530b26cc0a20c7ef62\": container with ID starting with 6a70cb9a75cc5f915d386219b8c5ab68bd8ffd36a1a018530b26cc0a20c7ef62 not found: ID does not exist"
Sep 18 17:52:53.795552 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.795507    2561 scope.go:117] "RemoveContainer" containerID="cac905f0d7e26eff0b121e5dac51d2dbbc047ff087d7c85991c1ad3270370cf6"
Sep 18 17:52:53.795697 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.795680    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cac905f0d7e26eff0b121e5dac51d2dbbc047ff087d7c85991c1ad3270370cf6"} err="failed to get container status \"cac905f0d7e26eff0b121e5dac51d2dbbc047ff087d7c85991c1ad3270370cf6\": rpc error: code = NotFound desc = could not find container \"cac905f0d7e26eff0b121e5dac51d2dbbc047ff087d7c85991c1ad3270370cf6\": container with ID starting with cac905f0d7e26eff0b121e5dac51d2dbbc047ff087d7c85991c1ad3270370cf6 not found: ID does not exist"
Sep 18 17:52:53.795764 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.795698    2561 scope.go:117] "RemoveContainer" containerID="5ee666f651430a2827e3e851debe8643e8ac394b23660b97351850784d1698da"
Sep 18 17:52:53.795933 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.795918    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ee666f651430a2827e3e851debe8643e8ac394b23660b97351850784d1698da"} err="failed to get container status \"5ee666f651430a2827e3e851debe8643e8ac394b23660b97351850784d1698da\": rpc error: code = NotFound desc = could not find container \"5ee666f651430a2827e3e851debe8643e8ac394b23660b97351850784d1698da\": container with ID starting with 5ee666f651430a2827e3e851debe8643e8ac394b23660b97351850784d1698da not found: ID does not exist"
Sep 18 17:52:53.795980 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.795934    2561 scope.go:117] "RemoveContainer" containerID="11d68315b27586f2006e2848127343625491c24b02b71ca0dbe7049a5de8bc82"
Sep 18 17:52:53.796175 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.796155    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11d68315b27586f2006e2848127343625491c24b02b71ca0dbe7049a5de8bc82"} err="failed to get container status \"11d68315b27586f2006e2848127343625491c24b02b71ca0dbe7049a5de8bc82\": rpc error: code = NotFound desc = could not find container \"11d68315b27586f2006e2848127343625491c24b02b71ca0dbe7049a5de8bc82\": container with ID starting with 11d68315b27586f2006e2848127343625491c24b02b71ca0dbe7049a5de8bc82 not found: ID does not exist"
Sep 18 17:52:53.796245 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.796175    2561 scope.go:117] "RemoveContainer" containerID="4fa431a5325cc8da90d0a10f85db0a7e546046597b2bab50093fd69874855b04"
Sep 18 17:52:53.796418 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.796397    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fa431a5325cc8da90d0a10f85db0a7e546046597b2bab50093fd69874855b04"} err="failed to get container status \"4fa431a5325cc8da90d0a10f85db0a7e546046597b2bab50093fd69874855b04\": rpc error: code = NotFound desc = could not find container \"4fa431a5325cc8da90d0a10f85db0a7e546046597b2bab50093fd69874855b04\": container with ID starting with 4fa431a5325cc8da90d0a10f85db0a7e546046597b2bab50093fd69874855b04 not found: ID does not exist"
Sep 18 17:52:53.796504 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.796419    2561 scope.go:117] "RemoveContainer" containerID="cae77bd82a6938cf314fffa5dbeb4ed4f67024f1d5e28447c519697f39c9bb8c"
Sep 18 17:52:53.796691 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.796673    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cae77bd82a6938cf314fffa5dbeb4ed4f67024f1d5e28447c519697f39c9bb8c"} err="failed to get container status \"cae77bd82a6938cf314fffa5dbeb4ed4f67024f1d5e28447c519697f39c9bb8c\": rpc error: code = NotFound desc = could not find container \"cae77bd82a6938cf314fffa5dbeb4ed4f67024f1d5e28447c519697f39c9bb8c\": container with ID starting with cae77bd82a6938cf314fffa5dbeb4ed4f67024f1d5e28447c519697f39c9bb8c not found: ID does not exist"
Sep 18 17:52:53.796747 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.796691    2561 scope.go:117] "RemoveContainer" containerID="5d47cee9475e7b69a9c0e95a5d58e85210157a0a019820b878bb75a0f376afb7"
Sep 18 17:52:53.796897 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.796880    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d47cee9475e7b69a9c0e95a5d58e85210157a0a019820b878bb75a0f376afb7"} err="failed to get container status \"5d47cee9475e7b69a9c0e95a5d58e85210157a0a019820b878bb75a0f376afb7\": rpc error: code = NotFound desc = could not find container \"5d47cee9475e7b69a9c0e95a5d58e85210157a0a019820b878bb75a0f376afb7\": container with ID starting with 5d47cee9475e7b69a9c0e95a5d58e85210157a0a019820b878bb75a0f376afb7 not found: ID does not exist"
Sep 18 17:52:53.796897 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.796895    2561 scope.go:117] "RemoveContainer" containerID="6a70cb9a75cc5f915d386219b8c5ab68bd8ffd36a1a018530b26cc0a20c7ef62"
Sep 18 17:52:53.797096 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.797078    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a70cb9a75cc5f915d386219b8c5ab68bd8ffd36a1a018530b26cc0a20c7ef62"} err="failed to get container status \"6a70cb9a75cc5f915d386219b8c5ab68bd8ffd36a1a018530b26cc0a20c7ef62\": rpc error: code = NotFound desc = could not find container \"6a70cb9a75cc5f915d386219b8c5ab68bd8ffd36a1a018530b26cc0a20c7ef62\": container with ID starting with 6a70cb9a75cc5f915d386219b8c5ab68bd8ffd36a1a018530b26cc0a20c7ef62 not found: ID does not exist"
Sep 18 17:52:53.797155 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.797097    2561 scope.go:117] "RemoveContainer" containerID="cac905f0d7e26eff0b121e5dac51d2dbbc047ff087d7c85991c1ad3270370cf6"
Sep 18 17:52:53.797340 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.797325    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cac905f0d7e26eff0b121e5dac51d2dbbc047ff087d7c85991c1ad3270370cf6"} err="failed to get container status \"cac905f0d7e26eff0b121e5dac51d2dbbc047ff087d7c85991c1ad3270370cf6\": rpc error: code = NotFound desc = could not find container \"cac905f0d7e26eff0b121e5dac51d2dbbc047ff087d7c85991c1ad3270370cf6\": container with ID starting with cac905f0d7e26eff0b121e5dac51d2dbbc047ff087d7c85991c1ad3270370cf6 not found: ID does not exist"
Sep 18 17:52:53.797384 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.797339    2561 scope.go:117] "RemoveContainer" containerID="5ee666f651430a2827e3e851debe8643e8ac394b23660b97351850784d1698da"
Sep 18 17:52:53.797558 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.797541    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ee666f651430a2827e3e851debe8643e8ac394b23660b97351850784d1698da"} err="failed to get container status \"5ee666f651430a2827e3e851debe8643e8ac394b23660b97351850784d1698da\": rpc error: code = NotFound desc = could not find container \"5ee666f651430a2827e3e851debe8643e8ac394b23660b97351850784d1698da\": container with ID starting with 5ee666f651430a2827e3e851debe8643e8ac394b23660b97351850784d1698da not found: ID does not exist"
Sep 18 17:52:53.797634 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.797560    2561 scope.go:117] "RemoveContainer" containerID="11d68315b27586f2006e2848127343625491c24b02b71ca0dbe7049a5de8bc82"
Sep 18 17:52:53.797776 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.797756    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11d68315b27586f2006e2848127343625491c24b02b71ca0dbe7049a5de8bc82"} err="failed to get container status \"11d68315b27586f2006e2848127343625491c24b02b71ca0dbe7049a5de8bc82\": rpc error: code = NotFound desc = could not find container \"11d68315b27586f2006e2848127343625491c24b02b71ca0dbe7049a5de8bc82\": container with ID starting with 11d68315b27586f2006e2848127343625491c24b02b71ca0dbe7049a5de8bc82 not found: ID does not exist"
Sep 18 17:52:53.797776 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.797775    2561 scope.go:117] "RemoveContainer" containerID="4fa431a5325cc8da90d0a10f85db0a7e546046597b2bab50093fd69874855b04"
Sep 18 17:52:53.798091 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.798071    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fa431a5325cc8da90d0a10f85db0a7e546046597b2bab50093fd69874855b04"} err="failed to get container status \"4fa431a5325cc8da90d0a10f85db0a7e546046597b2bab50093fd69874855b04\": rpc error: code = NotFound desc = could not find container \"4fa431a5325cc8da90d0a10f85db0a7e546046597b2bab50093fd69874855b04\": container with ID starting with 4fa431a5325cc8da90d0a10f85db0a7e546046597b2bab50093fd69874855b04 not found: ID does not exist"
Sep 18 17:52:53.798091 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.798092    2561 scope.go:117] "RemoveContainer" containerID="cae77bd82a6938cf314fffa5dbeb4ed4f67024f1d5e28447c519697f39c9bb8c"
Sep 18 17:52:53.798445 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.798342    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cae77bd82a6938cf314fffa5dbeb4ed4f67024f1d5e28447c519697f39c9bb8c"} err="failed to get container status \"cae77bd82a6938cf314fffa5dbeb4ed4f67024f1d5e28447c519697f39c9bb8c\": rpc error: code = NotFound desc = could not find container \"cae77bd82a6938cf314fffa5dbeb4ed4f67024f1d5e28447c519697f39c9bb8c\": container with ID starting with cae77bd82a6938cf314fffa5dbeb4ed4f67024f1d5e28447c519697f39c9bb8c not found: ID does not exist"
Sep 18 17:52:53.798445 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.798366    2561 scope.go:117] "RemoveContainer" containerID="5d47cee9475e7b69a9c0e95a5d58e85210157a0a019820b878bb75a0f376afb7"
Sep 18 17:52:53.798632 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.798615    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d47cee9475e7b69a9c0e95a5d58e85210157a0a019820b878bb75a0f376afb7"} err="failed to get container status \"5d47cee9475e7b69a9c0e95a5d58e85210157a0a019820b878bb75a0f376afb7\": rpc error: code = NotFound desc = could not find container \"5d47cee9475e7b69a9c0e95a5d58e85210157a0a019820b878bb75a0f376afb7\": container with ID starting with 5d47cee9475e7b69a9c0e95a5d58e85210157a0a019820b878bb75a0f376afb7 not found: ID does not exist"
Sep 18 17:52:53.798674 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.798633    2561 scope.go:117] "RemoveContainer" containerID="6a70cb9a75cc5f915d386219b8c5ab68bd8ffd36a1a018530b26cc0a20c7ef62"
Sep 18 17:52:53.798875 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.798857    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a70cb9a75cc5f915d386219b8c5ab68bd8ffd36a1a018530b26cc0a20c7ef62"} err="failed to get container status \"6a70cb9a75cc5f915d386219b8c5ab68bd8ffd36a1a018530b26cc0a20c7ef62\": rpc error: code = NotFound desc = could not find container \"6a70cb9a75cc5f915d386219b8c5ab68bd8ffd36a1a018530b26cc0a20c7ef62\": container with ID starting with 6a70cb9a75cc5f915d386219b8c5ab68bd8ffd36a1a018530b26cc0a20c7ef62 not found: ID does not exist"
Sep 18 17:52:53.798913 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.798877    2561 scope.go:117] "RemoveContainer" containerID="cac905f0d7e26eff0b121e5dac51d2dbbc047ff087d7c85991c1ad3270370cf6"
Sep 18 17:52:53.799093 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.799077    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cac905f0d7e26eff0b121e5dac51d2dbbc047ff087d7c85991c1ad3270370cf6"} err="failed to get container status \"cac905f0d7e26eff0b121e5dac51d2dbbc047ff087d7c85991c1ad3270370cf6\": rpc error: code = NotFound desc = could not find container \"cac905f0d7e26eff0b121e5dac51d2dbbc047ff087d7c85991c1ad3270370cf6\": container with ID starting with cac905f0d7e26eff0b121e5dac51d2dbbc047ff087d7c85991c1ad3270370cf6 not found: ID does not exist"
Sep 18 17:52:53.799153 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.799094    2561 scope.go:117] "RemoveContainer" containerID="5ee666f651430a2827e3e851debe8643e8ac394b23660b97351850784d1698da"
Sep 18 17:52:53.799322 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.799305    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ee666f651430a2827e3e851debe8643e8ac394b23660b97351850784d1698da"} err="failed to get container status \"5ee666f651430a2827e3e851debe8643e8ac394b23660b97351850784d1698da\": rpc error: code = NotFound desc = could not find container \"5ee666f651430a2827e3e851debe8643e8ac394b23660b97351850784d1698da\": container with ID starting with 5ee666f651430a2827e3e851debe8643e8ac394b23660b97351850784d1698da not found: ID does not exist"
Sep 18 17:52:53.799358 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.799322    2561 scope.go:117] "RemoveContainer" containerID="11d68315b27586f2006e2848127343625491c24b02b71ca0dbe7049a5de8bc82"
Sep 18 17:52:53.799521 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.799506    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11d68315b27586f2006e2848127343625491c24b02b71ca0dbe7049a5de8bc82"} err="failed to get container status \"11d68315b27586f2006e2848127343625491c24b02b71ca0dbe7049a5de8bc82\": rpc error: code = NotFound desc = could not find container \"11d68315b27586f2006e2848127343625491c24b02b71ca0dbe7049a5de8bc82\": container with ID starting with 11d68315b27586f2006e2848127343625491c24b02b71ca0dbe7049a5de8bc82 not found: ID does not exist"
Sep 18 17:52:53.799555 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.799521    2561 scope.go:117] "RemoveContainer" containerID="4fa431a5325cc8da90d0a10f85db0a7e546046597b2bab50093fd69874855b04"
Sep 18 17:52:53.799750 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.799728    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fa431a5325cc8da90d0a10f85db0a7e546046597b2bab50093fd69874855b04"} err="failed to get container status \"4fa431a5325cc8da90d0a10f85db0a7e546046597b2bab50093fd69874855b04\": rpc error: code = NotFound desc = could not find container \"4fa431a5325cc8da90d0a10f85db0a7e546046597b2bab50093fd69874855b04\": container with ID starting with 4fa431a5325cc8da90d0a10f85db0a7e546046597b2bab50093fd69874855b04 not found: ID does not exist"
Sep 18 17:52:53.799786 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.799753    2561 scope.go:117] "RemoveContainer" containerID="cae77bd82a6938cf314fffa5dbeb4ed4f67024f1d5e28447c519697f39c9bb8c"
Sep 18 17:52:53.799965 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.799947    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cae77bd82a6938cf314fffa5dbeb4ed4f67024f1d5e28447c519697f39c9bb8c"} err="failed to get container status \"cae77bd82a6938cf314fffa5dbeb4ed4f67024f1d5e28447c519697f39c9bb8c\": rpc error: code = NotFound desc = could not find container \"cae77bd82a6938cf314fffa5dbeb4ed4f67024f1d5e28447c519697f39c9bb8c\": container with ID starting with cae77bd82a6938cf314fffa5dbeb4ed4f67024f1d5e28447c519697f39c9bb8c not found: ID does not exist"
Sep 18 17:52:53.800003 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.799967    2561 scope.go:117] "RemoveContainer" containerID="5d47cee9475e7b69a9c0e95a5d58e85210157a0a019820b878bb75a0f376afb7"
Sep 18 17:52:53.800169 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.800154    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d47cee9475e7b69a9c0e95a5d58e85210157a0a019820b878bb75a0f376afb7"} err="failed to get container status \"5d47cee9475e7b69a9c0e95a5d58e85210157a0a019820b878bb75a0f376afb7\": rpc error: code = NotFound desc = could not find container \"5d47cee9475e7b69a9c0e95a5d58e85210157a0a019820b878bb75a0f376afb7\": container with ID starting with 5d47cee9475e7b69a9c0e95a5d58e85210157a0a019820b878bb75a0f376afb7 not found: ID does not exist"
Sep 18 17:52:53.800169 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.800169    2561 scope.go:117] "RemoveContainer" containerID="6a70cb9a75cc5f915d386219b8c5ab68bd8ffd36a1a018530b26cc0a20c7ef62"
Sep 18 17:52:53.800374 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.800360    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a70cb9a75cc5f915d386219b8c5ab68bd8ffd36a1a018530b26cc0a20c7ef62"} err="failed to get container status \"6a70cb9a75cc5f915d386219b8c5ab68bd8ffd36a1a018530b26cc0a20c7ef62\": rpc error: code = NotFound desc = could not find container \"6a70cb9a75cc5f915d386219b8c5ab68bd8ffd36a1a018530b26cc0a20c7ef62\": container with ID starting with 6a70cb9a75cc5f915d386219b8c5ab68bd8ffd36a1a018530b26cc0a20c7ef62 not found: ID does not exist"
Sep 18 17:52:53.800412 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.800374    2561 scope.go:117] "RemoveContainer" containerID="cac905f0d7e26eff0b121e5dac51d2dbbc047ff087d7c85991c1ad3270370cf6"
Sep 18 17:52:53.800589 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.800573    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cac905f0d7e26eff0b121e5dac51d2dbbc047ff087d7c85991c1ad3270370cf6"} err="failed to get container status \"cac905f0d7e26eff0b121e5dac51d2dbbc047ff087d7c85991c1ad3270370cf6\": rpc error: code = NotFound desc = could not find container \"cac905f0d7e26eff0b121e5dac51d2dbbc047ff087d7c85991c1ad3270370cf6\": container with ID starting with cac905f0d7e26eff0b121e5dac51d2dbbc047ff087d7c85991c1ad3270370cf6 not found: ID does not exist"
Sep 18 17:52:53.800638 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.800588    2561 scope.go:117] "RemoveContainer" containerID="5ee666f651430a2827e3e851debe8643e8ac394b23660b97351850784d1698da"
Sep 18 17:52:53.800762 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.800747    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ee666f651430a2827e3e851debe8643e8ac394b23660b97351850784d1698da"} err="failed to get container status \"5ee666f651430a2827e3e851debe8643e8ac394b23660b97351850784d1698da\": rpc error: code = NotFound desc = could not find container \"5ee666f651430a2827e3e851debe8643e8ac394b23660b97351850784d1698da\": container with ID starting with 5ee666f651430a2827e3e851debe8643e8ac394b23660b97351850784d1698da not found: ID does not exist"
Sep 18 17:52:53.800807 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.800762    2561 scope.go:117] "RemoveContainer" containerID="11d68315b27586f2006e2848127343625491c24b02b71ca0dbe7049a5de8bc82"
Sep 18 17:52:53.800938 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.800923    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11d68315b27586f2006e2848127343625491c24b02b71ca0dbe7049a5de8bc82"} err="failed to get container status \"11d68315b27586f2006e2848127343625491c24b02b71ca0dbe7049a5de8bc82\": rpc error: code = NotFound desc = could not find container \"11d68315b27586f2006e2848127343625491c24b02b71ca0dbe7049a5de8bc82\": container with ID starting with 11d68315b27586f2006e2848127343625491c24b02b71ca0dbe7049a5de8bc82 not found: ID does not exist"
Sep 18 17:52:53.800979 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.800938    2561 scope.go:117] "RemoveContainer" containerID="4fa431a5325cc8da90d0a10f85db0a7e546046597b2bab50093fd69874855b04"
Sep 18 17:52:53.801110 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.801096    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fa431a5325cc8da90d0a10f85db0a7e546046597b2bab50093fd69874855b04"} err="failed to get container status \"4fa431a5325cc8da90d0a10f85db0a7e546046597b2bab50093fd69874855b04\": rpc error: code = NotFound desc = could not find container \"4fa431a5325cc8da90d0a10f85db0a7e546046597b2bab50093fd69874855b04\": container with ID starting with 4fa431a5325cc8da90d0a10f85db0a7e546046597b2bab50093fd69874855b04 not found: ID does not exist"
Sep 18 17:52:53.801171 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.801110    2561 scope.go:117] "RemoveContainer" containerID="cae77bd82a6938cf314fffa5dbeb4ed4f67024f1d5e28447c519697f39c9bb8c"
Sep 18 17:52:53.801330 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.801310    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cae77bd82a6938cf314fffa5dbeb4ed4f67024f1d5e28447c519697f39c9bb8c"} err="failed to get container status \"cae77bd82a6938cf314fffa5dbeb4ed4f67024f1d5e28447c519697f39c9bb8c\": rpc error: code = NotFound desc = could not find container \"cae77bd82a6938cf314fffa5dbeb4ed4f67024f1d5e28447c519697f39c9bb8c\": container with ID starting with cae77bd82a6938cf314fffa5dbeb4ed4f67024f1d5e28447c519697f39c9bb8c not found: ID does not exist"
Sep 18 17:52:53.801371 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.801331    2561 scope.go:117] "RemoveContainer" containerID="5d47cee9475e7b69a9c0e95a5d58e85210157a0a019820b878bb75a0f376afb7"
Sep 18 17:52:53.801557 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.801540    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d47cee9475e7b69a9c0e95a5d58e85210157a0a019820b878bb75a0f376afb7"} err="failed to get container status \"5d47cee9475e7b69a9c0e95a5d58e85210157a0a019820b878bb75a0f376afb7\": rpc error: code = NotFound desc = could not find container \"5d47cee9475e7b69a9c0e95a5d58e85210157a0a019820b878bb75a0f376afb7\": container with ID starting with 5d47cee9475e7b69a9c0e95a5d58e85210157a0a019820b878bb75a0f376afb7 not found: ID does not exist"
Sep 18 17:52:53.801600 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.801558    2561 scope.go:117] "RemoveContainer" containerID="6a70cb9a75cc5f915d386219b8c5ab68bd8ffd36a1a018530b26cc0a20c7ef62"
Sep 18 17:52:53.801736 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.801714    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a70cb9a75cc5f915d386219b8c5ab68bd8ffd36a1a018530b26cc0a20c7ef62"} err="failed to get container status \"6a70cb9a75cc5f915d386219b8c5ab68bd8ffd36a1a018530b26cc0a20c7ef62\": rpc error: code = NotFound desc = could not find container \"6a70cb9a75cc5f915d386219b8c5ab68bd8ffd36a1a018530b26cc0a20c7ef62\": container with ID starting with 6a70cb9a75cc5f915d386219b8c5ab68bd8ffd36a1a018530b26cc0a20c7ef62 not found: ID does not exist"
Sep 18 17:52:53.814094 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.814074    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 17:52:53.814430 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.814413    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1096ace1-2554-43bc-bf2e-5e6d09e1c48f" containerName="registry"
Sep 18 17:52:53.814478 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.814433    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1096ace1-2554-43bc-bf2e-5e6d09e1c48f" containerName="registry"
Sep 18 17:52:53.814478 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.814446    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02e201f9-5c67-4fc3-8822-8c39e516dddb" containerName="config-reloader"
Sep 18 17:52:53.814478 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.814451    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="02e201f9-5c67-4fc3-8822-8c39e516dddb" containerName="config-reloader"
Sep 18 17:52:53.814478 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.814459    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02e201f9-5c67-4fc3-8822-8c39e516dddb" containerName="kube-rbac-proxy-web"
Sep 18 17:52:53.814478 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.814464    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="02e201f9-5c67-4fc3-8822-8c39e516dddb" containerName="kube-rbac-proxy-web"
Sep 18 17:52:53.814478 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.814474    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24d70c78-d64e-4aa0-9d34-e1a23d5996af" containerName="console"
Sep 18 17:52:53.814478 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.814479    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="24d70c78-d64e-4aa0-9d34-e1a23d5996af" containerName="console"
Sep 18 17:52:53.814692 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.814485    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02e201f9-5c67-4fc3-8822-8c39e516dddb" containerName="prometheus"
Sep 18 17:52:53.814692 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.814490    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="02e201f9-5c67-4fc3-8822-8c39e516dddb" containerName="prometheus"
Sep 18 17:52:53.814692 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.814498    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02e201f9-5c67-4fc3-8822-8c39e516dddb" containerName="thanos-sidecar"
Sep 18 17:52:53.814692 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.814503    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="02e201f9-5c67-4fc3-8822-8c39e516dddb" containerName="thanos-sidecar"
Sep 18 17:52:53.814692 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.814512    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02e201f9-5c67-4fc3-8822-8c39e516dddb" containerName="init-config-reloader"
Sep 18 17:52:53.814692 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.814517    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="02e201f9-5c67-4fc3-8822-8c39e516dddb" containerName="init-config-reloader"
Sep 18 17:52:53.814692 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.814522    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02e201f9-5c67-4fc3-8822-8c39e516dddb" containerName="kube-rbac-proxy"
Sep 18 17:52:53.814692 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.814527    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="02e201f9-5c67-4fc3-8822-8c39e516dddb" containerName="kube-rbac-proxy"
Sep 18 17:52:53.814692 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.814539    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02e201f9-5c67-4fc3-8822-8c39e516dddb" containerName="kube-rbac-proxy-thanos"
Sep 18 17:52:53.814692 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.814544    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="02e201f9-5c67-4fc3-8822-8c39e516dddb" containerName="kube-rbac-proxy-thanos"
Sep 18 17:52:53.814692 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.814592    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="02e201f9-5c67-4fc3-8822-8c39e516dddb" containerName="config-reloader"
Sep 18 17:52:53.814692 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.814601    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="24d70c78-d64e-4aa0-9d34-e1a23d5996af" containerName="console"
Sep 18 17:52:53.814692 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.814606    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="02e201f9-5c67-4fc3-8822-8c39e516dddb" containerName="prometheus"
Sep 18 17:52:53.814692 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.814613    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="02e201f9-5c67-4fc3-8822-8c39e516dddb" containerName="kube-rbac-proxy-web"
Sep 18 17:52:53.814692 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.814618    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="02e201f9-5c67-4fc3-8822-8c39e516dddb" containerName="kube-rbac-proxy-thanos"
Sep 18 17:52:53.814692 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.814623    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="02e201f9-5c67-4fc3-8822-8c39e516dddb" containerName="thanos-sidecar"
Sep 18 17:52:53.814692 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.814628    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1096ace1-2554-43bc-bf2e-5e6d09e1c48f" containerName="registry"
Sep 18 17:52:53.814692 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.814633    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="02e201f9-5c67-4fc3-8822-8c39e516dddb" containerName="kube-rbac-proxy"
Sep 18 17:52:53.820357 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.820342    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:52:53.824045 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.824028    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-v2f4q\""
Sep 18 17:52:53.826447 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.826287    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 18 17:52:53.826447 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.826287    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 18 17:52:53.826447 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.826406    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 18 17:52:53.826650 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.826634    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 18 17:52:53.826851 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.826719    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 18 17:52:53.826851 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.826732    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 18 17:52:53.826851 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.826752    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 18 17:52:53.827093 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.827076    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 18 17:52:53.827231 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.827211    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 18 17:52:53.827330 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.827242    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 18 17:52:53.827330 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.827276    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-7fsdhaco8c34s\""
Sep 18 17:52:53.827330 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.827276    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 18 17:52:53.827488 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.827329    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 18 17:52:53.831412 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.831392    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8b31feaf-fe2d-4a3e-b37d-7c979d716cf0-config-out\") pod \"prometheus-k8s-0\" (UID: \"8b31feaf-fe2d-4a3e-b37d-7c979d716cf0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:52:53.831476 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.831426    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8b31feaf-fe2d-4a3e-b37d-7c979d716cf0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8b31feaf-fe2d-4a3e-b37d-7c979d716cf0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:52:53.831476 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.831450    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b31feaf-fe2d-4a3e-b37d-7c979d716cf0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8b31feaf-fe2d-4a3e-b37d-7c979d716cf0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:52:53.831476 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.831467    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8b31feaf-fe2d-4a3e-b37d-7c979d716cf0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8b31feaf-fe2d-4a3e-b37d-7c979d716cf0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:52:53.831580 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.831486    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b31feaf-fe2d-4a3e-b37d-7c979d716cf0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8b31feaf-fe2d-4a3e-b37d-7c979d716cf0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:52:53.831580 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.831508    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8b31feaf-fe2d-4a3e-b37d-7c979d716cf0-config\") pod \"prometheus-k8s-0\" (UID: \"8b31feaf-fe2d-4a3e-b37d-7c979d716cf0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:52:53.831580 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.831558    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8b31feaf-fe2d-4a3e-b37d-7c979d716cf0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8b31feaf-fe2d-4a3e-b37d-7c979d716cf0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:52:53.831686 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.831583    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8b31feaf-fe2d-4a3e-b37d-7c979d716cf0-web-config\") pod \"prometheus-k8s-0\" (UID: \"8b31feaf-fe2d-4a3e-b37d-7c979d716cf0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:52:53.831686 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.831617    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b31feaf-fe2d-4a3e-b37d-7c979d716cf0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8b31feaf-fe2d-4a3e-b37d-7c979d716cf0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:52:53.831686 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.831635    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8b31feaf-fe2d-4a3e-b37d-7c979d716cf0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8b31feaf-fe2d-4a3e-b37d-7c979d716cf0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:52:53.831686 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.831654    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8b31feaf-fe2d-4a3e-b37d-7c979d716cf0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8b31feaf-fe2d-4a3e-b37d-7c979d716cf0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:52:53.831686 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.831672    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clz67\" (UniqueName: \"kubernetes.io/projected/8b31feaf-fe2d-4a3e-b37d-7c979d716cf0-kube-api-access-clz67\") pod \"prometheus-k8s-0\" (UID: \"8b31feaf-fe2d-4a3e-b37d-7c979d716cf0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:52:53.831887 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.831686    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8b31feaf-fe2d-4a3e-b37d-7c979d716cf0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8b31feaf-fe2d-4a3e-b37d-7c979d716cf0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:52:53.831887 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.831702    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b31feaf-fe2d-4a3e-b37d-7c979d716cf0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8b31feaf-fe2d-4a3e-b37d-7c979d716cf0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:52:53.831887 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.831727    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8b31feaf-fe2d-4a3e-b37d-7c979d716cf0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8b31feaf-fe2d-4a3e-b37d-7c979d716cf0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:52:53.831887 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.831746    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8b31feaf-fe2d-4a3e-b37d-7c979d716cf0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8b31feaf-fe2d-4a3e-b37d-7c979d716cf0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:52:53.831887 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.831778    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8b31feaf-fe2d-4a3e-b37d-7c979d716cf0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8b31feaf-fe2d-4a3e-b37d-7c979d716cf0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:52:53.831887 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.831800    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8b31feaf-fe2d-4a3e-b37d-7c979d716cf0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8b31feaf-fe2d-4a3e-b37d-7c979d716cf0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:52:53.831887 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.831824    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8b31feaf-fe2d-4a3e-b37d-7c979d716cf0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8b31feaf-fe2d-4a3e-b37d-7c979d716cf0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:52:53.831887 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.831839    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8b31feaf-fe2d-4a3e-b37d-7c979d716cf0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8b31feaf-fe2d-4a3e-b37d-7c979d716cf0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:52:53.832698 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.832678    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 18 17:52:53.835607 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.835590    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 18 17:52:53.860393 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.860372    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02e201f9-5c67-4fc3-8822-8c39e516dddb" path="/var/lib/kubelet/pods/02e201f9-5c67-4fc3-8822-8c39e516dddb/volumes"
Sep 18 17:52:53.866241 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.866222    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 17:52:53.933077 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.933054    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b31feaf-fe2d-4a3e-b37d-7c979d716cf0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8b31feaf-fe2d-4a3e-b37d-7c979d716cf0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:52:53.933239 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.933082    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8b31feaf-fe2d-4a3e-b37d-7c979d716cf0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8b31feaf-fe2d-4a3e-b37d-7c979d716cf0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:52:53.933239 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.933102    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8b31feaf-fe2d-4a3e-b37d-7c979d716cf0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8b31feaf-fe2d-4a3e-b37d-7c979d716cf0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:52:53.933239 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.933131    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clz67\" (UniqueName: \"kubernetes.io/projected/8b31feaf-fe2d-4a3e-b37d-7c979d716cf0-kube-api-access-clz67\") pod \"prometheus-k8s-0\" (UID: \"8b31feaf-fe2d-4a3e-b37d-7c979d716cf0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:52:53.933408 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.933330    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8b31feaf-fe2d-4a3e-b37d-7c979d716cf0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8b31feaf-fe2d-4a3e-b37d-7c979d716cf0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:52:53.933408 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.933376    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b31feaf-fe2d-4a3e-b37d-7c979d716cf0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8b31feaf-fe2d-4a3e-b37d-7c979d716cf0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:52:53.933516 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.933412    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8b31feaf-fe2d-4a3e-b37d-7c979d716cf0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8b31feaf-fe2d-4a3e-b37d-7c979d716cf0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:52:53.933516 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.933440    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8b31feaf-fe2d-4a3e-b37d-7c979d716cf0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8b31feaf-fe2d-4a3e-b37d-7c979d716cf0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:52:53.933760 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.933739    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8b31feaf-fe2d-4a3e-b37d-7c979d716cf0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8b31feaf-fe2d-4a3e-b37d-7c979d716cf0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:52:53.933860 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.933781    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8b31feaf-fe2d-4a3e-b37d-7c979d716cf0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8b31feaf-fe2d-4a3e-b37d-7c979d716cf0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:52:53.933860 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.933814    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8b31feaf-fe2d-4a3e-b37d-7c979d716cf0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8b31feaf-fe2d-4a3e-b37d-7c979d716cf0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:52:53.933860 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.933838    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8b31feaf-fe2d-4a3e-b37d-7c979d716cf0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8b31feaf-fe2d-4a3e-b37d-7c979d716cf0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:52:53.934027 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.933889    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8b31feaf-fe2d-4a3e-b37d-7c979d716cf0-config-out\") pod \"prometheus-k8s-0\" (UID: \"8b31feaf-fe2d-4a3e-b37d-7c979d716cf0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:52:53.934027 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.933937    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8b31feaf-fe2d-4a3e-b37d-7c979d716cf0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8b31feaf-fe2d-4a3e-b37d-7c979d716cf0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:52:53.934027 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.933963    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b31feaf-fe2d-4a3e-b37d-7c979d716cf0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8b31feaf-fe2d-4a3e-b37d-7c979d716cf0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:52:53.934027 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.933975    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8b31feaf-fe2d-4a3e-b37d-7c979d716cf0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8b31feaf-fe2d-4a3e-b37d-7c979d716cf0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:52:53.934027 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.933988    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8b31feaf-fe2d-4a3e-b37d-7c979d716cf0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8b31feaf-fe2d-4a3e-b37d-7c979d716cf0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:52:53.934307 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.934045    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b31feaf-fe2d-4a3e-b37d-7c979d716cf0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8b31feaf-fe2d-4a3e-b37d-7c979d716cf0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:52:53.934307 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.934079    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8b31feaf-fe2d-4a3e-b37d-7c979d716cf0-config\") pod \"prometheus-k8s-0\" (UID: \"8b31feaf-fe2d-4a3e-b37d-7c979d716cf0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:52:53.934307 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.934112    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8b31feaf-fe2d-4a3e-b37d-7c979d716cf0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8b31feaf-fe2d-4a3e-b37d-7c979d716cf0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:52:53.934307 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.934165    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8b31feaf-fe2d-4a3e-b37d-7c979d716cf0-web-config\") pod \"prometheus-k8s-0\" (UID: \"8b31feaf-fe2d-4a3e-b37d-7c979d716cf0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:52:53.934496 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.934313    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8b31feaf-fe2d-4a3e-b37d-7c979d716cf0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8b31feaf-fe2d-4a3e-b37d-7c979d716cf0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:52:53.934857 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.934831    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b31feaf-fe2d-4a3e-b37d-7c979d716cf0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8b31feaf-fe2d-4a3e-b37d-7c979d716cf0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:52:53.934986 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.934960    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b31feaf-fe2d-4a3e-b37d-7c979d716cf0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8b31feaf-fe2d-4a3e-b37d-7c979d716cf0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:52:53.935309 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.935282    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8b31feaf-fe2d-4a3e-b37d-7c979d716cf0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8b31feaf-fe2d-4a3e-b37d-7c979d716cf0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:52:53.936163 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.936139    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8b31feaf-fe2d-4a3e-b37d-7c979d716cf0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8b31feaf-fe2d-4a3e-b37d-7c979d716cf0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:52:53.936276 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.936240    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8b31feaf-fe2d-4a3e-b37d-7c979d716cf0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8b31feaf-fe2d-4a3e-b37d-7c979d716cf0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:52:53.936391 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.934049    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b31feaf-fe2d-4a3e-b37d-7c979d716cf0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8b31feaf-fe2d-4a3e-b37d-7c979d716cf0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:52:53.937222 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.937197    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8b31feaf-fe2d-4a3e-b37d-7c979d716cf0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8b31feaf-fe2d-4a3e-b37d-7c979d716cf0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:52:53.937319 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.937292    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b31feaf-fe2d-4a3e-b37d-7c979d716cf0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8b31feaf-fe2d-4a3e-b37d-7c979d716cf0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:52:53.938025 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.937653    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8b31feaf-fe2d-4a3e-b37d-7c979d716cf0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8b31feaf-fe2d-4a3e-b37d-7c979d716cf0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:52:53.938025 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.937956    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8b31feaf-fe2d-4a3e-b37d-7c979d716cf0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8b31feaf-fe2d-4a3e-b37d-7c979d716cf0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:52:53.938678 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.938652    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8b31feaf-fe2d-4a3e-b37d-7c979d716cf0-web-config\") pod \"prometheus-k8s-0\" (UID: \"8b31feaf-fe2d-4a3e-b37d-7c979d716cf0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:52:53.938888 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.938868    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8b31feaf-fe2d-4a3e-b37d-7c979d716cf0-config-out\") pod \"prometheus-k8s-0\" (UID: \"8b31feaf-fe2d-4a3e-b37d-7c979d716cf0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:52:53.939033 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.939011    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8b31feaf-fe2d-4a3e-b37d-7c979d716cf0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8b31feaf-fe2d-4a3e-b37d-7c979d716cf0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:52:53.939293 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.939272    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8b31feaf-fe2d-4a3e-b37d-7c979d716cf0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8b31feaf-fe2d-4a3e-b37d-7c979d716cf0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:52:53.939487 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.939469    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8b31feaf-fe2d-4a3e-b37d-7c979d716cf0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8b31feaf-fe2d-4a3e-b37d-7c979d716cf0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:52:53.939644 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.939623    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/8b31feaf-fe2d-4a3e-b37d-7c979d716cf0-config\") pod \"prometheus-k8s-0\" (UID: \"8b31feaf-fe2d-4a3e-b37d-7c979d716cf0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:52:53.939721 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.939705    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8b31feaf-fe2d-4a3e-b37d-7c979d716cf0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8b31feaf-fe2d-4a3e-b37d-7c979d716cf0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:52:53.943101 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:53.943075    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clz67\" (UniqueName: \"kubernetes.io/projected/8b31feaf-fe2d-4a3e-b37d-7c979d716cf0-kube-api-access-clz67\") pod \"prometheus-k8s-0\" (UID: \"8b31feaf-fe2d-4a3e-b37d-7c979d716cf0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:52:54.130052 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:54.129963    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:52:54.421687 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:54.421653    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 17:52:54.425523 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:52:54.425493    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b31feaf_fe2d_4a3e_b37d_7c979d716cf0.slice/crio-1384d6dc7196de25ccfbe255b871155cc42fc629d0b219048bf56cd08754e8b7 WatchSource:0}: Error finding container 1384d6dc7196de25ccfbe255b871155cc42fc629d0b219048bf56cd08754e8b7: Status 404 returned error can't find the container with id 1384d6dc7196de25ccfbe255b871155cc42fc629d0b219048bf56cd08754e8b7
Sep 18 17:52:54.735601 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:54.735515    2561 generic.go:356] "Generic (PLEG): container finished" podID="8b31feaf-fe2d-4a3e-b37d-7c979d716cf0" containerID="fb30773569ae051dec592108681cff12c5d576130ee5ee01cfe41960717433ed" exitCode=0
Sep 18 17:52:54.735601 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:54.735584    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8b31feaf-fe2d-4a3e-b37d-7c979d716cf0","Type":"ContainerDied","Data":"fb30773569ae051dec592108681cff12c5d576130ee5ee01cfe41960717433ed"}
Sep 18 17:52:54.735778 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:54.735654    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8b31feaf-fe2d-4a3e-b37d-7c979d716cf0","Type":"ContainerStarted","Data":"1384d6dc7196de25ccfbe255b871155cc42fc629d0b219048bf56cd08754e8b7"}
Sep 18 17:52:55.743324 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:55.743293    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8b31feaf-fe2d-4a3e-b37d-7c979d716cf0","Type":"ContainerStarted","Data":"58ccd1e41ec50deeacf9b85986fed016f4ae1e0b97b48b3427d302ed017128ab"}
Sep 18 17:52:55.743603 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:55.743409    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8b31feaf-fe2d-4a3e-b37d-7c979d716cf0","Type":"ContainerStarted","Data":"790a50dee8541937d35bd691118c421bf0126ba7800731879032a1582e860971"}
Sep 18 17:52:55.743603 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:55.743455    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8b31feaf-fe2d-4a3e-b37d-7c979d716cf0","Type":"ContainerStarted","Data":"562d8139a28ea466a49170410e08865442433750d435acb82987e01a57b4acb5"}
Sep 18 17:52:55.743603 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:55.743499    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8b31feaf-fe2d-4a3e-b37d-7c979d716cf0","Type":"ContainerStarted","Data":"04c20f82cb20ee3df706135cc08d10bea4fbb0ac5ac77283e7e98c1c8b20e2bf"}
Sep 18 17:52:55.743603 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:55.743536    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8b31feaf-fe2d-4a3e-b37d-7c979d716cf0","Type":"ContainerStarted","Data":"7fec0d6a3e8b0fa48b851e39a16737eb5080dceb048e9a42ccf99eb6d5329f9e"}
Sep 18 17:52:55.743603 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:55.743575    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8b31feaf-fe2d-4a3e-b37d-7c979d716cf0","Type":"ContainerStarted","Data":"1e8637395e37c07986d4692602b419f8e7cb236cc603deed375b5a2dca28278c"}
Sep 18 17:52:55.787065 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:55.787016    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.768246342 podStartE2EDuration="2.786994851s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.748509ms" isStatefulPod=false podCreationTimestamp="2026-09-18 17:52:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:52:55.784661546 +0000 UTC m=+180.411793157" watchObservedRunningTime="2026-09-18 17:52:55.786994851 +0000 UTC m=+180.414126472"
Sep 18 17:52:56.447973 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:56.447936    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:52:56.574141 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:56.574083    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:52:59.130339 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:52:59.130300    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:53:07.333259 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:53:07.333225    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:53:07.443377 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:53:07.443347    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:53:07.545103 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:53:07.545065    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:53:18.956715 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:53:18.956680    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:53:21.355884 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:53:21.355848    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:53:21.457343 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:53:21.457310    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:53:37.757830 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:53:37.757799    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:53:54.131212 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:53:54.131178    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:53:54.146535 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:53:54.146504    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:53:54.998989 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:53:54.998963    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 17:54:02.530294 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:54:02.530257    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:54:02.601347 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:54:02.601309    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:54:03.055503 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:54:03.055464    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:54:03.156010 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:54:03.155979    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:54:12.647256 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:54:12.647226    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:54:13.712172 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:54:13.712141    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:54:13.822730 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:54:13.822699    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:54:14.634762 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:54:14.634727    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:54:14.747036 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:54:14.746997    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:54:17.514947 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:54:17.514875    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:54:17.682587 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:54:17.682552    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:54:55.724295 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:54:55.724264    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-5kjlz_75f2a1f7-95a7-462e-9ea7-23c51dd29599/console-operator/1.log"
Sep 18 17:54:55.737496 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:54:55.737468    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkcgk_c5d026b7-af6b-4899-85f4-6554c0f7cd52/ovn-acl-logging/0.log"
Sep 18 17:54:55.738382 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:54:55.738360    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-5kjlz_75f2a1f7-95a7-462e-9ea7-23c51dd29599/console-operator/1.log"
Sep 18 17:54:55.741703 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:54:55.741686    2561 kubelet.go:1648] "Image garbage collection succeeded"
Sep 18 17:54:55.745776 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:54:55.745760    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkcgk_c5d026b7-af6b-4899-85f4-6554c0f7cd52/ovn-acl-logging/0.log"
Sep 18 17:55:08.530104 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:55:08.530070    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:55:10.147787 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:55:10.147757    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:55:31.890741 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:55:31.890703    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:55:32.064363 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:55:32.064334    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:55:36.245409 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:55:36.245373    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:55:36.401400 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:55:36.401372    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:55:37.576998 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:55:37.576960    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:55:37.777580 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:55:37.777551    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:55:40.170794 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:55:40.170761    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:55:40.535521 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:55:40.535444    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:55:41.571176 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:55:41.571140    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:55:41.644171 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:55:41.644141    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:55:47.502642 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:55:47.502601    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:55:48.524477 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:55:48.524403    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:55:48.623060 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:55:48.623031    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:55:56.534786 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:55:56.534753    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:55:56.636763 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:55:56.636734    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:55:58.686432 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:55:58.686400    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:56:07.566698 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:07.566643    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:56:07.645206 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:07.645176    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:56:13.266909 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:13.266874    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:56:13.459477 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:13.459442    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:56:13.629454 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:13.629372    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:56:14.257073 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:14.257040    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:56:14.386783 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:14.386750    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:56:14.439839 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:14.439799    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-649dc7b766-qqtwg"]
Sep 18 17:56:14.443501 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:14.443480    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-649dc7b766-qqtwg"
Sep 18 17:56:14.446654 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:14.446635    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 18 17:56:14.447242 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:14.447226    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 18 17:56:14.447329 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:14.447288    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-6jc6c\""
Sep 18 17:56:14.447962 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:14.447949    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 18 17:56:14.448030 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:14.447951    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 18 17:56:14.476205 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:14.476175    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-649dc7b766-qqtwg"]
Sep 18 17:56:14.607513 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:14.607437    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:56:14.622706 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:14.622671    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/89b6292a-2396-4d3e-88bb-f645a5db8832-tmp\") pod \"opendatahub-operator-controller-manager-649dc7b766-qqtwg\" (UID: \"89b6292a-2396-4d3e-88bb-f645a5db8832\") " pod="opendatahub/opendatahub-operator-controller-manager-649dc7b766-qqtwg"
Sep 18 17:56:14.622876 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:14.622715    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/89b6292a-2396-4d3e-88bb-f645a5db8832-webhook-cert\") pod \"opendatahub-operator-controller-manager-649dc7b766-qqtwg\" (UID: \"89b6292a-2396-4d3e-88bb-f645a5db8832\") " pod="opendatahub/opendatahub-operator-controller-manager-649dc7b766-qqtwg"
Sep 18 17:56:14.622876 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:14.622770    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/89b6292a-2396-4d3e-88bb-f645a5db8832-apiservice-cert\") pod \"opendatahub-operator-controller-manager-649dc7b766-qqtwg\" (UID: \"89b6292a-2396-4d3e-88bb-f645a5db8832\") " pod="opendatahub/opendatahub-operator-controller-manager-649dc7b766-qqtwg"
Sep 18 17:56:14.622876 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:14.622826    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/89b6292a-2396-4d3e-88bb-f645a5db8832-manifests\") pod \"opendatahub-operator-controller-manager-649dc7b766-qqtwg\" (UID: \"89b6292a-2396-4d3e-88bb-f645a5db8832\") " pod="opendatahub/opendatahub-operator-controller-manager-649dc7b766-qqtwg"
Sep 18 17:56:14.622985 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:14.622886    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4v4s\" (UniqueName: \"kubernetes.io/projected/89b6292a-2396-4d3e-88bb-f645a5db8832-kube-api-access-v4v4s\") pod \"opendatahub-operator-controller-manager-649dc7b766-qqtwg\" (UID: \"89b6292a-2396-4d3e-88bb-f645a5db8832\") " pod="opendatahub/opendatahub-operator-controller-manager-649dc7b766-qqtwg"
Sep 18 17:56:14.703728 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:14.703694    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:56:14.723909 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:14.723870    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4v4s\" (UniqueName: \"kubernetes.io/projected/89b6292a-2396-4d3e-88bb-f645a5db8832-kube-api-access-v4v4s\") pod \"opendatahub-operator-controller-manager-649dc7b766-qqtwg\" (UID: \"89b6292a-2396-4d3e-88bb-f645a5db8832\") " pod="opendatahub/opendatahub-operator-controller-manager-649dc7b766-qqtwg"
Sep 18 17:56:14.724085 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:14.723914    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/89b6292a-2396-4d3e-88bb-f645a5db8832-tmp\") pod \"opendatahub-operator-controller-manager-649dc7b766-qqtwg\" (UID: \"89b6292a-2396-4d3e-88bb-f645a5db8832\") " pod="opendatahub/opendatahub-operator-controller-manager-649dc7b766-qqtwg"
Sep 18 17:56:14.724085 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:14.723941    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/89b6292a-2396-4d3e-88bb-f645a5db8832-webhook-cert\") pod \"opendatahub-operator-controller-manager-649dc7b766-qqtwg\" (UID: \"89b6292a-2396-4d3e-88bb-f645a5db8832\") " pod="opendatahub/opendatahub-operator-controller-manager-649dc7b766-qqtwg"
Sep 18 17:56:14.724085 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:14.723977    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/89b6292a-2396-4d3e-88bb-f645a5db8832-apiservice-cert\") pod \"opendatahub-operator-controller-manager-649dc7b766-qqtwg\" (UID: \"89b6292a-2396-4d3e-88bb-f645a5db8832\") " pod="opendatahub/opendatahub-operator-controller-manager-649dc7b766-qqtwg"
Sep 18 17:56:14.724281 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:14.724130    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/89b6292a-2396-4d3e-88bb-f645a5db8832-manifests\") pod \"opendatahub-operator-controller-manager-649dc7b766-qqtwg\" (UID: \"89b6292a-2396-4d3e-88bb-f645a5db8832\") " pod="opendatahub/opendatahub-operator-controller-manager-649dc7b766-qqtwg"
Sep 18 17:56:14.724419 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:14.724395    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/89b6292a-2396-4d3e-88bb-f645a5db8832-manifests\") pod \"opendatahub-operator-controller-manager-649dc7b766-qqtwg\" (UID: \"89b6292a-2396-4d3e-88bb-f645a5db8832\") " pod="opendatahub/opendatahub-operator-controller-manager-649dc7b766-qqtwg"
Sep 18 17:56:14.726339 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:14.726306    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/89b6292a-2396-4d3e-88bb-f645a5db8832-tmp\") pod \"opendatahub-operator-controller-manager-649dc7b766-qqtwg\" (UID: \"89b6292a-2396-4d3e-88bb-f645a5db8832\") " pod="opendatahub/opendatahub-operator-controller-manager-649dc7b766-qqtwg"
Sep 18 17:56:14.726479 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:14.726460    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/89b6292a-2396-4d3e-88bb-f645a5db8832-webhook-cert\") pod \"opendatahub-operator-controller-manager-649dc7b766-qqtwg\" (UID: \"89b6292a-2396-4d3e-88bb-f645a5db8832\") " pod="opendatahub/opendatahub-operator-controller-manager-649dc7b766-qqtwg"
Sep 18 17:56:14.726542 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:14.726507    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/89b6292a-2396-4d3e-88bb-f645a5db8832-apiservice-cert\") pod \"opendatahub-operator-controller-manager-649dc7b766-qqtwg\" (UID: \"89b6292a-2396-4d3e-88bb-f645a5db8832\") " pod="opendatahub/opendatahub-operator-controller-manager-649dc7b766-qqtwg"
Sep 18 17:56:14.735612 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:14.735587    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4v4s\" (UniqueName: \"kubernetes.io/projected/89b6292a-2396-4d3e-88bb-f645a5db8832-kube-api-access-v4v4s\") pod \"opendatahub-operator-controller-manager-649dc7b766-qqtwg\" (UID: \"89b6292a-2396-4d3e-88bb-f645a5db8832\") " pod="opendatahub/opendatahub-operator-controller-manager-649dc7b766-qqtwg"
Sep 18 17:56:14.753358 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:14.753330    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-649dc7b766-qqtwg"
Sep 18 17:56:14.836943 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:14.836911    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:56:15.022043 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:15.022010    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:56:15.049173 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:15.049143    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-649dc7b766-qqtwg"]
Sep 18 17:56:15.051731 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:56:15.051702    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89b6292a_2396_4d3e_88bb_f645a5db8832.slice/crio-80179300031c89a38b3770bd0e4f67c1b73c6a5857dfae50a86b0b885382399c WatchSource:0}: Error finding container 80179300031c89a38b3770bd0e4f67c1b73c6a5857dfae50a86b0b885382399c: Status 404 returned error can't find the container with id 80179300031c89a38b3770bd0e4f67c1b73c6a5857dfae50a86b0b885382399c
Sep 18 17:56:15.053525 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:15.053510    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 17:56:15.177238 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:15.177208    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:56:15.207173 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:15.207065    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-786dfb84b9-bgs76"]
Sep 18 17:56:15.211741 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:15.211718    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-786dfb84b9-bgs76"
Sep 18 17:56:15.215831 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:15.215809    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 18 17:56:15.216062 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:15.215992    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-jh2nx\""
Sep 18 17:56:15.216236 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:15.216082    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 18 17:56:15.216236 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:15.216094    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 18 17:56:15.216614 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:15.216599    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 18 17:56:15.216736 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:15.216634    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 18 17:56:15.243110 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:15.243078    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-786dfb84b9-bgs76"]
Sep 18 17:56:15.330038 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:15.329956    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw7vb\" (UniqueName: \"kubernetes.io/projected/ab192091-eb80-4dd9-83bc-54ea383c7c09-kube-api-access-hw7vb\") pod \"lws-controller-manager-786dfb84b9-bgs76\" (UID: \"ab192091-eb80-4dd9-83bc-54ea383c7c09\") " pod="openshift-lws-operator/lws-controller-manager-786dfb84b9-bgs76"
Sep 18 17:56:15.330038 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:15.329994    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ab192091-eb80-4dd9-83bc-54ea383c7c09-manager-config\") pod \"lws-controller-manager-786dfb84b9-bgs76\" (UID: \"ab192091-eb80-4dd9-83bc-54ea383c7c09\") " pod="openshift-lws-operator/lws-controller-manager-786dfb84b9-bgs76"
Sep 18 17:56:15.330256 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:15.330156    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ab192091-eb80-4dd9-83bc-54ea383c7c09-cert\") pod \"lws-controller-manager-786dfb84b9-bgs76\" (UID: \"ab192091-eb80-4dd9-83bc-54ea383c7c09\") " pod="openshift-lws-operator/lws-controller-manager-786dfb84b9-bgs76"
Sep 18 17:56:15.330256 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:15.330198    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ab192091-eb80-4dd9-83bc-54ea383c7c09-metrics-cert\") pod \"lws-controller-manager-786dfb84b9-bgs76\" (UID: \"ab192091-eb80-4dd9-83bc-54ea383c7c09\") " pod="openshift-lws-operator/lws-controller-manager-786dfb84b9-bgs76"
Sep 18 17:56:15.431399 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:15.431364    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ab192091-eb80-4dd9-83bc-54ea383c7c09-cert\") pod \"lws-controller-manager-786dfb84b9-bgs76\" (UID: \"ab192091-eb80-4dd9-83bc-54ea383c7c09\") " pod="openshift-lws-operator/lws-controller-manager-786dfb84b9-bgs76"
Sep 18 17:56:15.431879 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:15.431417    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ab192091-eb80-4dd9-83bc-54ea383c7c09-metrics-cert\") pod \"lws-controller-manager-786dfb84b9-bgs76\" (UID: \"ab192091-eb80-4dd9-83bc-54ea383c7c09\") " pod="openshift-lws-operator/lws-controller-manager-786dfb84b9-bgs76"
Sep 18 17:56:15.431879 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:15.431460    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw7vb\" (UniqueName: \"kubernetes.io/projected/ab192091-eb80-4dd9-83bc-54ea383c7c09-kube-api-access-hw7vb\") pod \"lws-controller-manager-786dfb84b9-bgs76\" (UID: \"ab192091-eb80-4dd9-83bc-54ea383c7c09\") " pod="openshift-lws-operator/lws-controller-manager-786dfb84b9-bgs76"
Sep 18 17:56:15.431879 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:15.431481    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ab192091-eb80-4dd9-83bc-54ea383c7c09-manager-config\") pod \"lws-controller-manager-786dfb84b9-bgs76\" (UID: \"ab192091-eb80-4dd9-83bc-54ea383c7c09\") " pod="openshift-lws-operator/lws-controller-manager-786dfb84b9-bgs76"
Sep 18 17:56:15.432461 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:15.432436    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ab192091-eb80-4dd9-83bc-54ea383c7c09-manager-config\") pod \"lws-controller-manager-786dfb84b9-bgs76\" (UID: \"ab192091-eb80-4dd9-83bc-54ea383c7c09\") " pod="openshift-lws-operator/lws-controller-manager-786dfb84b9-bgs76"
Sep 18 17:56:15.434048 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:15.434029    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ab192091-eb80-4dd9-83bc-54ea383c7c09-cert\") pod \"lws-controller-manager-786dfb84b9-bgs76\" (UID: \"ab192091-eb80-4dd9-83bc-54ea383c7c09\") " pod="openshift-lws-operator/lws-controller-manager-786dfb84b9-bgs76"
Sep 18 17:56:15.434161 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:15.434143    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ab192091-eb80-4dd9-83bc-54ea383c7c09-metrics-cert\") pod \"lws-controller-manager-786dfb84b9-bgs76\" (UID: \"ab192091-eb80-4dd9-83bc-54ea383c7c09\") " pod="openshift-lws-operator/lws-controller-manager-786dfb84b9-bgs76"
Sep 18 17:56:15.444435 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:15.444404    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw7vb\" (UniqueName: \"kubernetes.io/projected/ab192091-eb80-4dd9-83bc-54ea383c7c09-kube-api-access-hw7vb\") pod \"lws-controller-manager-786dfb84b9-bgs76\" (UID: \"ab192091-eb80-4dd9-83bc-54ea383c7c09\") " pod="openshift-lws-operator/lws-controller-manager-786dfb84b9-bgs76"
Sep 18 17:56:15.523309 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:15.523270    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-786dfb84b9-bgs76"
Sep 18 17:56:15.550845 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:15.550805    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-649dc7b766-qqtwg" event={"ID":"89b6292a-2396-4d3e-88bb-f645a5db8832","Type":"ContainerStarted","Data":"80179300031c89a38b3770bd0e4f67c1b73c6a5857dfae50a86b0b885382399c"}
Sep 18 17:56:15.616063 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:15.615979    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:56:15.847323 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:15.847292    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-786dfb84b9-bgs76"]
Sep 18 17:56:15.850270 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:56:15.850242    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab192091_eb80_4dd9_83bc_54ea383c7c09.slice/crio-3fe7714d1f37ea18e0824ea0d6ecfc906eabb421c59342c4144d127d6d02d156 WatchSource:0}: Error finding container 3fe7714d1f37ea18e0824ea0d6ecfc906eabb421c59342c4144d127d6d02d156: Status 404 returned error can't find the container with id 3fe7714d1f37ea18e0824ea0d6ecfc906eabb421c59342c4144d127d6d02d156
Sep 18 17:56:15.966209 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:15.966179    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:56:16.558061 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:16.558022    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-786dfb84b9-bgs76" event={"ID":"ab192091-eb80-4dd9-83bc-54ea383c7c09","Type":"ContainerStarted","Data":"3fe7714d1f37ea18e0824ea0d6ecfc906eabb421c59342c4144d127d6d02d156"}
Sep 18 17:56:19.575904 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:19.575813    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-649dc7b766-qqtwg" event={"ID":"89b6292a-2396-4d3e-88bb-f645a5db8832","Type":"ContainerStarted","Data":"aa80c19e5df3e3e744a971c85e98ad6e56c8e1d7e1604ec809bf531fd64dc393"}
Sep 18 17:56:20.581516 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:20.581478    2561 generic.go:356] "Generic (PLEG): container finished" podID="89b6292a-2396-4d3e-88bb-f645a5db8832" containerID="aa80c19e5df3e3e744a971c85e98ad6e56c8e1d7e1604ec809bf531fd64dc393" exitCode=0
Sep 18 17:56:20.581915 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:20.581565    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-649dc7b766-qqtwg" event={"ID":"89b6292a-2396-4d3e-88bb-f645a5db8832","Type":"ContainerDied","Data":"aa80c19e5df3e3e744a971c85e98ad6e56c8e1d7e1604ec809bf531fd64dc393"}
Sep 18 17:56:21.588195 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:21.588160    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-649dc7b766-qqtwg" event={"ID":"89b6292a-2396-4d3e-88bb-f645a5db8832","Type":"ContainerStarted","Data":"60a375653ddde9cdd2dae43f8ecc53ee5cc6e9fc49ea7a89644134d589b1797d"}
Sep 18 17:56:21.588662 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:21.588248    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-649dc7b766-qqtwg"
Sep 18 17:56:21.686059 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:21.686001    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-649dc7b766-qqtwg" podStartSLOduration=2.122665521 podStartE2EDuration="7.68598387s" totalImagesPullingTime="4.476022299s" totalInitContainerRuntime="1.08729605s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:56:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:56:21.68229765 +0000 UTC m=+386.309429281" watchObservedRunningTime="2026-09-18 17:56:21.68598387 +0000 UTC m=+386.313115490"
Sep 18 17:56:26.085901 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:26.085870    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:56:26.189042 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:26.189011    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:56:32.595435 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:32.595402    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-649dc7b766-qqtwg"
Sep 18 17:56:32.718339 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:32.718306    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:56:32.824879 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:32.824845    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:56:34.633422 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:34.633391    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:56:34.881859 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:34.881773    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:56:35.478400 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:35.478365    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:56:35.518543 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:35.518511    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6b7b7f7d54-gwfj9"]
Sep 18 17:56:35.530395 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:35.530370    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6b7b7f7d54-gwfj9"
Sep 18 17:56:35.534188 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:35.534169    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 18 17:56:35.534420 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:35.534399    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 18 17:56:35.535079 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:35.535059    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-dq6w2\""
Sep 18 17:56:35.537224 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:35.537196    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6b7b7f7d54-gwfj9"]
Sep 18 17:56:35.596519 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:35.596486    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/626cdc85-2717-4abb-82e8-77da8fb69033-tls-certs\") pod \"kube-auth-proxy-6b7b7f7d54-gwfj9\" (UID: \"626cdc85-2717-4abb-82e8-77da8fb69033\") " pod="openshift-ingress/kube-auth-proxy-6b7b7f7d54-gwfj9"
Sep 18 17:56:35.596519 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:35.596519    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/626cdc85-2717-4abb-82e8-77da8fb69033-tmp\") pod \"kube-auth-proxy-6b7b7f7d54-gwfj9\" (UID: \"626cdc85-2717-4abb-82e8-77da8fb69033\") " pod="openshift-ingress/kube-auth-proxy-6b7b7f7d54-gwfj9"
Sep 18 17:56:35.596705 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:35.596569    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pprqm\" (UniqueName: \"kubernetes.io/projected/626cdc85-2717-4abb-82e8-77da8fb69033-kube-api-access-pprqm\") pod \"kube-auth-proxy-6b7b7f7d54-gwfj9\" (UID: \"626cdc85-2717-4abb-82e8-77da8fb69033\") " pod="openshift-ingress/kube-auth-proxy-6b7b7f7d54-gwfj9"
Sep 18 17:56:35.697428 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:35.697393    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/626cdc85-2717-4abb-82e8-77da8fb69033-tls-certs\") pod \"kube-auth-proxy-6b7b7f7d54-gwfj9\" (UID: \"626cdc85-2717-4abb-82e8-77da8fb69033\") " pod="openshift-ingress/kube-auth-proxy-6b7b7f7d54-gwfj9"
Sep 18 17:56:35.697428 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:35.697428    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/626cdc85-2717-4abb-82e8-77da8fb69033-tmp\") pod \"kube-auth-proxy-6b7b7f7d54-gwfj9\" (UID: \"626cdc85-2717-4abb-82e8-77da8fb69033\") " pod="openshift-ingress/kube-auth-proxy-6b7b7f7d54-gwfj9"
Sep 18 17:56:35.697829 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:35.697470    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pprqm\" (UniqueName: \"kubernetes.io/projected/626cdc85-2717-4abb-82e8-77da8fb69033-kube-api-access-pprqm\") pod \"kube-auth-proxy-6b7b7f7d54-gwfj9\" (UID: \"626cdc85-2717-4abb-82e8-77da8fb69033\") " pod="openshift-ingress/kube-auth-proxy-6b7b7f7d54-gwfj9"
Sep 18 17:56:35.697829 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:56:35.697532    2561 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 18 17:56:35.697829 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:56:35.697590    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/626cdc85-2717-4abb-82e8-77da8fb69033-tls-certs podName:626cdc85-2717-4abb-82e8-77da8fb69033 nodeName:}" failed. No retries permitted until 2026-09-18 17:56:36.197575288 +0000 UTC m=+400.824706889 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/626cdc85-2717-4abb-82e8-77da8fb69033-tls-certs") pod "kube-auth-proxy-6b7b7f7d54-gwfj9" (UID: "626cdc85-2717-4abb-82e8-77da8fb69033") : secret "kube-auth-proxy-tls" not found
Sep 18 17:56:35.699576 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:35.699561    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/626cdc85-2717-4abb-82e8-77da8fb69033-tmp\") pod \"kube-auth-proxy-6b7b7f7d54-gwfj9\" (UID: \"626cdc85-2717-4abb-82e8-77da8fb69033\") " pod="openshift-ingress/kube-auth-proxy-6b7b7f7d54-gwfj9"
Sep 18 17:56:35.720615 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:35.720591    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pprqm\" (UniqueName: \"kubernetes.io/projected/626cdc85-2717-4abb-82e8-77da8fb69033-kube-api-access-pprqm\") pod \"kube-auth-proxy-6b7b7f7d54-gwfj9\" (UID: \"626cdc85-2717-4abb-82e8-77da8fb69033\") " pod="openshift-ingress/kube-auth-proxy-6b7b7f7d54-gwfj9"
Sep 18 17:56:36.165488 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:36.165453    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:56:36.201930 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:36.201898    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/626cdc85-2717-4abb-82e8-77da8fb69033-tls-certs\") pod \"kube-auth-proxy-6b7b7f7d54-gwfj9\" (UID: \"626cdc85-2717-4abb-82e8-77da8fb69033\") " pod="openshift-ingress/kube-auth-proxy-6b7b7f7d54-gwfj9"
Sep 18 17:56:36.204320 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:36.204302    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/626cdc85-2717-4abb-82e8-77da8fb69033-tls-certs\") pod \"kube-auth-proxy-6b7b7f7d54-gwfj9\" (UID: \"626cdc85-2717-4abb-82e8-77da8fb69033\") " pod="openshift-ingress/kube-auth-proxy-6b7b7f7d54-gwfj9"
Sep 18 17:56:36.441450 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:36.441378    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6b7b7f7d54-gwfj9"
Sep 18 17:56:36.723821 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:36.723745    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6b7b7f7d54-gwfj9"]
Sep 18 17:56:36.726751 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:56:36.726726    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod626cdc85_2717_4abb_82e8_77da8fb69033.slice/crio-1ad66fb773400ada9c32d838f918fe69515f50b49d80c755de1c1097a32f2334 WatchSource:0}: Error finding container 1ad66fb773400ada9c32d838f918fe69515f50b49d80c755de1c1097a32f2334: Status 404 returned error can't find the container with id 1ad66fb773400ada9c32d838f918fe69515f50b49d80c755de1c1097a32f2334
Sep 18 17:56:37.657525 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:37.657489    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6b7b7f7d54-gwfj9" event={"ID":"626cdc85-2717-4abb-82e8-77da8fb69033","Type":"ContainerStarted","Data":"1ad66fb773400ada9c32d838f918fe69515f50b49d80c755de1c1097a32f2334"}
Sep 18 17:56:38.110100 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:38.110036    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:56:38.232416 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:38.232382    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:56:39.843317 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:39.843282    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:56:40.675342 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:40.675298    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-786dfb84b9-bgs76" event={"ID":"ab192091-eb80-4dd9-83bc-54ea383c7c09","Type":"ContainerStarted","Data":"2e3d4886e4454a9fe94236f73e1d7bc269a76001418fb5dae9e5f164f7ea655d"}
Sep 18 17:56:40.675513 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:40.675374    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-786dfb84b9-bgs76"
Sep 18 17:56:40.676756 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:40.676727    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6b7b7f7d54-gwfj9" event={"ID":"626cdc85-2717-4abb-82e8-77da8fb69033","Type":"ContainerStarted","Data":"8cf8ef2c70af78f9c39e30932d597a6a0dae78e5a86642a3e5f88b2a78df0836"}
Sep 18 17:56:40.702383 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:40.702334    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-786dfb84b9-bgs76" podStartSLOduration=1.674972895 podStartE2EDuration="25.702317748s" totalImagesPullingTime="24.027344853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:56:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:56:40.70220183 +0000 UTC m=+405.329333450" watchObservedRunningTime="2026-09-18 17:56:40.702317748 +0000 UTC m=+405.329449379"
Sep 18 17:56:40.727922 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:40.727868    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6b7b7f7d54-gwfj9" podStartSLOduration=2.575826734 podStartE2EDuration="5.727854302s" totalImagesPullingTime="3.152027568s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:56:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:56:40.72707768 +0000 UTC m=+405.354209304" watchObservedRunningTime="2026-09-18 17:56:40.727854302 +0000 UTC m=+405.354985921"
Sep 18 17:56:43.960924 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:43.960880    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:56:44.055001 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:44.054967    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:56:44.284064 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:44.283984    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:56:44.577480 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:44.577404    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:56:44.798420 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:44.798385    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:56:45.948841 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:45.948810    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:56:46.063943 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:46.063911    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:56:46.194494 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:46.194464    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:56:46.695834 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:46.695804    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:56:46.769281 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:46.769250    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:56:47.112645 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:47.112567    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:56:51.684361 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:51.684326    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-786dfb84b9-bgs76"
Sep 18 17:56:57.697542 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:57.697509    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:56:57.826331 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:57.826297    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:56:59.320323 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:59.320279    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:56:59.556746 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:59.556714    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:56:59.815901 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:56:59.815867    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:57:00.398362 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:57:00.398326    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:57:01.001666 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:57:01.001639    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:57:01.138032 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:57:01.138001    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:57:01.254419 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:57:01.254328    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:57:46.787744 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:57:46.787711    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:57:46.882473 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:57:46.882438    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:57:58.505005 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:57:58.504957    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:57:58.605674 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:57:58.605645    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:58:04.086582 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:58:04.086548    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:58:04.818854 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:58:04.818815    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:58:05.603313 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:58:05.603281    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:58:07.373538 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:58:07.373496    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:58:09.060037 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:58:09.060000    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:58:09.692381 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:58:09.692352    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:58:11.043107 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:58:11.043076    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:58:11.549715 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:58:11.549685    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:58:12.182904 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:58:12.182868    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:58:14.428226 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:58:14.428187    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:58:16.319727 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:58:16.319695    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:58:16.684513 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:58:16.684484    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:58:17.189074 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:58:17.189049    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:58:19.834922 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:58:19.834889    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:58:19.911160 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:58:19.911104    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:58:20.235542 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:58:20.235510    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:58:20.324173 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:58:20.324133    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:58:20.644102 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:58:20.644010    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:58:20.735721 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:58:20.735685    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:58:26.830662 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:58:26.830630    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:58:26.957026 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:58:26.956989    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:58:30.120783 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:58:30.120727    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:58:30.218894 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:58:30.218869    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:58:32.899461 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:58:32.899407    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:58:33.019375 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:58:33.019347    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:58:33.144393 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:58:33.144360    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:58:35.877829 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:58:35.877796    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:58:35.990134 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:58:35.990086    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:58:40.864387 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:58:40.864357    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:58:41.056427 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:58:41.056395    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:58:41.340548 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:58:41.340506    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:58:47.669237 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:58:47.669201    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:58:47.834266 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:58:47.834236    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:58:54.021051 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:58:54.020979    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:58:54.121294 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:58:54.121264    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:58:57.064320 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:58:57.064285    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:58:57.183501 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:58:57.183470    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:58:59.307855 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:58:59.307825    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:58:59.438303 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:58:59.438274    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:59:01.273748 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:01.273717    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:59:01.279992 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:01.279967    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-xks86"]
Sep 18 17:59:01.283522 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:01.283505    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-xks86"
Sep 18 17:59:01.286558 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:01.286537    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-v8xtm\""
Sep 18 17:59:01.286669 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:01.286594    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 18 17:59:01.286730 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:01.286664    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 18 17:59:01.287209 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:01.287192    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 18 17:59:01.297676 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:01.297657    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-xks86"]
Sep 18 17:59:01.366572 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:01.366544    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n78hb\" (UniqueName: \"kubernetes.io/projected/87a83d43-7304-4da4-a93b-2685624b1a12-kube-api-access-n78hb\") pod \"limitador-limitador-8588749977-xks86\" (UID: \"87a83d43-7304-4da4-a93b-2685624b1a12\") " pod="kuadrant-system/limitador-limitador-8588749977-xks86"
Sep 18 17:59:01.366757 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:01.366601    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/87a83d43-7304-4da4-a93b-2685624b1a12-config-file\") pod \"limitador-limitador-8588749977-xks86\" (UID: \"87a83d43-7304-4da4-a93b-2685624b1a12\") " pod="kuadrant-system/limitador-limitador-8588749977-xks86"
Sep 18 17:59:01.382210 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:01.382180    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-xks86"]
Sep 18 17:59:01.400960 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:01.400936    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:59:01.467636 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:01.467604    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n78hb\" (UniqueName: \"kubernetes.io/projected/87a83d43-7304-4da4-a93b-2685624b1a12-kube-api-access-n78hb\") pod \"limitador-limitador-8588749977-xks86\" (UID: \"87a83d43-7304-4da4-a93b-2685624b1a12\") " pod="kuadrant-system/limitador-limitador-8588749977-xks86"
Sep 18 17:59:01.467788 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:01.467647    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/87a83d43-7304-4da4-a93b-2685624b1a12-config-file\") pod \"limitador-limitador-8588749977-xks86\" (UID: \"87a83d43-7304-4da4-a93b-2685624b1a12\") " pod="kuadrant-system/limitador-limitador-8588749977-xks86"
Sep 18 17:59:01.468281 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:01.468258    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/87a83d43-7304-4da4-a93b-2685624b1a12-config-file\") pod \"limitador-limitador-8588749977-xks86\" (UID: \"87a83d43-7304-4da4-a93b-2685624b1a12\") " pod="kuadrant-system/limitador-limitador-8588749977-xks86"
Sep 18 17:59:01.477532 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:01.477513    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n78hb\" (UniqueName: \"kubernetes.io/projected/87a83d43-7304-4da4-a93b-2685624b1a12-kube-api-access-n78hb\") pod \"limitador-limitador-8588749977-xks86\" (UID: \"87a83d43-7304-4da4-a93b-2685624b1a12\") " pod="kuadrant-system/limitador-limitador-8588749977-xks86"
Sep 18 17:59:01.504579 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:01.504554    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:59:01.594460 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:01.594384    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-xks86"
Sep 18 17:59:01.865995 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:01.865972    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-xks86"]
Sep 18 17:59:01.868624 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:59:01.868599    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87a83d43_7304_4da4_a93b_2685624b1a12.slice/crio-e448ef89e943b28b3fc5431dc3c3c7c4a563f03a645768a1187d207be1f74542 WatchSource:0}: Error finding container e448ef89e943b28b3fc5431dc3c3c7c4a563f03a645768a1187d207be1f74542: Status 404 returned error can't find the container with id e448ef89e943b28b3fc5431dc3c3c7c4a563f03a645768a1187d207be1f74542
Sep 18 17:59:02.125230 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:02.125142    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-64xw2"]
Sep 18 17:59:02.129753 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:02.129732    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-64xw2"
Sep 18 17:59:02.132308 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:02.132291    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-qlz5b\""
Sep 18 17:59:02.144876 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:02.144849    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-64xw2"]
Sep 18 17:59:02.173475 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:02.173453    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6x87\" (UniqueName: \"kubernetes.io/projected/743e5e52-eb5c-4224-ad84-f0f9d293f44d-kube-api-access-r6x87\") pod \"authorino-7bcfd8f9cd-64xw2\" (UID: \"743e5e52-eb5c-4224-ad84-f0f9d293f44d\") " pod="kuadrant-system/authorino-7bcfd8f9cd-64xw2"
Sep 18 17:59:02.179588 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:02.179566    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:59:02.274323 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:02.274297    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6x87\" (UniqueName: \"kubernetes.io/projected/743e5e52-eb5c-4224-ad84-f0f9d293f44d-kube-api-access-r6x87\") pod \"authorino-7bcfd8f9cd-64xw2\" (UID: \"743e5e52-eb5c-4224-ad84-f0f9d293f44d\") " pod="kuadrant-system/authorino-7bcfd8f9cd-64xw2"
Sep 18 17:59:02.278038 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:02.278019    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:59:02.287945 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:02.287925    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6x87\" (UniqueName: \"kubernetes.io/projected/743e5e52-eb5c-4224-ad84-f0f9d293f44d-kube-api-access-r6x87\") pod \"authorino-7bcfd8f9cd-64xw2\" (UID: \"743e5e52-eb5c-4224-ad84-f0f9d293f44d\") " pod="kuadrant-system/authorino-7bcfd8f9cd-64xw2"
Sep 18 17:59:02.297355 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:02.297334    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-xks86" event={"ID":"87a83d43-7304-4da4-a93b-2685624b1a12","Type":"ContainerStarted","Data":"e448ef89e943b28b3fc5431dc3c3c7c4a563f03a645768a1187d207be1f74542"}
Sep 18 17:59:02.456532 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:02.456503    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-64xw2"
Sep 18 17:59:02.624281 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:02.624249    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:59:02.709542 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:02.709382    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:59:02.735019 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:02.734985    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-64xw2"]
Sep 18 17:59:02.740222 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:59:02.740189    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod743e5e52_eb5c_4224_ad84_f0f9d293f44d.slice/crio-65ac30e54a1911c60cd1378fe579793c661bf5d91df2c5e0056b586abfda18ad WatchSource:0}: Error finding container 65ac30e54a1911c60cd1378fe579793c661bf5d91df2c5e0056b586abfda18ad: Status 404 returned error can't find the container with id 65ac30e54a1911c60cd1378fe579793c661bf5d91df2c5e0056b586abfda18ad
Sep 18 17:59:03.006605 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:03.006498    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:59:03.182986 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:03.182953    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:59:03.286601 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:03.286525    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:59:03.303190 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:03.303154    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-64xw2" event={"ID":"743e5e52-eb5c-4224-ad84-f0f9d293f44d","Type":"ContainerStarted","Data":"65ac30e54a1911c60cd1378fe579793c661bf5d91df2c5e0056b586abfda18ad"}
Sep 18 17:59:03.834595 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:03.834557    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:59:03.961469 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:03.961443    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:59:04.312789 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:04.312747    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-xks86" event={"ID":"87a83d43-7304-4da4-a93b-2685624b1a12","Type":"ContainerStarted","Data":"915f416a46a827433aa28f1fe8c6e62a9b7cf2c500c64a77134ddcf1f27febb8"}
Sep 18 17:59:04.313451 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:04.313061    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-xks86"
Sep 18 17:59:04.334575 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:04.334529    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-xks86" podStartSLOduration=1.020840254 podStartE2EDuration="3.334510767s" totalImagesPullingTime="2.313670513s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:59:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:59:04.333963511 +0000 UTC m=+548.961095152" watchObservedRunningTime="2026-09-18 17:59:04.334510767 +0000 UTC m=+548.961642387"
Sep 18 17:59:06.323875 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:06.323836    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-64xw2" event={"ID":"743e5e52-eb5c-4224-ad84-f0f9d293f44d","Type":"ContainerStarted","Data":"561087f1167cc1db88ef463a2b880533023976407c5fe4c68609bfb9ac8c5831"}
Sep 18 17:59:06.347427 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:06.347381    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-64xw2" podStartSLOduration=1.841563525 podStartE2EDuration="4.34736275s" totalImagesPullingTime="2.505799225s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:59:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:59:06.34637294 +0000 UTC m=+550.973504559" watchObservedRunningTime="2026-09-18 17:59:06.34736275 +0000 UTC m=+550.974494371"
Sep 18 17:59:08.747716 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:08.747686    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-64xw2"]
Sep 18 17:59:08.748150 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:08.747838    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-64xw2" podUID="743e5e52-eb5c-4224-ad84-f0f9d293f44d" containerName="authorino" containerID="cri-o://561087f1167cc1db88ef463a2b880533023976407c5fe4c68609bfb9ac8c5831" gracePeriod=30
Sep 18 17:59:08.988083 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:08.988057    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-64xw2"
Sep 18 17:59:09.132458 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:09.132370    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r6x87\" (UniqueName: \"kubernetes.io/projected/743e5e52-eb5c-4224-ad84-f0f9d293f44d-kube-api-access-r6x87\") pod \"743e5e52-eb5c-4224-ad84-f0f9d293f44d\" (UID: \"743e5e52-eb5c-4224-ad84-f0f9d293f44d\") "
Sep 18 17:59:09.134529 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:09.134506    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/743e5e52-eb5c-4224-ad84-f0f9d293f44d-kube-api-access-r6x87" (OuterVolumeSpecName: "kube-api-access-r6x87") pod "743e5e52-eb5c-4224-ad84-f0f9d293f44d" (UID: "743e5e52-eb5c-4224-ad84-f0f9d293f44d"). InnerVolumeSpecName "kube-api-access-r6x87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:59:09.233133 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:09.233101    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6x87\" (UniqueName: \"kubernetes.io/projected/743e5e52-eb5c-4224-ad84-f0f9d293f44d-kube-api-access-r6x87\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 17:59:09.337797 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:09.337767    2561 generic.go:356] "Generic (PLEG): container finished" podID="743e5e52-eb5c-4224-ad84-f0f9d293f44d" containerID="561087f1167cc1db88ef463a2b880533023976407c5fe4c68609bfb9ac8c5831" exitCode=0
Sep 18 17:59:09.337981 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:09.337811    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-64xw2"
Sep 18 17:59:09.337981 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:09.337828    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-64xw2" event={"ID":"743e5e52-eb5c-4224-ad84-f0f9d293f44d","Type":"ContainerDied","Data":"561087f1167cc1db88ef463a2b880533023976407c5fe4c68609bfb9ac8c5831"}
Sep 18 17:59:09.337981 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:09.337877    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-64xw2" event={"ID":"743e5e52-eb5c-4224-ad84-f0f9d293f44d","Type":"ContainerDied","Data":"65ac30e54a1911c60cd1378fe579793c661bf5d91df2c5e0056b586abfda18ad"}
Sep 18 17:59:09.337981 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:09.337902    2561 scope.go:117] "RemoveContainer" containerID="561087f1167cc1db88ef463a2b880533023976407c5fe4c68609bfb9ac8c5831"
Sep 18 17:59:09.346316 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:09.346298    2561 scope.go:117] "RemoveContainer" containerID="561087f1167cc1db88ef463a2b880533023976407c5fe4c68609bfb9ac8c5831"
Sep 18 17:59:09.346574 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:59:09.346555    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"561087f1167cc1db88ef463a2b880533023976407c5fe4c68609bfb9ac8c5831\": container with ID starting with 561087f1167cc1db88ef463a2b880533023976407c5fe4c68609bfb9ac8c5831 not found: ID does not exist" containerID="561087f1167cc1db88ef463a2b880533023976407c5fe4c68609bfb9ac8c5831"
Sep 18 17:59:09.346639 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:09.346580    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"561087f1167cc1db88ef463a2b880533023976407c5fe4c68609bfb9ac8c5831"} err="failed to get container status \"561087f1167cc1db88ef463a2b880533023976407c5fe4c68609bfb9ac8c5831\": rpc error: code = NotFound desc = could not find container \"561087f1167cc1db88ef463a2b880533023976407c5fe4c68609bfb9ac8c5831\": container with ID starting with 561087f1167cc1db88ef463a2b880533023976407c5fe4c68609bfb9ac8c5831 not found: ID does not exist"
Sep 18 17:59:09.359434 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:09.359410    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-64xw2"]
Sep 18 17:59:09.364052 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:09.364032    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-64xw2"]
Sep 18 17:59:09.860575 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:09.860544    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="743e5e52-eb5c-4224-ad84-f0f9d293f44d" path="/var/lib/kubelet/pods/743e5e52-eb5c-4224-ad84-f0f9d293f44d/volumes"
Sep 18 17:59:10.905631 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:10.905577    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:59:11.063214 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:11.063186    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:59:15.319017 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:15.318991    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-xks86"
Sep 18 17:59:15.750350 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:15.750319    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-xks86"]
Sep 18 17:59:15.750545 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:15.750501    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-xks86" podUID="87a83d43-7304-4da4-a93b-2685624b1a12" containerName="limitador" containerID="cri-o://915f416a46a827433aa28f1fe8c6e62a9b7cf2c500c64a77134ddcf1f27febb8" gracePeriod=30
Sep 18 17:59:16.371240 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:16.371149    2561 generic.go:356] "Generic (PLEG): container finished" podID="87a83d43-7304-4da4-a93b-2685624b1a12" containerID="915f416a46a827433aa28f1fe8c6e62a9b7cf2c500c64a77134ddcf1f27febb8" exitCode=0
Sep 18 17:59:16.371591 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:16.371234    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-xks86" event={"ID":"87a83d43-7304-4da4-a93b-2685624b1a12","Type":"ContainerDied","Data":"915f416a46a827433aa28f1fe8c6e62a9b7cf2c500c64a77134ddcf1f27febb8"}
Sep 18 17:59:16.694566 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:16.694545    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-xks86"
Sep 18 17:59:16.802019 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:16.801989    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/87a83d43-7304-4da4-a93b-2685624b1a12-config-file\") pod \"87a83d43-7304-4da4-a93b-2685624b1a12\" (UID: \"87a83d43-7304-4da4-a93b-2685624b1a12\") "
Sep 18 17:59:16.802177 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:16.802078    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n78hb\" (UniqueName: \"kubernetes.io/projected/87a83d43-7304-4da4-a93b-2685624b1a12-kube-api-access-n78hb\") pod \"87a83d43-7304-4da4-a93b-2685624b1a12\" (UID: \"87a83d43-7304-4da4-a93b-2685624b1a12\") "
Sep 18 17:59:16.802459 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:16.802430    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87a83d43-7304-4da4-a93b-2685624b1a12-config-file" (OuterVolumeSpecName: "config-file") pod "87a83d43-7304-4da4-a93b-2685624b1a12" (UID: "87a83d43-7304-4da4-a93b-2685624b1a12"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:59:16.804202 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:16.804172    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87a83d43-7304-4da4-a93b-2685624b1a12-kube-api-access-n78hb" (OuterVolumeSpecName: "kube-api-access-n78hb") pod "87a83d43-7304-4da4-a93b-2685624b1a12" (UID: "87a83d43-7304-4da4-a93b-2685624b1a12"). InnerVolumeSpecName "kube-api-access-n78hb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:59:16.822505 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:16.822481    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:59:16.903023 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:16.902996    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n78hb\" (UniqueName: \"kubernetes.io/projected/87a83d43-7304-4da4-a93b-2685624b1a12-kube-api-access-n78hb\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 17:59:16.903023 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:16.903018    2561 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/87a83d43-7304-4da4-a93b-2685624b1a12-config-file\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 17:59:16.923178 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:16.923151    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:59:17.376720 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:17.376687    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-xks86" event={"ID":"87a83d43-7304-4da4-a93b-2685624b1a12","Type":"ContainerDied","Data":"e448ef89e943b28b3fc5431dc3c3c7c4a563f03a645768a1187d207be1f74542"}
Sep 18 17:59:17.377206 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:17.376739    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-xks86"
Sep 18 17:59:17.377206 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:17.376745    2561 scope.go:117] "RemoveContainer" containerID="915f416a46a827433aa28f1fe8c6e62a9b7cf2c500c64a77134ddcf1f27febb8"
Sep 18 17:59:17.421749 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:17.421715    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-xks86"]
Sep 18 17:59:17.431708 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:17.431684    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:59:17.438139 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:17.438109    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-xks86"]
Sep 18 17:59:17.632620 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:17.632532    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:59:17.803158 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:17.803130    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-646hr"]
Sep 18 17:59:17.803501 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:17.803484    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87a83d43-7304-4da4-a93b-2685624b1a12" containerName="limitador"
Sep 18 17:59:17.803501 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:17.803503    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="87a83d43-7304-4da4-a93b-2685624b1a12" containerName="limitador"
Sep 18 17:59:17.803668 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:17.803526    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="743e5e52-eb5c-4224-ad84-f0f9d293f44d" containerName="authorino"
Sep 18 17:59:17.803668 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:17.803532    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="743e5e52-eb5c-4224-ad84-f0f9d293f44d" containerName="authorino"
Sep 18 17:59:17.803668 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:17.803603    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="743e5e52-eb5c-4224-ad84-f0f9d293f44d" containerName="authorino"
Sep 18 17:59:17.803668 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:17.803616    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="87a83d43-7304-4da4-a93b-2685624b1a12" containerName="limitador"
Sep 18 17:59:17.807761 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:17.807741    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-646hr"
Sep 18 17:59:17.811556 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:17.811536    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 18 17:59:17.811849 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:17.811832    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 17:59:17.811998 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:17.811982    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-jvx9p\""
Sep 18 17:59:17.812079 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:17.811980    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 18 17:59:17.825172 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:17.825151    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-646hr"]
Sep 18 17:59:17.827583 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:17.827557    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:59:17.861609 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:17.861578    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87a83d43-7304-4da4-a93b-2685624b1a12" path="/var/lib/kubelet/pods/87a83d43-7304-4da4-a93b-2685624b1a12/volumes"
Sep 18 17:59:17.910988 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:17.910966    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d8ac5f0d-0c80-4a92-b41d-d49c6a66cbf5-data\") pod \"postgres-69fb4594cf-646hr\" (UID: \"d8ac5f0d-0c80-4a92-b41d-d49c6a66cbf5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-646hr"
Sep 18 17:59:17.911080 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:17.911006    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m662k\" (UniqueName: \"kubernetes.io/projected/d8ac5f0d-0c80-4a92-b41d-d49c6a66cbf5-kube-api-access-m662k\") pod \"postgres-69fb4594cf-646hr\" (UID: \"d8ac5f0d-0c80-4a92-b41d-d49c6a66cbf5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-646hr"
Sep 18 17:59:17.913225 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:17.913210    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:59:18.012089 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:18.012059    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d8ac5f0d-0c80-4a92-b41d-d49c6a66cbf5-data\") pod \"postgres-69fb4594cf-646hr\" (UID: \"d8ac5f0d-0c80-4a92-b41d-d49c6a66cbf5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-646hr"
Sep 18 17:59:18.012230 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:18.012105    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m662k\" (UniqueName: \"kubernetes.io/projected/d8ac5f0d-0c80-4a92-b41d-d49c6a66cbf5-kube-api-access-m662k\") pod \"postgres-69fb4594cf-646hr\" (UID: \"d8ac5f0d-0c80-4a92-b41d-d49c6a66cbf5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-646hr"
Sep 18 17:59:18.012451 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:18.012432    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d8ac5f0d-0c80-4a92-b41d-d49c6a66cbf5-data\") pod \"postgres-69fb4594cf-646hr\" (UID: \"d8ac5f0d-0c80-4a92-b41d-d49c6a66cbf5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-646hr"
Sep 18 17:59:18.022232 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:18.022207    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m662k\" (UniqueName: \"kubernetes.io/projected/d8ac5f0d-0c80-4a92-b41d-d49c6a66cbf5-kube-api-access-m662k\") pod \"postgres-69fb4594cf-646hr\" (UID: \"d8ac5f0d-0c80-4a92-b41d-d49c6a66cbf5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-646hr"
Sep 18 17:59:18.120658 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:18.120618    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-646hr"
Sep 18 17:59:18.396148 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:18.396005    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-646hr"]
Sep 18 17:59:18.398573 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:59:18.398545    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8ac5f0d_0c80_4a92_b41d_d49c6a66cbf5.slice/crio-e3c77356e23826f7b2928adc6af2e2cd5ea89d9ac788a77792c1d0126129d4e6 WatchSource:0}: Error finding container e3c77356e23826f7b2928adc6af2e2cd5ea89d9ac788a77792c1d0126129d4e6: Status 404 returned error can't find the container with id e3c77356e23826f7b2928adc6af2e2cd5ea89d9ac788a77792c1d0126129d4e6
Sep 18 17:59:19.390472 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:19.390417    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-646hr" event={"ID":"d8ac5f0d-0c80-4a92-b41d-d49c6a66cbf5","Type":"ContainerStarted","Data":"e3c77356e23826f7b2928adc6af2e2cd5ea89d9ac788a77792c1d0126129d4e6"}
Sep 18 17:59:21.460604 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:21.460570    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:59:21.584445 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:21.584409    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:59:23.413377 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:23.413343    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-646hr" event={"ID":"d8ac5f0d-0c80-4a92-b41d-d49c6a66cbf5","Type":"ContainerStarted","Data":"6e560b1b48b46deb20b0053dda126460c12c09af9242ea3337a4d354b7a4e4a8"}
Sep 18 17:59:23.413847 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:23.413472    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-646hr"
Sep 18 17:59:23.436952 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:23.436908    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-646hr" podStartSLOduration=1.962075319 podStartE2EDuration="6.436894691s" totalImagesPullingTime="4.474819372s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:59:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:59:23.435331571 +0000 UTC m=+568.062463186" watchObservedRunningTime="2026-09-18 17:59:23.436894691 +0000 UTC m=+568.064026311"
Sep 18 17:59:29.446830 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:29.446801    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-646hr"
Sep 18 17:59:29.763677 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:29.763565    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:59:29.986857 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:29.986817    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-fgpk8"]
Sep 18 17:59:29.992858 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:29.992835    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-fgpk8"
Sep 18 17:59:29.995953 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:29.995932    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-qlz5b\""
Sep 18 17:59:29.996095 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:29.995958    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 18 17:59:29.996672 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:29.996653    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 18 17:59:30.002278 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:30.002255    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-fgpk8"]
Sep 18 17:59:30.118926 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:30.118842    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjvmr\" (UniqueName: \"kubernetes.io/projected/c9120009-be37-4544-8f6b-f873e60d53fd-kube-api-access-pjvmr\") pod \"authorino-557f876f69-fgpk8\" (UID: \"c9120009-be37-4544-8f6b-f873e60d53fd\") " pod="kuadrant-system/authorino-557f876f69-fgpk8"
Sep 18 17:59:30.219445 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:30.219412    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjvmr\" (UniqueName: \"kubernetes.io/projected/c9120009-be37-4544-8f6b-f873e60d53fd-kube-api-access-pjvmr\") pod \"authorino-557f876f69-fgpk8\" (UID: \"c9120009-be37-4544-8f6b-f873e60d53fd\") " pod="kuadrant-system/authorino-557f876f69-fgpk8"
Sep 18 17:59:30.219988 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:30.219967    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-fgpk8"]
Sep 18 17:59:30.220288 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:59:30.220263    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-pjvmr], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-fgpk8" podUID="c9120009-be37-4544-8f6b-f873e60d53fd"
Sep 18 17:59:30.228507 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:30.228485    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:59:30.232128 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:30.232101    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjvmr\" (UniqueName: \"kubernetes.io/projected/c9120009-be37-4544-8f6b-f873e60d53fd-kube-api-access-pjvmr\") pod \"authorino-557f876f69-fgpk8\" (UID: \"c9120009-be37-4544-8f6b-f873e60d53fd\") " pod="kuadrant-system/authorino-557f876f69-fgpk8"
Sep 18 17:59:30.256023 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:30.255994    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-ffcb49f79-xx9ps"]
Sep 18 17:59:30.259699 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:30.259684    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-ffcb49f79-xx9ps"
Sep 18 17:59:30.269204 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:30.269184    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-ffcb49f79-xx9ps"]
Sep 18 17:59:30.315412 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:30.315376    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-ffcb49f79-xx9ps"]
Sep 18 17:59:30.315676 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:59:30.315656    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-2jc4t], unattached volumes=[], failed to process volumes=[kube-api-access-2jc4t]: context canceled" pod="kuadrant-system/authorino-ffcb49f79-xx9ps" podUID="0981a9f4-5d98-416e-95a6-9973a8fcd324"
Sep 18 17:59:30.348504 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:30.348471    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-77544959bf-j8vjd"]
Sep 18 17:59:30.352173 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:30.352154    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-77544959bf-j8vjd"
Sep 18 17:59:30.354990 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:30.354974    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 18 17:59:30.358977 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:30.358957    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-77544959bf-j8vjd"]
Sep 18 17:59:30.421343 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:30.421300    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jc4t\" (UniqueName: \"kubernetes.io/projected/0981a9f4-5d98-416e-95a6-9973a8fcd324-kube-api-access-2jc4t\") pod \"authorino-ffcb49f79-xx9ps\" (UID: \"0981a9f4-5d98-416e-95a6-9973a8fcd324\") " pod="kuadrant-system/authorino-ffcb49f79-xx9ps"
Sep 18 17:59:30.451345 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:30.451316    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-ffcb49f79-xx9ps"
Sep 18 17:59:30.451726 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:30.451316    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-fgpk8"
Sep 18 17:59:30.470155 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:30.470133    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-ffcb49f79-xx9ps"
Sep 18 17:59:30.473683 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:30.473662    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-fgpk8"
Sep 18 17:59:30.522354 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:30.522317    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qjpb\" (UniqueName: \"kubernetes.io/projected/ff96b8bb-4fbd-4a53-af89-bf54eabc36c0-kube-api-access-8qjpb\") pod \"authorino-77544959bf-j8vjd\" (UID: \"ff96b8bb-4fbd-4a53-af89-bf54eabc36c0\") " pod="kuadrant-system/authorino-77544959bf-j8vjd"
Sep 18 17:59:30.522354 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:30.522361    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ff96b8bb-4fbd-4a53-af89-bf54eabc36c0-tls-cert\") pod \"authorino-77544959bf-j8vjd\" (UID: \"ff96b8bb-4fbd-4a53-af89-bf54eabc36c0\") " pod="kuadrant-system/authorino-77544959bf-j8vjd"
Sep 18 17:59:30.522561 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:30.522475    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jc4t\" (UniqueName: \"kubernetes.io/projected/0981a9f4-5d98-416e-95a6-9973a8fcd324-kube-api-access-2jc4t\") pod \"authorino-ffcb49f79-xx9ps\" (UID: \"0981a9f4-5d98-416e-95a6-9973a8fcd324\") " pod="kuadrant-system/authorino-ffcb49f79-xx9ps"
Sep 18 17:59:30.531940 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:30.531914    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jc4t\" (UniqueName: \"kubernetes.io/projected/0981a9f4-5d98-416e-95a6-9973a8fcd324-kube-api-access-2jc4t\") pod \"authorino-ffcb49f79-xx9ps\" (UID: \"0981a9f4-5d98-416e-95a6-9973a8fcd324\") " pod="kuadrant-system/authorino-ffcb49f79-xx9ps"
Sep 18 17:59:30.622827 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:30.622796    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pjvmr\" (UniqueName: \"kubernetes.io/projected/c9120009-be37-4544-8f6b-f873e60d53fd-kube-api-access-pjvmr\") pod \"c9120009-be37-4544-8f6b-f873e60d53fd\" (UID: \"c9120009-be37-4544-8f6b-f873e60d53fd\") "
Sep 18 17:59:30.623026 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:30.622985    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qjpb\" (UniqueName: \"kubernetes.io/projected/ff96b8bb-4fbd-4a53-af89-bf54eabc36c0-kube-api-access-8qjpb\") pod \"authorino-77544959bf-j8vjd\" (UID: \"ff96b8bb-4fbd-4a53-af89-bf54eabc36c0\") " pod="kuadrant-system/authorino-77544959bf-j8vjd"
Sep 18 17:59:30.623142 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:30.623032    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ff96b8bb-4fbd-4a53-af89-bf54eabc36c0-tls-cert\") pod \"authorino-77544959bf-j8vjd\" (UID: \"ff96b8bb-4fbd-4a53-af89-bf54eabc36c0\") " pod="kuadrant-system/authorino-77544959bf-j8vjd"
Sep 18 17:59:30.625052 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:30.624979    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9120009-be37-4544-8f6b-f873e60d53fd-kube-api-access-pjvmr" (OuterVolumeSpecName: "kube-api-access-pjvmr") pod "c9120009-be37-4544-8f6b-f873e60d53fd" (UID: "c9120009-be37-4544-8f6b-f873e60d53fd"). InnerVolumeSpecName "kube-api-access-pjvmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:59:30.625479 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:30.625461    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ff96b8bb-4fbd-4a53-af89-bf54eabc36c0-tls-cert\") pod \"authorino-77544959bf-j8vjd\" (UID: \"ff96b8bb-4fbd-4a53-af89-bf54eabc36c0\") " pod="kuadrant-system/authorino-77544959bf-j8vjd"
Sep 18 17:59:30.633065 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:30.633042    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qjpb\" (UniqueName: \"kubernetes.io/projected/ff96b8bb-4fbd-4a53-af89-bf54eabc36c0-kube-api-access-8qjpb\") pod \"authorino-77544959bf-j8vjd\" (UID: \"ff96b8bb-4fbd-4a53-af89-bf54eabc36c0\") " pod="kuadrant-system/authorino-77544959bf-j8vjd"
Sep 18 17:59:30.676527 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:30.676502    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-77544959bf-j8vjd"
Sep 18 17:59:30.724422 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:30.724230    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2jc4t\" (UniqueName: \"kubernetes.io/projected/0981a9f4-5d98-416e-95a6-9973a8fcd324-kube-api-access-2jc4t\") pod \"0981a9f4-5d98-416e-95a6-9973a8fcd324\" (UID: \"0981a9f4-5d98-416e-95a6-9973a8fcd324\") "
Sep 18 17:59:30.724515 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:30.724435    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjvmr\" (UniqueName: \"kubernetes.io/projected/c9120009-be37-4544-8f6b-f873e60d53fd-kube-api-access-pjvmr\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 17:59:30.726957 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:30.726846    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0981a9f4-5d98-416e-95a6-9973a8fcd324-kube-api-access-2jc4t" (OuterVolumeSpecName: "kube-api-access-2jc4t") pod "0981a9f4-5d98-416e-95a6-9973a8fcd324" (UID: "0981a9f4-5d98-416e-95a6-9973a8fcd324"). InnerVolumeSpecName "kube-api-access-2jc4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:59:30.825451 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:30.825403    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jc4t\" (UniqueName: \"kubernetes.io/projected/0981a9f4-5d98-416e-95a6-9973a8fcd324-kube-api-access-2jc4t\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 17:59:30.947452 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:30.947426    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-77544959bf-j8vjd"]
Sep 18 17:59:30.949897 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:59:30.949868    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podff96b8bb_4fbd_4a53_af89_bf54eabc36c0.slice/crio-d3a0285db94b27bf89a38f8dc73b16995b25fa4fa2d31315c4a1b11b383e061d WatchSource:0}: Error finding container d3a0285db94b27bf89a38f8dc73b16995b25fa4fa2d31315c4a1b11b383e061d: Status 404 returned error can't find the container with id d3a0285db94b27bf89a38f8dc73b16995b25fa4fa2d31315c4a1b11b383e061d
Sep 18 17:59:31.458197 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:31.458158    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-77544959bf-j8vjd" event={"ID":"ff96b8bb-4fbd-4a53-af89-bf54eabc36c0","Type":"ContainerStarted","Data":"d18e82cfb4f4f2ed1d284c16274de792cb58041d47b66ff57bb5b332c45d7030"}
Sep 18 17:59:31.458531 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:31.458217    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-fgpk8"
Sep 18 17:59:31.458531 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:31.458218    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-77544959bf-j8vjd" event={"ID":"ff96b8bb-4fbd-4a53-af89-bf54eabc36c0","Type":"ContainerStarted","Data":"d3a0285db94b27bf89a38f8dc73b16995b25fa4fa2d31315c4a1b11b383e061d"}
Sep 18 17:59:31.458531 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:31.458223    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-ffcb49f79-xx9ps"
Sep 18 17:59:31.479273 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:31.479229    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-77544959bf-j8vjd" podStartSLOduration=1.054963659 podStartE2EDuration="1.479213119s" totalImagesPullingTime="424.24946ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:59:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:59:31.477258978 +0000 UTC m=+576.104390596" watchObservedRunningTime="2026-09-18 17:59:31.479213119 +0000 UTC m=+576.106344739"
Sep 18 17:59:31.538656 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:31.538567    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-fgpk8"]
Sep 18 17:59:31.542892 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:31.542866    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-fgpk8"]
Sep 18 17:59:31.576200 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:31.576173    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-ffcb49f79-xx9ps"]
Sep 18 17:59:31.580524 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:31.580494    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-ffcb49f79-xx9ps"]
Sep 18 17:59:31.865707 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:31.865625    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0981a9f4-5d98-416e-95a6-9973a8fcd324" path="/var/lib/kubelet/pods/0981a9f4-5d98-416e-95a6-9973a8fcd324/volumes"
Sep 18 17:59:31.865866 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:31.865853    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9120009-be37-4544-8f6b-f873e60d53fd" path="/var/lib/kubelet/pods/c9120009-be37-4544-8f6b-f873e60d53fd/volumes"
Sep 18 17:59:36.442015 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:36.441968    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:59:36.498874 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:36.498838    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5d67c7fcff-6t9rk"]
Sep 18 17:59:36.516926 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:36.516899    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5d67c7fcff-6t9rk"
Sep 18 17:59:36.521827 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:36.521787    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 18 17:59:36.521967 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:36.521953    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-f744x\""
Sep 18 17:59:36.523347 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:36.523295    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 18 17:59:36.523493 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:36.523364    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 18 17:59:36.525618 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:36.525597    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5d67c7fcff-6t9rk"]
Sep 18 17:59:36.552958 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:36.552912    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:59:36.570261 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:36.570235    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80b4f894-6ff7-41c1-b7d6-7d92ea276873-metrics-certs\") pod \"maas-controller-5d67c7fcff-6t9rk\" (UID: \"80b4f894-6ff7-41c1-b7d6-7d92ea276873\") " pod="opendatahub/maas-controller-5d67c7fcff-6t9rk"
Sep 18 17:59:36.570383 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:36.570281    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/80b4f894-6ff7-41c1-b7d6-7d92ea276873-webhook-cert\") pod \"maas-controller-5d67c7fcff-6t9rk\" (UID: \"80b4f894-6ff7-41c1-b7d6-7d92ea276873\") " pod="opendatahub/maas-controller-5d67c7fcff-6t9rk"
Sep 18 17:59:36.570383 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:36.570308    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvfp4\" (UniqueName: \"kubernetes.io/projected/80b4f894-6ff7-41c1-b7d6-7d92ea276873-kube-api-access-kvfp4\") pod \"maas-controller-5d67c7fcff-6t9rk\" (UID: \"80b4f894-6ff7-41c1-b7d6-7d92ea276873\") " pod="opendatahub/maas-controller-5d67c7fcff-6t9rk"
Sep 18 17:59:36.670692 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:36.670663    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/80b4f894-6ff7-41c1-b7d6-7d92ea276873-webhook-cert\") pod \"maas-controller-5d67c7fcff-6t9rk\" (UID: \"80b4f894-6ff7-41c1-b7d6-7d92ea276873\") " pod="opendatahub/maas-controller-5d67c7fcff-6t9rk"
Sep 18 17:59:36.670850 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:36.670701    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvfp4\" (UniqueName: \"kubernetes.io/projected/80b4f894-6ff7-41c1-b7d6-7d92ea276873-kube-api-access-kvfp4\") pod \"maas-controller-5d67c7fcff-6t9rk\" (UID: \"80b4f894-6ff7-41c1-b7d6-7d92ea276873\") " pod="opendatahub/maas-controller-5d67c7fcff-6t9rk"
Sep 18 17:59:36.670850 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:36.670767    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80b4f894-6ff7-41c1-b7d6-7d92ea276873-metrics-certs\") pod \"maas-controller-5d67c7fcff-6t9rk\" (UID: \"80b4f894-6ff7-41c1-b7d6-7d92ea276873\") " pod="opendatahub/maas-controller-5d67c7fcff-6t9rk"
Sep 18 17:59:36.670850 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:59:36.670813    2561 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Sep 18 17:59:36.670984 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:59:36.670886    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80b4f894-6ff7-41c1-b7d6-7d92ea276873-webhook-cert podName:80b4f894-6ff7-41c1-b7d6-7d92ea276873 nodeName:}" failed. No retries permitted until 2026-09-18 17:59:37.170869633 +0000 UTC m=+581.798001231 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/80b4f894-6ff7-41c1-b7d6-7d92ea276873-webhook-cert") pod "maas-controller-5d67c7fcff-6t9rk" (UID: "80b4f894-6ff7-41c1-b7d6-7d92ea276873") : secret "maas-controller-webhook-cert" not found
Sep 18 17:59:36.673251 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:36.673235    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80b4f894-6ff7-41c1-b7d6-7d92ea276873-metrics-certs\") pod \"maas-controller-5d67c7fcff-6t9rk\" (UID: \"80b4f894-6ff7-41c1-b7d6-7d92ea276873\") " pod="opendatahub/maas-controller-5d67c7fcff-6t9rk"
Sep 18 17:59:36.686345 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:36.686323    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvfp4\" (UniqueName: \"kubernetes.io/projected/80b4f894-6ff7-41c1-b7d6-7d92ea276873-kube-api-access-kvfp4\") pod \"maas-controller-5d67c7fcff-6t9rk\" (UID: \"80b4f894-6ff7-41c1-b7d6-7d92ea276873\") " pod="opendatahub/maas-controller-5d67c7fcff-6t9rk"
Sep 18 17:59:36.734465 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:36.734376    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:59:36.738660 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:36.738631    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-655d8db75-qgn94"]
Sep 18 17:59:36.742689 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:36.742656    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-655d8db75-qgn94"
Sep 18 17:59:36.771907 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:36.771882    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/25e183d8-1a1e-429e-81f6-ea656ad36be5-webhook-cert\") pod \"maas-controller-655d8db75-qgn94\" (UID: \"25e183d8-1a1e-429e-81f6-ea656ad36be5\") " pod="opendatahub/maas-controller-655d8db75-qgn94"
Sep 18 17:59:36.772022 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:36.771981    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t72mq\" (UniqueName: \"kubernetes.io/projected/25e183d8-1a1e-429e-81f6-ea656ad36be5-kube-api-access-t72mq\") pod \"maas-controller-655d8db75-qgn94\" (UID: \"25e183d8-1a1e-429e-81f6-ea656ad36be5\") " pod="opendatahub/maas-controller-655d8db75-qgn94"
Sep 18 17:59:36.772077 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:36.772029    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25e183d8-1a1e-429e-81f6-ea656ad36be5-metrics-certs\") pod \"maas-controller-655d8db75-qgn94\" (UID: \"25e183d8-1a1e-429e-81f6-ea656ad36be5\") " pod="opendatahub/maas-controller-655d8db75-qgn94"
Sep 18 17:59:36.834251 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:36.834217    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-655d8db75-qgn94"]
Sep 18 17:59:36.872634 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:36.872607    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t72mq\" (UniqueName: \"kubernetes.io/projected/25e183d8-1a1e-429e-81f6-ea656ad36be5-kube-api-access-t72mq\") pod \"maas-controller-655d8db75-qgn94\" (UID: \"25e183d8-1a1e-429e-81f6-ea656ad36be5\") " pod="opendatahub/maas-controller-655d8db75-qgn94"
Sep 18 17:59:36.872780 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:36.872648    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25e183d8-1a1e-429e-81f6-ea656ad36be5-metrics-certs\") pod \"maas-controller-655d8db75-qgn94\" (UID: \"25e183d8-1a1e-429e-81f6-ea656ad36be5\") " pod="opendatahub/maas-controller-655d8db75-qgn94"
Sep 18 17:59:36.872780 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:36.872678    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/25e183d8-1a1e-429e-81f6-ea656ad36be5-webhook-cert\") pod \"maas-controller-655d8db75-qgn94\" (UID: \"25e183d8-1a1e-429e-81f6-ea656ad36be5\") " pod="opendatahub/maas-controller-655d8db75-qgn94"
Sep 18 17:59:36.875007 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:36.874984    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/25e183d8-1a1e-429e-81f6-ea656ad36be5-webhook-cert\") pod \"maas-controller-655d8db75-qgn94\" (UID: \"25e183d8-1a1e-429e-81f6-ea656ad36be5\") " pod="opendatahub/maas-controller-655d8db75-qgn94"
Sep 18 17:59:36.875143 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:36.875016    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25e183d8-1a1e-429e-81f6-ea656ad36be5-metrics-certs\") pod \"maas-controller-655d8db75-qgn94\" (UID: \"25e183d8-1a1e-429e-81f6-ea656ad36be5\") " pod="opendatahub/maas-controller-655d8db75-qgn94"
Sep 18 17:59:36.885207 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:36.885184    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t72mq\" (UniqueName: \"kubernetes.io/projected/25e183d8-1a1e-429e-81f6-ea656ad36be5-kube-api-access-t72mq\") pod \"maas-controller-655d8db75-qgn94\" (UID: \"25e183d8-1a1e-429e-81f6-ea656ad36be5\") " pod="opendatahub/maas-controller-655d8db75-qgn94"
Sep 18 17:59:36.919574 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:36.919553    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:59:37.053548 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:37.053465    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-655d8db75-qgn94"
Sep 18 17:59:37.175071 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:37.175040    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/80b4f894-6ff7-41c1-b7d6-7d92ea276873-webhook-cert\") pod \"maas-controller-5d67c7fcff-6t9rk\" (UID: \"80b4f894-6ff7-41c1-b7d6-7d92ea276873\") " pod="opendatahub/maas-controller-5d67c7fcff-6t9rk"
Sep 18 17:59:37.177422 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:37.177400    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/80b4f894-6ff7-41c1-b7d6-7d92ea276873-webhook-cert\") pod \"maas-controller-5d67c7fcff-6t9rk\" (UID: \"80b4f894-6ff7-41c1-b7d6-7d92ea276873\") " pod="opendatahub/maas-controller-5d67c7fcff-6t9rk"
Sep 18 17:59:37.332588 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:37.332498    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-655d8db75-qgn94"]
Sep 18 17:59:37.335902 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:59:37.335875    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25e183d8_1a1e_429e_81f6_ea656ad36be5.slice/crio-c80e1211d57418c80f6576a05a11d84e2ba643e1fb6fa58e066bf0688d4b256f WatchSource:0}: Error finding container c80e1211d57418c80f6576a05a11d84e2ba643e1fb6fa58e066bf0688d4b256f: Status 404 returned error can't find the container with id c80e1211d57418c80f6576a05a11d84e2ba643e1fb6fa58e066bf0688d4b256f
Sep 18 17:59:37.428695 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:37.428659    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5d67c7fcff-6t9rk"
Sep 18 17:59:37.490156 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:37.490094    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-655d8db75-qgn94" event={"ID":"25e183d8-1a1e-429e-81f6-ea656ad36be5","Type":"ContainerStarted","Data":"c80e1211d57418c80f6576a05a11d84e2ba643e1fb6fa58e066bf0688d4b256f"}
Sep 18 17:59:37.706137 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:37.706094    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5d67c7fcff-6t9rk"]
Sep 18 17:59:37.710192 ip-10-0-140-54 kubenswrapper[2561]: W0918 17:59:37.710163    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80b4f894_6ff7_41c1_b7d6_7d92ea276873.slice/crio-73b70174fab0eb8eb1b86fa86cdcdbb25b575caf657ef793a480dcc9e700cb4a WatchSource:0}: Error finding container 73b70174fab0eb8eb1b86fa86cdcdbb25b575caf657ef793a480dcc9e700cb4a: Status 404 returned error can't find the container with id 73b70174fab0eb8eb1b86fa86cdcdbb25b575caf657ef793a480dcc9e700cb4a
Sep 18 17:59:38.506083 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:38.506044    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5d67c7fcff-6t9rk" event={"ID":"80b4f894-6ff7-41c1-b7d6-7d92ea276873","Type":"ContainerStarted","Data":"73b70174fab0eb8eb1b86fa86cdcdbb25b575caf657ef793a480dcc9e700cb4a"}
Sep 18 17:59:40.122989 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:40.122915    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:59:40.231027 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:40.230993    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 17:59:40.517801 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:40.517767    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-655d8db75-qgn94" event={"ID":"25e183d8-1a1e-429e-81f6-ea656ad36be5","Type":"ContainerStarted","Data":"0680b8b2f3db1e1f1ec26af26bc89d36a231a0ac0fe7b214bb03d854ab8cd023"}
Sep 18 17:59:40.517987 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:40.517923    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-655d8db75-qgn94"
Sep 18 17:59:40.519276 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:40.519254    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5d67c7fcff-6t9rk" event={"ID":"80b4f894-6ff7-41c1-b7d6-7d92ea276873","Type":"ContainerStarted","Data":"1006229b6f76534b1f22b16a9ac997b2bf6cec59081e8823d38173ba0b4f33fc"}
Sep 18 17:59:40.519386 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:40.519359    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5d67c7fcff-6t9rk"
Sep 18 17:59:40.586861 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:40.586799    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-655d8db75-qgn94" podStartSLOduration=2.095701449 podStartE2EDuration="4.586781639s" totalImagesPullingTime="2.49108019s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:59:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:59:40.559878264 +0000 UTC m=+585.187009884" watchObservedRunningTime="2026-09-18 17:59:40.586781639 +0000 UTC m=+585.213913259"
Sep 18 17:59:40.587812 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:40.587772    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5d67c7fcff-6t9rk" podStartSLOduration=2.468103598 podStartE2EDuration="4.587761995s" totalImagesPullingTime="2.119658397s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:59:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:59:40.586180183 +0000 UTC m=+585.213311824" watchObservedRunningTime="2026-09-18 17:59:40.587761995 +0000 UTC m=+585.214893615"
Sep 18 17:59:51.525227 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:51.525199    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-655d8db75-qgn94"
Sep 18 17:59:51.525578 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:51.525265    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5d67c7fcff-6t9rk"
Sep 18 17:59:51.589977 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:51.589943    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5d67c7fcff-6t9rk"]
Sep 18 17:59:51.590185 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:51.590163    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5d67c7fcff-6t9rk" podUID="80b4f894-6ff7-41c1-b7d6-7d92ea276873" containerName="manager" containerID="cri-o://1006229b6f76534b1f22b16a9ac997b2bf6cec59081e8823d38173ba0b4f33fc" gracePeriod=30
Sep 18 17:59:51.834201 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:51.834177    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5d67c7fcff-6t9rk"
Sep 18 17:59:51.900340 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:51.900315    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kvfp4\" (UniqueName: \"kubernetes.io/projected/80b4f894-6ff7-41c1-b7d6-7d92ea276873-kube-api-access-kvfp4\") pod \"80b4f894-6ff7-41c1-b7d6-7d92ea276873\" (UID: \"80b4f894-6ff7-41c1-b7d6-7d92ea276873\") "
Sep 18 17:59:51.900472 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:51.900380    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/80b4f894-6ff7-41c1-b7d6-7d92ea276873-webhook-cert\") pod \"80b4f894-6ff7-41c1-b7d6-7d92ea276873\" (UID: \"80b4f894-6ff7-41c1-b7d6-7d92ea276873\") "
Sep 18 17:59:51.900472 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:51.900424    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80b4f894-6ff7-41c1-b7d6-7d92ea276873-metrics-certs\") pod \"80b4f894-6ff7-41c1-b7d6-7d92ea276873\" (UID: \"80b4f894-6ff7-41c1-b7d6-7d92ea276873\") "
Sep 18 17:59:51.902691 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:51.902661    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80b4f894-6ff7-41c1-b7d6-7d92ea276873-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "80b4f894-6ff7-41c1-b7d6-7d92ea276873" (UID: "80b4f894-6ff7-41c1-b7d6-7d92ea276873"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:59:51.902959 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:51.902938    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80b4f894-6ff7-41c1-b7d6-7d92ea276873-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "80b4f894-6ff7-41c1-b7d6-7d92ea276873" (UID: "80b4f894-6ff7-41c1-b7d6-7d92ea276873"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:59:51.903026 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:51.902941    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80b4f894-6ff7-41c1-b7d6-7d92ea276873-kube-api-access-kvfp4" (OuterVolumeSpecName: "kube-api-access-kvfp4") pod "80b4f894-6ff7-41c1-b7d6-7d92ea276873" (UID: "80b4f894-6ff7-41c1-b7d6-7d92ea276873"). InnerVolumeSpecName "kube-api-access-kvfp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:59:52.001385 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:52.001351    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvfp4\" (UniqueName: \"kubernetes.io/projected/80b4f894-6ff7-41c1-b7d6-7d92ea276873-kube-api-access-kvfp4\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 17:59:52.001385 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:52.001380    2561 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/80b4f894-6ff7-41c1-b7d6-7d92ea276873-webhook-cert\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 17:59:52.001555 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:52.001393    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80b4f894-6ff7-41c1-b7d6-7d92ea276873-metrics-certs\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 17:59:52.584448 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:52.584410    2561 generic.go:356] "Generic (PLEG): container finished" podID="80b4f894-6ff7-41c1-b7d6-7d92ea276873" containerID="1006229b6f76534b1f22b16a9ac997b2bf6cec59081e8823d38173ba0b4f33fc" exitCode=0
Sep 18 17:59:52.584846 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:52.584457    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5d67c7fcff-6t9rk"
Sep 18 17:59:52.584846 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:52.584454    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5d67c7fcff-6t9rk" event={"ID":"80b4f894-6ff7-41c1-b7d6-7d92ea276873","Type":"ContainerDied","Data":"1006229b6f76534b1f22b16a9ac997b2bf6cec59081e8823d38173ba0b4f33fc"}
Sep 18 17:59:52.584846 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:52.584588    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5d67c7fcff-6t9rk" event={"ID":"80b4f894-6ff7-41c1-b7d6-7d92ea276873","Type":"ContainerDied","Data":"73b70174fab0eb8eb1b86fa86cdcdbb25b575caf657ef793a480dcc9e700cb4a"}
Sep 18 17:59:52.584846 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:52.584618    2561 scope.go:117] "RemoveContainer" containerID="1006229b6f76534b1f22b16a9ac997b2bf6cec59081e8823d38173ba0b4f33fc"
Sep 18 17:59:52.593794 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:52.593620    2561 scope.go:117] "RemoveContainer" containerID="1006229b6f76534b1f22b16a9ac997b2bf6cec59081e8823d38173ba0b4f33fc"
Sep 18 17:59:52.593932 ip-10-0-140-54 kubenswrapper[2561]: E0918 17:59:52.593912    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1006229b6f76534b1f22b16a9ac997b2bf6cec59081e8823d38173ba0b4f33fc\": container with ID starting with 1006229b6f76534b1f22b16a9ac997b2bf6cec59081e8823d38173ba0b4f33fc not found: ID does not exist" containerID="1006229b6f76534b1f22b16a9ac997b2bf6cec59081e8823d38173ba0b4f33fc"
Sep 18 17:59:52.593975 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:52.593942    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1006229b6f76534b1f22b16a9ac997b2bf6cec59081e8823d38173ba0b4f33fc"} err="failed to get container status \"1006229b6f76534b1f22b16a9ac997b2bf6cec59081e8823d38173ba0b4f33fc\": rpc error: code = NotFound desc = could not find container \"1006229b6f76534b1f22b16a9ac997b2bf6cec59081e8823d38173ba0b4f33fc\": container with ID starting with 1006229b6f76534b1f22b16a9ac997b2bf6cec59081e8823d38173ba0b4f33fc not found: ID does not exist"
Sep 18 17:59:52.608389 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:52.608365    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5d67c7fcff-6t9rk"]
Sep 18 17:59:52.612086 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:52.612063    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5d67c7fcff-6t9rk"]
Sep 18 17:59:53.861090 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:53.861056    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80b4f894-6ff7-41c1-b7d6-7d92ea276873" path="/var/lib/kubelet/pods/80b4f894-6ff7-41c1-b7d6-7d92ea276873/volumes"
Sep 18 17:59:55.762241 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:55.762215    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-5kjlz_75f2a1f7-95a7-462e-9ea7-23c51dd29599/console-operator/1.log"
Sep 18 17:59:55.767262 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:55.767243    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-5kjlz_75f2a1f7-95a7-462e-9ea7-23c51dd29599/console-operator/1.log"
Sep 18 17:59:55.768904 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:55.768883    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkcgk_c5d026b7-af6b-4899-85f4-6554c0f7cd52/ovn-acl-logging/0.log"
Sep 18 17:59:55.773509 ip-10-0-140-54 kubenswrapper[2561]: I0918 17:59:55.773494    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkcgk_c5d026b7-af6b-4899-85f4-6554c0f7cd52/ovn-acl-logging/0.log"
Sep 18 18:00:11.173314 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:11.173283    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:00:12.683237 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:12.683201    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:00:12.802389 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:12.802359    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:00:12.802744 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:12.802725    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7dddcbdd5b-rvczn"]
Sep 18 18:00:12.803231 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:12.803208    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80b4f894-6ff7-41c1-b7d6-7d92ea276873" containerName="manager"
Sep 18 18:00:12.803231 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:12.803227    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="80b4f894-6ff7-41c1-b7d6-7d92ea276873" containerName="manager"
Sep 18 18:00:12.803407 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:12.803324    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="80b4f894-6ff7-41c1-b7d6-7d92ea276873" containerName="manager"
Sep 18 18:00:12.809145 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:12.809129    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7dddcbdd5b-rvczn"
Sep 18 18:00:12.811503 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:12.811484    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 18 18:00:12.811593 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:12.811554    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-tlsbx\""
Sep 18 18:00:12.811783 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:12.811770    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 18 18:00:12.830390 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:12.830352    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7dddcbdd5b-rvczn"]
Sep 18 18:00:12.888557 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:12.887938    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4487ef5-ff26-41ba-9991-b52904cd0910-maas-api-tls\") pod \"maas-api-7dddcbdd5b-rvczn\" (UID: \"f4487ef5-ff26-41ba-9991-b52904cd0910\") " pod="odh-ai-gateway-infra/maas-api-7dddcbdd5b-rvczn"
Sep 18 18:00:12.888557 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:12.888026    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nt52\" (UniqueName: \"kubernetes.io/projected/f4487ef5-ff26-41ba-9991-b52904cd0910-kube-api-access-5nt52\") pod \"maas-api-7dddcbdd5b-rvczn\" (UID: \"f4487ef5-ff26-41ba-9991-b52904cd0910\") " pod="odh-ai-gateway-infra/maas-api-7dddcbdd5b-rvczn"
Sep 18 18:00:12.888557 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:12.888081    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4487ef5-ff26-41ba-9991-b52904cd0910-metrics-certs\") pod \"maas-api-7dddcbdd5b-rvczn\" (UID: \"f4487ef5-ff26-41ba-9991-b52904cd0910\") " pod="odh-ai-gateway-infra/maas-api-7dddcbdd5b-rvczn"
Sep 18 18:00:12.888557 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:12.888168    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4487ef5-ff26-41ba-9991-b52904cd0910-openshift-service-ca\") pod \"maas-api-7dddcbdd5b-rvczn\" (UID: \"f4487ef5-ff26-41ba-9991-b52904cd0910\") " pod="odh-ai-gateway-infra/maas-api-7dddcbdd5b-rvczn"
Sep 18 18:00:12.989157 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:12.989051    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4487ef5-ff26-41ba-9991-b52904cd0910-metrics-certs\") pod \"maas-api-7dddcbdd5b-rvczn\" (UID: \"f4487ef5-ff26-41ba-9991-b52904cd0910\") " pod="odh-ai-gateway-infra/maas-api-7dddcbdd5b-rvczn"
Sep 18 18:00:12.989157 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:12.989099    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4487ef5-ff26-41ba-9991-b52904cd0910-openshift-service-ca\") pod \"maas-api-7dddcbdd5b-rvczn\" (UID: \"f4487ef5-ff26-41ba-9991-b52904cd0910\") " pod="odh-ai-gateway-infra/maas-api-7dddcbdd5b-rvczn"
Sep 18 18:00:12.989377 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:12.989195    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4487ef5-ff26-41ba-9991-b52904cd0910-maas-api-tls\") pod \"maas-api-7dddcbdd5b-rvczn\" (UID: \"f4487ef5-ff26-41ba-9991-b52904cd0910\") " pod="odh-ai-gateway-infra/maas-api-7dddcbdd5b-rvczn"
Sep 18 18:00:12.989377 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:12.989226    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nt52\" (UniqueName: \"kubernetes.io/projected/f4487ef5-ff26-41ba-9991-b52904cd0910-kube-api-access-5nt52\") pod \"maas-api-7dddcbdd5b-rvczn\" (UID: \"f4487ef5-ff26-41ba-9991-b52904cd0910\") " pod="odh-ai-gateway-infra/maas-api-7dddcbdd5b-rvczn"
Sep 18 18:00:12.989377 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:00:12.989287    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found
Sep 18 18:00:12.989377 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:00:12.989356    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4487ef5-ff26-41ba-9991-b52904cd0910-maas-api-tls podName:f4487ef5-ff26-41ba-9991-b52904cd0910 nodeName:}" failed. No retries permitted until 2026-09-18 18:00:13.489338438 +0000 UTC m=+618.116470052 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f4487ef5-ff26-41ba-9991-b52904cd0910-maas-api-tls") pod "maas-api-7dddcbdd5b-rvczn" (UID: "f4487ef5-ff26-41ba-9991-b52904cd0910") : secret "maas-api-serving-cert" not found
Sep 18 18:00:12.989855 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:12.989830    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4487ef5-ff26-41ba-9991-b52904cd0910-openshift-service-ca\") pod \"maas-api-7dddcbdd5b-rvczn\" (UID: \"f4487ef5-ff26-41ba-9991-b52904cd0910\") " pod="odh-ai-gateway-infra/maas-api-7dddcbdd5b-rvczn"
Sep 18 18:00:12.991598 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:12.991578    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4487ef5-ff26-41ba-9991-b52904cd0910-metrics-certs\") pod \"maas-api-7dddcbdd5b-rvczn\" (UID: \"f4487ef5-ff26-41ba-9991-b52904cd0910\") " pod="odh-ai-gateway-infra/maas-api-7dddcbdd5b-rvczn"
Sep 18 18:00:13.005635 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:13.005605    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nt52\" (UniqueName: \"kubernetes.io/projected/f4487ef5-ff26-41ba-9991-b52904cd0910-kube-api-access-5nt52\") pod \"maas-api-7dddcbdd5b-rvczn\" (UID: \"f4487ef5-ff26-41ba-9991-b52904cd0910\") " pod="odh-ai-gateway-infra/maas-api-7dddcbdd5b-rvczn"
Sep 18 18:00:13.074031 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:13.074002    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:00:13.495460 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:13.495422    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4487ef5-ff26-41ba-9991-b52904cd0910-maas-api-tls\") pod \"maas-api-7dddcbdd5b-rvczn\" (UID: \"f4487ef5-ff26-41ba-9991-b52904cd0910\") " pod="odh-ai-gateway-infra/maas-api-7dddcbdd5b-rvczn"
Sep 18 18:00:13.498214 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:13.498184    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4487ef5-ff26-41ba-9991-b52904cd0910-maas-api-tls\") pod \"maas-api-7dddcbdd5b-rvczn\" (UID: \"f4487ef5-ff26-41ba-9991-b52904cd0910\") " pod="odh-ai-gateway-infra/maas-api-7dddcbdd5b-rvczn"
Sep 18 18:00:13.595163 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:13.595109    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:00:13.701289 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:13.701257    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:00:13.719332 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:13.719303    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7dddcbdd5b-rvczn"
Sep 18 18:00:13.837732 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:13.837696    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-fp2fb"]
Sep 18 18:00:13.841667 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:13.841639    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fp2fb"
Sep 18 18:00:13.847378 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:13.847350    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-ggfc9\""
Sep 18 18:00:13.847526 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:13.847363    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 18 18:00:13.877303 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:13.877272    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-fp2fb"]
Sep 18 18:00:13.958595 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:13.958558    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-j77pl"]
Sep 18 18:00:13.961728 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:13.961708    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-j77pl"
Sep 18 18:00:14.000702 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:14.000456    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcrkf\" (UniqueName: \"kubernetes.io/projected/5d9b729f-107b-4ab4-8516-42a1731679a5-kube-api-access-jcrkf\") pod \"payload-pre-processing-59fbc47bf8-fp2fb\" (UID: \"5d9b729f-107b-4ab4-8516-42a1731679a5\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fp2fb"
Sep 18 18:00:14.000702 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:14.000591    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d9b729f-107b-4ab4-8516-42a1731679a5-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-fp2fb\" (UID: \"5d9b729f-107b-4ab4-8516-42a1731679a5\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fp2fb"
Sep 18 18:00:14.007046 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:14.007015    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-j77pl"]
Sep 18 18:00:14.028651 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:14.028542    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7dddcbdd5b-rvczn"]
Sep 18 18:00:14.036986 ip-10-0-140-54 kubenswrapper[2561]: W0918 18:00:14.036956    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4487ef5_ff26_41ba_9991_b52904cd0910.slice/crio-7333b45ea626b3f322ecd54a670af0b1ad15880686bf278dbaba0f8123ddc8c8 WatchSource:0}: Error finding container 7333b45ea626b3f322ecd54a670af0b1ad15880686bf278dbaba0f8123ddc8c8: Status 404 returned error can't find the container with id 7333b45ea626b3f322ecd54a670af0b1ad15880686bf278dbaba0f8123ddc8c8
Sep 18 18:00:14.101899 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:14.101863    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcrkf\" (UniqueName: \"kubernetes.io/projected/5d9b729f-107b-4ab4-8516-42a1731679a5-kube-api-access-jcrkf\") pod \"payload-pre-processing-59fbc47bf8-fp2fb\" (UID: \"5d9b729f-107b-4ab4-8516-42a1731679a5\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fp2fb"
Sep 18 18:00:14.102061 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:14.102021    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1e487ec-d2f3-45a0-9b0c-0a14e89ea981-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-j77pl\" (UID: \"d1e487ec-d2f3-45a0-9b0c-0a14e89ea981\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-j77pl"
Sep 18 18:00:14.102061 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:14.102048    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd4jv\" (UniqueName: \"kubernetes.io/projected/d1e487ec-d2f3-45a0-9b0c-0a14e89ea981-kube-api-access-qd4jv\") pod \"payload-processing-5b54b7fbb7-j77pl\" (UID: \"d1e487ec-d2f3-45a0-9b0c-0a14e89ea981\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-j77pl"
Sep 18 18:00:14.102171 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:14.102077    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d9b729f-107b-4ab4-8516-42a1731679a5-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-fp2fb\" (UID: \"5d9b729f-107b-4ab4-8516-42a1731679a5\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fp2fb"
Sep 18 18:00:14.102780 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:14.102760    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d9b729f-107b-4ab4-8516-42a1731679a5-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-fp2fb\" (UID: \"5d9b729f-107b-4ab4-8516-42a1731679a5\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fp2fb"
Sep 18 18:00:14.111324 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:14.111296    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcrkf\" (UniqueName: \"kubernetes.io/projected/5d9b729f-107b-4ab4-8516-42a1731679a5-kube-api-access-jcrkf\") pod \"payload-pre-processing-59fbc47bf8-fp2fb\" (UID: \"5d9b729f-107b-4ab4-8516-42a1731679a5\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fp2fb"
Sep 18 18:00:14.152698 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:14.152664    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fp2fb"
Sep 18 18:00:14.202900 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:14.202864    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1e487ec-d2f3-45a0-9b0c-0a14e89ea981-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-j77pl\" (UID: \"d1e487ec-d2f3-45a0-9b0c-0a14e89ea981\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-j77pl"
Sep 18 18:00:14.203077 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:14.202913    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd4jv\" (UniqueName: \"kubernetes.io/projected/d1e487ec-d2f3-45a0-9b0c-0a14e89ea981-kube-api-access-qd4jv\") pod \"payload-processing-5b54b7fbb7-j77pl\" (UID: \"d1e487ec-d2f3-45a0-9b0c-0a14e89ea981\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-j77pl"
Sep 18 18:00:14.203645 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:14.203621    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1e487ec-d2f3-45a0-9b0c-0a14e89ea981-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-j77pl\" (UID: \"d1e487ec-d2f3-45a0-9b0c-0a14e89ea981\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-j77pl"
Sep 18 18:00:14.212514 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:14.212402    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd4jv\" (UniqueName: \"kubernetes.io/projected/d1e487ec-d2f3-45a0-9b0c-0a14e89ea981-kube-api-access-qd4jv\") pod \"payload-processing-5b54b7fbb7-j77pl\" (UID: \"d1e487ec-d2f3-45a0-9b0c-0a14e89ea981\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-j77pl"
Sep 18 18:00:14.273300 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:14.273263    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-j77pl"
Sep 18 18:00:14.434250 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:14.434226    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-fp2fb"]
Sep 18 18:00:14.436844 ip-10-0-140-54 kubenswrapper[2561]: W0918 18:00:14.436811    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d9b729f_107b_4ab4_8516_42a1731679a5.slice/crio-8775e1445cc44c1fc9697e401f65523c157275c9fdba9570699fb81bc4b1403e WatchSource:0}: Error finding container 8775e1445cc44c1fc9697e401f65523c157275c9fdba9570699fb81bc4b1403e: Status 404 returned error can't find the container with id 8775e1445cc44c1fc9697e401f65523c157275c9fdba9570699fb81bc4b1403e
Sep 18 18:00:14.559811 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:14.559784    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-j77pl"]
Sep 18 18:00:14.561704 ip-10-0-140-54 kubenswrapper[2561]: W0918 18:00:14.561676    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1e487ec_d2f3_45a0_9b0c_0a14e89ea981.slice/crio-6976dc826c06dc140c8d957d531bf4026ff216c4116b8bf727200bdfff24db83 WatchSource:0}: Error finding container 6976dc826c06dc140c8d957d531bf4026ff216c4116b8bf727200bdfff24db83: Status 404 returned error can't find the container with id 6976dc826c06dc140c8d957d531bf4026ff216c4116b8bf727200bdfff24db83
Sep 18 18:00:14.686264 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:14.686226    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7dddcbdd5b-rvczn" event={"ID":"f4487ef5-ff26-41ba-9991-b52904cd0910","Type":"ContainerStarted","Data":"7333b45ea626b3f322ecd54a670af0b1ad15880686bf278dbaba0f8123ddc8c8"}
Sep 18 18:00:14.687557 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:14.687526    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-j77pl" event={"ID":"d1e487ec-d2f3-45a0-9b0c-0a14e89ea981","Type":"ContainerStarted","Data":"6976dc826c06dc140c8d957d531bf4026ff216c4116b8bf727200bdfff24db83"}
Sep 18 18:00:14.688707 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:14.688682    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fp2fb" event={"ID":"5d9b729f-107b-4ab4-8516-42a1731679a5","Type":"ContainerStarted","Data":"8775e1445cc44c1fc9697e401f65523c157275c9fdba9570699fb81bc4b1403e"}
Sep 18 18:00:18.714299 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:18.714194    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7dddcbdd5b-rvczn" event={"ID":"f4487ef5-ff26-41ba-9991-b52904cd0910","Type":"ContainerStarted","Data":"2a5a9f1eabd20fe67cd94dfac5ab2a0298fb6b6078128c2a004a4004f37efc55"}
Sep 18 18:00:18.714713 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:18.714337    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7dddcbdd5b-rvczn"
Sep 18 18:00:18.715884 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:18.715855    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-j77pl" event={"ID":"d1e487ec-d2f3-45a0-9b0c-0a14e89ea981","Type":"ContainerStarted","Data":"8ad718c3b45d2df4cb789179d78a122306f83eb8ccafb180c0d6dd2659e33924"}
Sep 18 18:00:18.716020 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:18.715902    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-j77pl"
Sep 18 18:00:18.717438 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:18.717383    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fp2fb" event={"ID":"5d9b729f-107b-4ab4-8516-42a1731679a5","Type":"ContainerStarted","Data":"b197bd2a44d5fe47d549ec91e0b381f322189e32305dd795e2e326d504fa0736"}
Sep 18 18:00:18.717438 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:18.717433    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fp2fb"
Sep 18 18:00:18.733836 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:18.733779    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7dddcbdd5b-rvczn" podStartSLOduration=2.992790646 podStartE2EDuration="6.733762737s" totalImagesPullingTime="3.740972091s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:00:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:00:18.731721254 +0000 UTC m=+623.358852898" watchObservedRunningTime="2026-09-18 18:00:18.733762737 +0000 UTC m=+623.360894356"
Sep 18 18:00:18.749473 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:18.749418    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-j77pl" podStartSLOduration=2.5277611479999997 podStartE2EDuration="5.749398662s" totalImagesPullingTime="3.221637514s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:00:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:00:18.74736928 +0000 UTC m=+623.374500905" watchObservedRunningTime="2026-09-18 18:00:18.749398662 +0000 UTC m=+623.376530282"
Sep 18 18:00:18.763666 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:18.763619    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fp2fb" podStartSLOduration=2.42294703 podStartE2EDuration="5.763603169s" totalImagesPullingTime="3.340656139s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:00:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:00:18.762751433 +0000 UTC m=+623.389883066" watchObservedRunningTime="2026-09-18 18:00:18.763603169 +0000 UTC m=+623.390734789"
Sep 18 18:00:24.727612 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:24.727581    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7dddcbdd5b-rvczn"
Sep 18 18:00:25.714138 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:25.714076    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:00:25.802084 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:25.802050    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:00:27.556720 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:27.556691    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:00:27.657439 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:27.657389    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:00:29.724338 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:29.724303    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-j77pl"
Sep 18 18:00:29.724809 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:29.724430    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fp2fb"
Sep 18 18:00:34.703874 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:34.703844    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:00:35.453424 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:35.453390    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:00:45.328872 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:45.328835    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-n26lb"]
Sep 18 18:00:45.333755 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:45.333734    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n26lb"
Sep 18 18:00:45.336392 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:45.336371    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 18 18:00:45.336525 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:45.336375    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 18 18:00:45.337077 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:45.337026    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-tq6dk\""
Sep 18 18:00:45.337238 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:45.337223    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 18 18:00:45.345306 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:45.345286    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-n26lb"]
Sep 18 18:00:45.378133 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:45.378099    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b21f8cd0-20c7-4c0c-ae49-4bc752a38065-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-n26lb\" (UID: \"b21f8cd0-20c7-4c0c-ae49-4bc752a38065\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n26lb"
Sep 18 18:00:45.378268 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:45.378189    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b21f8cd0-20c7-4c0c-ae49-4bc752a38065-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-n26lb\" (UID: \"b21f8cd0-20c7-4c0c-ae49-4bc752a38065\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n26lb"
Sep 18 18:00:45.378268 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:45.378215    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b21f8cd0-20c7-4c0c-ae49-4bc752a38065-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-n26lb\" (UID: \"b21f8cd0-20c7-4c0c-ae49-4bc752a38065\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n26lb"
Sep 18 18:00:45.378268 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:45.378267    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b21f8cd0-20c7-4c0c-ae49-4bc752a38065-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-n26lb\" (UID: \"b21f8cd0-20c7-4c0c-ae49-4bc752a38065\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n26lb"
Sep 18 18:00:45.378394 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:45.378286    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fknkf\" (UniqueName: \"kubernetes.io/projected/b21f8cd0-20c7-4c0c-ae49-4bc752a38065-kube-api-access-fknkf\") pod \"e2e-distinct-simulated-kserve-59975d69f9-n26lb\" (UID: \"b21f8cd0-20c7-4c0c-ae49-4bc752a38065\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n26lb"
Sep 18 18:00:45.378394 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:45.378309    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b21f8cd0-20c7-4c0c-ae49-4bc752a38065-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-n26lb\" (UID: \"b21f8cd0-20c7-4c0c-ae49-4bc752a38065\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n26lb"
Sep 18 18:00:45.415908 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:45.415877    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:00:45.479309 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:45.479279    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b21f8cd0-20c7-4c0c-ae49-4bc752a38065-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-n26lb\" (UID: \"b21f8cd0-20c7-4c0c-ae49-4bc752a38065\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n26lb"
Sep 18 18:00:45.479309 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:45.479312    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b21f8cd0-20c7-4c0c-ae49-4bc752a38065-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-n26lb\" (UID: \"b21f8cd0-20c7-4c0c-ae49-4bc752a38065\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n26lb"
Sep 18 18:00:45.479478 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:45.479347    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b21f8cd0-20c7-4c0c-ae49-4bc752a38065-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-n26lb\" (UID: \"b21f8cd0-20c7-4c0c-ae49-4bc752a38065\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n26lb"
Sep 18 18:00:45.479478 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:45.479366    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fknkf\" (UniqueName: \"kubernetes.io/projected/b21f8cd0-20c7-4c0c-ae49-4bc752a38065-kube-api-access-fknkf\") pod \"e2e-distinct-simulated-kserve-59975d69f9-n26lb\" (UID: \"b21f8cd0-20c7-4c0c-ae49-4bc752a38065\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n26lb"
Sep 18 18:00:45.479478 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:45.479393    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b21f8cd0-20c7-4c0c-ae49-4bc752a38065-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-n26lb\" (UID: \"b21f8cd0-20c7-4c0c-ae49-4bc752a38065\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n26lb"
Sep 18 18:00:45.479478 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:45.479446    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b21f8cd0-20c7-4c0c-ae49-4bc752a38065-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-n26lb\" (UID: \"b21f8cd0-20c7-4c0c-ae49-4bc752a38065\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n26lb"
Sep 18 18:00:45.479755 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:45.479727    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b21f8cd0-20c7-4c0c-ae49-4bc752a38065-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-n26lb\" (UID: \"b21f8cd0-20c7-4c0c-ae49-4bc752a38065\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n26lb"
Sep 18 18:00:45.479869 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:45.479815    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b21f8cd0-20c7-4c0c-ae49-4bc752a38065-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-n26lb\" (UID: \"b21f8cd0-20c7-4c0c-ae49-4bc752a38065\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n26lb"
Sep 18 18:00:45.479979 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:45.479961    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b21f8cd0-20c7-4c0c-ae49-4bc752a38065-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-n26lb\" (UID: \"b21f8cd0-20c7-4c0c-ae49-4bc752a38065\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n26lb"
Sep 18 18:00:45.481787 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:45.481768    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b21f8cd0-20c7-4c0c-ae49-4bc752a38065-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-n26lb\" (UID: \"b21f8cd0-20c7-4c0c-ae49-4bc752a38065\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n26lb"
Sep 18 18:00:45.482094 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:45.482071    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b21f8cd0-20c7-4c0c-ae49-4bc752a38065-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-n26lb\" (UID: \"b21f8cd0-20c7-4c0c-ae49-4bc752a38065\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n26lb"
Sep 18 18:00:45.491664 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:45.491643    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fknkf\" (UniqueName: \"kubernetes.io/projected/b21f8cd0-20c7-4c0c-ae49-4bc752a38065-kube-api-access-fknkf\") pod \"e2e-distinct-simulated-kserve-59975d69f9-n26lb\" (UID: \"b21f8cd0-20c7-4c0c-ae49-4bc752a38065\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n26lb"
Sep 18 18:00:45.526469 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:45.526432    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:00:45.644053 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:45.643963    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n26lb"
Sep 18 18:00:45.928655 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:45.928630    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-n26lb"]
Sep 18 18:00:45.930213 ip-10-0-140-54 kubenswrapper[2561]: W0918 18:00:45.930176    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb21f8cd0_20c7_4c0c_ae49_4bc752a38065.slice/crio-16b4c6c63168349137dc0fb50afe8f4ed6375f8bdc5beeab04681629dad2b3f1 WatchSource:0}: Error finding container 16b4c6c63168349137dc0fb50afe8f4ed6375f8bdc5beeab04681629dad2b3f1: Status 404 returned error can't find the container with id 16b4c6c63168349137dc0fb50afe8f4ed6375f8bdc5beeab04681629dad2b3f1
Sep 18 18:00:46.848345 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:46.848304    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n26lb" event={"ID":"b21f8cd0-20c7-4c0c-ae49-4bc752a38065","Type":"ContainerStarted","Data":"16b4c6c63168349137dc0fb50afe8f4ed6375f8bdc5beeab04681629dad2b3f1"}
Sep 18 18:00:47.628313 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:47.628282    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:00:47.854636 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:47.854602    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n26lb" event={"ID":"b21f8cd0-20c7-4c0c-ae49-4bc752a38065","Type":"ContainerStarted","Data":"0b736af95a62716a5e18d89bb6ec0f75c011e039a986ee9c6243f37ff48ac48b"}
Sep 18 18:00:47.886060 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:47.885963    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n26lb" podStartSLOduration=1.7296916329999998 podStartE2EDuration="2.885929553s" totalImagesPullingTime="1.15623792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:00:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:00:47.885928447 +0000 UTC m=+652.513060066" watchObservedRunningTime="2026-09-18 18:00:47.885929553 +0000 UTC m=+652.513061173"
Sep 18 18:00:55.644499 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:55.644464    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n26lb"
Sep 18 18:00:55.644975 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:55.644739    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n26lb"
Sep 18 18:00:55.656907 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:55.656886    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n26lb"
Sep 18 18:00:55.900689 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:00:55.900659    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n26lb"
Sep 18 18:01:05.320578 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:01:05.320548    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:15.097318 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:01:15.097282    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:23.072742 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:01:23.072714    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:31.023675 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:01:31.023642    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:49.957054 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:01:49.956974    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-j4hwc"]
Sep 18 18:01:49.960710 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:01:49.960691    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-j4hwc"
Sep 18 18:01:49.963416 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:01:49.963392    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 18 18:01:49.980525 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:01:49.980502    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-j4hwc"]
Sep 18 18:01:50.062486 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:01:50.062451    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:01:50.075473 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:01:50.075451    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d091ce3f-fbef-456b-b91c-217133bb3a09-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-j4hwc\" (UID: \"d091ce3f-fbef-456b-b91c-217133bb3a09\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-j4hwc"
Sep 18 18:01:50.075621 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:01:50.075479    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86vlh\" (UniqueName: \"kubernetes.io/projected/d091ce3f-fbef-456b-b91c-217133bb3a09-kube-api-access-86vlh\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-j4hwc\" (UID: \"d091ce3f-fbef-456b-b91c-217133bb3a09\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-j4hwc"
Sep 18 18:01:50.075621 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:01:50.075505    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d091ce3f-fbef-456b-b91c-217133bb3a09-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-j4hwc\" (UID: \"d091ce3f-fbef-456b-b91c-217133bb3a09\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-j4hwc"
Sep 18 18:01:50.075621 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:01:50.075563    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d091ce3f-fbef-456b-b91c-217133bb3a09-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-j4hwc\" (UID: \"d091ce3f-fbef-456b-b91c-217133bb3a09\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-j4hwc"
Sep 18 18:01:50.075621 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:01:50.075600    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d091ce3f-fbef-456b-b91c-217133bb3a09-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-j4hwc\" (UID: \"d091ce3f-fbef-456b-b91c-217133bb3a09\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-j4hwc"
Sep 18 18:01:50.075621 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:01:50.075619    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d091ce3f-fbef-456b-b91c-217133bb3a09-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-j4hwc\" (UID: \"d091ce3f-fbef-456b-b91c-217133bb3a09\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-j4hwc"
Sep 18 18:01:50.177072 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:01:50.177043    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d091ce3f-fbef-456b-b91c-217133bb3a09-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-j4hwc\" (UID: \"d091ce3f-fbef-456b-b91c-217133bb3a09\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-j4hwc"
Sep 18 18:01:50.177072 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:01:50.177075    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d091ce3f-fbef-456b-b91c-217133bb3a09-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-j4hwc\" (UID: \"d091ce3f-fbef-456b-b91c-217133bb3a09\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-j4hwc"
Sep 18 18:01:50.177340 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:01:50.177133    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d091ce3f-fbef-456b-b91c-217133bb3a09-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-j4hwc\" (UID: \"d091ce3f-fbef-456b-b91c-217133bb3a09\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-j4hwc"
Sep 18 18:01:50.177340 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:01:50.177156    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86vlh\" (UniqueName: \"kubernetes.io/projected/d091ce3f-fbef-456b-b91c-217133bb3a09-kube-api-access-86vlh\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-j4hwc\" (UID: \"d091ce3f-fbef-456b-b91c-217133bb3a09\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-j4hwc"
Sep 18 18:01:50.177340 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:01:50.177266    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d091ce3f-fbef-456b-b91c-217133bb3a09-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-j4hwc\" (UID: \"d091ce3f-fbef-456b-b91c-217133bb3a09\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-j4hwc"
Sep 18 18:01:50.177505 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:01:50.177344    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d091ce3f-fbef-456b-b91c-217133bb3a09-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-j4hwc\" (UID: \"d091ce3f-fbef-456b-b91c-217133bb3a09\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-j4hwc"
Sep 18 18:01:50.177561 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:01:50.177515    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d091ce3f-fbef-456b-b91c-217133bb3a09-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-j4hwc\" (UID: \"d091ce3f-fbef-456b-b91c-217133bb3a09\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-j4hwc"
Sep 18 18:01:50.177561 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:01:50.177525    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d091ce3f-fbef-456b-b91c-217133bb3a09-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-j4hwc\" (UID: \"d091ce3f-fbef-456b-b91c-217133bb3a09\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-j4hwc"
Sep 18 18:01:50.177664 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:01:50.177604    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d091ce3f-fbef-456b-b91c-217133bb3a09-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-j4hwc\" (UID: \"d091ce3f-fbef-456b-b91c-217133bb3a09\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-j4hwc"
Sep 18 18:01:50.179356 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:01:50.179330    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d091ce3f-fbef-456b-b91c-217133bb3a09-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-j4hwc\" (UID: \"d091ce3f-fbef-456b-b91c-217133bb3a09\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-j4hwc"
Sep 18 18:01:50.179833 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:01:50.179812    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d091ce3f-fbef-456b-b91c-217133bb3a09-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-j4hwc\" (UID: \"d091ce3f-fbef-456b-b91c-217133bb3a09\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-j4hwc"
Sep 18 18:01:50.191435 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:01:50.191415    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86vlh\" (UniqueName: \"kubernetes.io/projected/d091ce3f-fbef-456b-b91c-217133bb3a09-kube-api-access-86vlh\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-j4hwc\" (UID: \"d091ce3f-fbef-456b-b91c-217133bb3a09\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-j4hwc"
Sep 18 18:01:50.270031 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:01:50.269948    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-j4hwc"
Sep 18 18:01:50.542350 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:01:50.542315    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-j4hwc"]
Sep 18 18:01:50.544567 ip-10-0-140-54 kubenswrapper[2561]: W0918 18:01:50.544541    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd091ce3f_fbef_456b_b91c_217133bb3a09.slice/crio-bbff90c7c5294ca6611776a9a8d85122ab85bb350d60e950aebed893eaf74137 WatchSource:0}: Error finding container bbff90c7c5294ca6611776a9a8d85122ab85bb350d60e950aebed893eaf74137: Status 404 returned error can't find the container with id bbff90c7c5294ca6611776a9a8d85122ab85bb350d60e950aebed893eaf74137
Sep 18 18:01:50.545936 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:01:50.545920    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 18:01:51.150450 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:01:51.150412    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-j4hwc" event={"ID":"d091ce3f-fbef-456b-b91c-217133bb3a09","Type":"ContainerStarted","Data":"8c69c0417334f711bf68846651cf846b00b140de9e7ccd85229a83efcbace4f7"}
Sep 18 18:01:51.150818 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:01:51.150460    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-j4hwc" event={"ID":"d091ce3f-fbef-456b-b91c-217133bb3a09","Type":"ContainerStarted","Data":"bbff90c7c5294ca6611776a9a8d85122ab85bb350d60e950aebed893eaf74137"}
Sep 18 18:01:51.173577 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:01:51.173529    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-j4hwc" podStartSLOduration=1.975628146 podStartE2EDuration="2.173512907s" totalImagesPullingTime="197.884761ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:01:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:01:51.168510091 +0000 UTC m=+715.795641710" watchObservedRunningTime="2026-09-18 18:01:51.173512907 +0000 UTC m=+715.800644542"
Sep 18 18:01:55.213500 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:01:55.213470    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:02:00.270535 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:02:00.270481    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-j4hwc"
Sep 18 18:02:00.271029 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:02:00.270552    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-j4hwc"
Sep 18 18:02:00.283237 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:02:00.283203    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-j4hwc"
Sep 18 18:02:01.212741 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:02:01.212710    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-j4hwc"
Sep 18 18:02:02.921163 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:02:02.921099    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:02:15.429722 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:02:15.429692    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:05.425696 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:05.425656    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:05.695840 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:05.695729    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:06.913362 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:06.913330    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:07.166246 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:07.166158    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-77544959bf-j8vjd"]
Sep 18 18:03:07.166416 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:07.166357    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-77544959bf-j8vjd" podUID="ff96b8bb-4fbd-4a53-af89-bf54eabc36c0" containerName="authorino" containerID="cri-o://d18e82cfb4f4f2ed1d284c16274de792cb58041d47b66ff57bb5b332c45d7030" gracePeriod=30
Sep 18 18:03:07.215045 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:07.215020    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:07.415025 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:07.414999    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-77544959bf-j8vjd"
Sep 18 18:03:07.520107 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:07.520020    2561 generic.go:356] "Generic (PLEG): container finished" podID="ff96b8bb-4fbd-4a53-af89-bf54eabc36c0" containerID="d18e82cfb4f4f2ed1d284c16274de792cb58041d47b66ff57bb5b332c45d7030" exitCode=0
Sep 18 18:03:07.520107 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:07.520063    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-77544959bf-j8vjd"
Sep 18 18:03:07.520107 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:07.520091    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-77544959bf-j8vjd" event={"ID":"ff96b8bb-4fbd-4a53-af89-bf54eabc36c0","Type":"ContainerDied","Data":"d18e82cfb4f4f2ed1d284c16274de792cb58041d47b66ff57bb5b332c45d7030"}
Sep 18 18:03:07.520368 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:07.520138    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-77544959bf-j8vjd" event={"ID":"ff96b8bb-4fbd-4a53-af89-bf54eabc36c0","Type":"ContainerDied","Data":"d3a0285db94b27bf89a38f8dc73b16995b25fa4fa2d31315c4a1b11b383e061d"}
Sep 18 18:03:07.520368 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:07.520160    2561 scope.go:117] "RemoveContainer" containerID="d18e82cfb4f4f2ed1d284c16274de792cb58041d47b66ff57bb5b332c45d7030"
Sep 18 18:03:07.528453 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:07.528437    2561 scope.go:117] "RemoveContainer" containerID="d18e82cfb4f4f2ed1d284c16274de792cb58041d47b66ff57bb5b332c45d7030"
Sep 18 18:03:07.528764 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:03:07.528745    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d18e82cfb4f4f2ed1d284c16274de792cb58041d47b66ff57bb5b332c45d7030\": container with ID starting with d18e82cfb4f4f2ed1d284c16274de792cb58041d47b66ff57bb5b332c45d7030 not found: ID does not exist" containerID="d18e82cfb4f4f2ed1d284c16274de792cb58041d47b66ff57bb5b332c45d7030"
Sep 18 18:03:07.528811 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:07.528773    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d18e82cfb4f4f2ed1d284c16274de792cb58041d47b66ff57bb5b332c45d7030"} err="failed to get container status \"d18e82cfb4f4f2ed1d284c16274de792cb58041d47b66ff57bb5b332c45d7030\": rpc error: code = NotFound desc = could not find container \"d18e82cfb4f4f2ed1d284c16274de792cb58041d47b66ff57bb5b332c45d7030\": container with ID starting with d18e82cfb4f4f2ed1d284c16274de792cb58041d47b66ff57bb5b332c45d7030 not found: ID does not exist"
Sep 18 18:03:07.569374 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:07.569345    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ff96b8bb-4fbd-4a53-af89-bf54eabc36c0-tls-cert\") pod \"ff96b8bb-4fbd-4a53-af89-bf54eabc36c0\" (UID: \"ff96b8bb-4fbd-4a53-af89-bf54eabc36c0\") "
Sep 18 18:03:07.569522 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:07.569414    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8qjpb\" (UniqueName: \"kubernetes.io/projected/ff96b8bb-4fbd-4a53-af89-bf54eabc36c0-kube-api-access-8qjpb\") pod \"ff96b8bb-4fbd-4a53-af89-bf54eabc36c0\" (UID: \"ff96b8bb-4fbd-4a53-af89-bf54eabc36c0\") "
Sep 18 18:03:07.571532 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:07.571503    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff96b8bb-4fbd-4a53-af89-bf54eabc36c0-kube-api-access-8qjpb" (OuterVolumeSpecName: "kube-api-access-8qjpb") pod "ff96b8bb-4fbd-4a53-af89-bf54eabc36c0" (UID: "ff96b8bb-4fbd-4a53-af89-bf54eabc36c0"). InnerVolumeSpecName "kube-api-access-8qjpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:03:07.579805 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:07.579777    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff96b8bb-4fbd-4a53-af89-bf54eabc36c0-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "ff96b8bb-4fbd-4a53-af89-bf54eabc36c0" (UID: "ff96b8bb-4fbd-4a53-af89-bf54eabc36c0"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:03:07.670929 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:07.670895    2561 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ff96b8bb-4fbd-4a53-af89-bf54eabc36c0-tls-cert\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:03:07.670929 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:07.670924    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qjpb\" (UniqueName: \"kubernetes.io/projected/ff96b8bb-4fbd-4a53-af89-bf54eabc36c0-kube-api-access-8qjpb\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:03:07.905494 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:07.905457    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-77544959bf-j8vjd"]
Sep 18 18:03:07.916819 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:07.916791    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-77544959bf-j8vjd"]
Sep 18 18:03:09.860870 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:09.860835    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff96b8bb-4fbd-4a53-af89-bf54eabc36c0" path="/var/lib/kubelet/pods/ff96b8bb-4fbd-4a53-af89-bf54eabc36c0/volumes"
Sep 18 18:03:24.100971 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:24.100864    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:24.270385 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:24.270353    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:24.739857 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:24.739825    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:24.990490 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:24.990409    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:25.235047 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:25.235010    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:25.378427 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:25.378345    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:25.710543 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:25.710509    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:25.913611 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:25.913577    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:26.263334 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:26.263301    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7611-d750ff"]
Sep 18 18:03:26.263906 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:26.263886    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff96b8bb-4fbd-4a53-af89-bf54eabc36c0" containerName="authorino"
Sep 18 18:03:26.263989 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:26.263909    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff96b8bb-4fbd-4a53-af89-bf54eabc36c0" containerName="authorino"
Sep 18 18:03:26.264044 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:26.263995    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff96b8bb-4fbd-4a53-af89-bf54eabc36c0" containerName="authorino"
Sep 18 18:03:26.267010 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:26.266989    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7611-d750ff"
Sep 18 18:03:26.280036 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:26.280012    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-fz228\""
Sep 18 18:03:26.283900 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:26.283879    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:26.296615 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:26.296591    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7611-d750ff"]
Sep 18 18:03:26.340846 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:26.340814    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwwkv\" (UniqueName: \"kubernetes.io/projected/d0e0fdad-98e6-4fbd-a3fc-17d2ce86e0e0-kube-api-access-qwwkv\") pod \"test-curl-7611-d750ff\" (UID: \"d0e0fdad-98e6-4fbd-a3fc-17d2ce86e0e0\") " pod="opendatahub/test-curl-7611-d750ff"
Sep 18 18:03:26.441708 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:26.441680    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwwkv\" (UniqueName: \"kubernetes.io/projected/d0e0fdad-98e6-4fbd-a3fc-17d2ce86e0e0-kube-api-access-qwwkv\") pod \"test-curl-7611-d750ff\" (UID: \"d0e0fdad-98e6-4fbd-a3fc-17d2ce86e0e0\") " pod="opendatahub/test-curl-7611-d750ff"
Sep 18 18:03:26.476248 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:26.476223    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwwkv\" (UniqueName: \"kubernetes.io/projected/d0e0fdad-98e6-4fbd-a3fc-17d2ce86e0e0-kube-api-access-qwwkv\") pod \"test-curl-7611-d750ff\" (UID: \"d0e0fdad-98e6-4fbd-a3fc-17d2ce86e0e0\") " pod="opendatahub/test-curl-7611-d750ff"
Sep 18 18:03:26.575925 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:26.575830    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7611-d750ff"
Sep 18 18:03:26.855970 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:26.855888    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7611-d750ff"]
Sep 18 18:03:26.859144 ip-10-0-140-54 kubenswrapper[2561]: W0918 18:03:26.859091    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0e0fdad_98e6_4fbd_a3fc_17d2ce86e0e0.slice/crio-89e0a51df758a33a7440e33086d814cd0916800dac540bad0eeb1d1e68929783 WatchSource:0}: Error finding container 89e0a51df758a33a7440e33086d814cd0916800dac540bad0eeb1d1e68929783: Status 404 returned error can't find the container with id 89e0a51df758a33a7440e33086d814cd0916800dac540bad0eeb1d1e68929783
Sep 18 18:03:27.627691 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:27.627641    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7611-d750ff" event={"ID":"d0e0fdad-98e6-4fbd-a3fc-17d2ce86e0e0","Type":"ContainerStarted","Data":"89e0a51df758a33a7440e33086d814cd0916800dac540bad0eeb1d1e68929783"}
Sep 18 18:03:28.133335 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:28.133296    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:28.330603 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:28.330563    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:28.909306 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:28.909282    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:29.173280 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:29.173246    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:29.391548 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:29.391507    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:29.640534 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:29.640449    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7611-d750ff" event={"ID":"d0e0fdad-98e6-4fbd-a3fc-17d2ce86e0e0","Type":"ContainerStarted","Data":"ea7f326259a81b04b222e18ae59d98c381aae63a497de74533e104976622e968"}
Sep 18 18:03:29.722755 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:29.722686    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7611-d750ff" podStartSLOduration=1.729006915 podStartE2EDuration="3.722665969s" totalImagesPullingTime="1.993659054s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:03:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:03:29.721547502 +0000 UTC m=+814.348679125" watchObservedRunningTime="2026-09-18 18:03:29.722665969 +0000 UTC m=+814.349797589"
Sep 18 18:03:30.525482 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:30.525452    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-502939-54d94b8c78-k54n7"]
Sep 18 18:03:30.529000 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:30.528982    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-502939-54d94b8c78-k54n7"
Sep 18 18:03:30.532518 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:30.532500    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-502939-serving-cert\""
Sep 18 18:03:30.558370 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:30.558346    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-502939-54d94b8c78-k54n7"]
Sep 18 18:03:30.579378 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:30.579350    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:30.581631 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:30.581607    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d4a5000-57d4-4443-b5fc-cafd2ef0ede3-maas-api-tls\") pod \"maas-api-e2e-inf-a-502939-54d94b8c78-k54n7\" (UID: \"9d4a5000-57d4-4443-b5fc-cafd2ef0ede3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-502939-54d94b8c78-k54n7"
Sep 18 18:03:30.581747 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:30.581637    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d4a5000-57d4-4443-b5fc-cafd2ef0ede3-metrics-certs\") pod \"maas-api-e2e-inf-a-502939-54d94b8c78-k54n7\" (UID: \"9d4a5000-57d4-4443-b5fc-cafd2ef0ede3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-502939-54d94b8c78-k54n7"
Sep 18 18:03:30.581747 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:30.581653    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4t8l\" (UniqueName: \"kubernetes.io/projected/9d4a5000-57d4-4443-b5fc-cafd2ef0ede3-kube-api-access-k4t8l\") pod \"maas-api-e2e-inf-a-502939-54d94b8c78-k54n7\" (UID: \"9d4a5000-57d4-4443-b5fc-cafd2ef0ede3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-502939-54d94b8c78-k54n7"
Sep 18 18:03:30.581868 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:30.581813    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d4a5000-57d4-4443-b5fc-cafd2ef0ede3-openshift-service-ca\") pod \"maas-api-e2e-inf-a-502939-54d94b8c78-k54n7\" (UID: \"9d4a5000-57d4-4443-b5fc-cafd2ef0ede3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-502939-54d94b8c78-k54n7"
Sep 18 18:03:30.613445 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:30.613372    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7611-d750ff"]
Sep 18 18:03:30.623208 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:30.623181    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7611-d750ff"]
Sep 18 18:03:30.647376 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:03:30.647335    2561 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7611-d750ff" container="test-curl-7611-d750ff"
Sep 18 18:03:30.682314 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:30.682285    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d4a5000-57d4-4443-b5fc-cafd2ef0ede3-maas-api-tls\") pod \"maas-api-e2e-inf-a-502939-54d94b8c78-k54n7\" (UID: \"9d4a5000-57d4-4443-b5fc-cafd2ef0ede3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-502939-54d94b8c78-k54n7"
Sep 18 18:03:30.682314 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:30.682315    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d4a5000-57d4-4443-b5fc-cafd2ef0ede3-metrics-certs\") pod \"maas-api-e2e-inf-a-502939-54d94b8c78-k54n7\" (UID: \"9d4a5000-57d4-4443-b5fc-cafd2ef0ede3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-502939-54d94b8c78-k54n7"
Sep 18 18:03:30.682494 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:30.682334    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4t8l\" (UniqueName: \"kubernetes.io/projected/9d4a5000-57d4-4443-b5fc-cafd2ef0ede3-kube-api-access-k4t8l\") pod \"maas-api-e2e-inf-a-502939-54d94b8c78-k54n7\" (UID: \"9d4a5000-57d4-4443-b5fc-cafd2ef0ede3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-502939-54d94b8c78-k54n7"
Sep 18 18:03:30.682494 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:30.682407    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d4a5000-57d4-4443-b5fc-cafd2ef0ede3-openshift-service-ca\") pod \"maas-api-e2e-inf-a-502939-54d94b8c78-k54n7\" (UID: \"9d4a5000-57d4-4443-b5fc-cafd2ef0ede3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-502939-54d94b8c78-k54n7"
Sep 18 18:03:30.682494 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:03:30.682443    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-502939-serving-cert: secret "maas-api-e2e-inf-a-502939-serving-cert" not found
Sep 18 18:03:30.682625 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:03:30.682541    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d4a5000-57d4-4443-b5fc-cafd2ef0ede3-maas-api-tls podName:9d4a5000-57d4-4443-b5fc-cafd2ef0ede3 nodeName:}" failed. No retries permitted until 2026-09-18 18:03:31.182521942 +0000 UTC m=+815.809653553 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9d4a5000-57d4-4443-b5fc-cafd2ef0ede3-maas-api-tls") pod "maas-api-e2e-inf-a-502939-54d94b8c78-k54n7" (UID: "9d4a5000-57d4-4443-b5fc-cafd2ef0ede3") : secret "maas-api-e2e-inf-a-502939-serving-cert" not found
Sep 18 18:03:30.683092 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:30.683066    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d4a5000-57d4-4443-b5fc-cafd2ef0ede3-openshift-service-ca\") pod \"maas-api-e2e-inf-a-502939-54d94b8c78-k54n7\" (UID: \"9d4a5000-57d4-4443-b5fc-cafd2ef0ede3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-502939-54d94b8c78-k54n7"
Sep 18 18:03:30.684632 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:30.684613    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d4a5000-57d4-4443-b5fc-cafd2ef0ede3-metrics-certs\") pod \"maas-api-e2e-inf-a-502939-54d94b8c78-k54n7\" (UID: \"9d4a5000-57d4-4443-b5fc-cafd2ef0ede3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-502939-54d94b8c78-k54n7"
Sep 18 18:03:30.704460 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:30.704436    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:30.705551 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:30.705528    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4t8l\" (UniqueName: \"kubernetes.io/projected/9d4a5000-57d4-4443-b5fc-cafd2ef0ede3-kube-api-access-k4t8l\") pod \"maas-api-e2e-inf-a-502939-54d94b8c78-k54n7\" (UID: \"9d4a5000-57d4-4443-b5fc-cafd2ef0ede3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-502939-54d94b8c78-k54n7"
Sep 18 18:03:30.755023 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:30.754993    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7611-f0c2a3"]
Sep 18 18:03:30.755410 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:30.755398    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0e0fdad-98e6-4fbd-a3fc-17d2ce86e0e0" containerName="test-curl-7611-d750ff"
Sep 18 18:03:30.755462 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:30.755412    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0e0fdad-98e6-4fbd-a3fc-17d2ce86e0e0" containerName="test-curl-7611-d750ff"
Sep 18 18:03:30.755502 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:30.755470    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0e0fdad-98e6-4fbd-a3fc-17d2ce86e0e0" containerName="test-curl-7611-d750ff"
Sep 18 18:03:30.758526 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:30.758510    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7611-f0c2a3"
Sep 18 18:03:30.788307 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:30.788278    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7611-f0c2a3"]
Sep 18 18:03:30.884378 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:30.884296    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnntc\" (UniqueName: \"kubernetes.io/projected/5d850953-c121-4229-939d-8156de224988-kube-api-access-qnntc\") pod \"test-curl-7611-f0c2a3\" (UID: \"5d850953-c121-4229-939d-8156de224988\") " pod="opendatahub/test-curl-7611-f0c2a3"
Sep 18 18:03:30.891897 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:30.891871    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:30.985154 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:30.985111    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnntc\" (UniqueName: \"kubernetes.io/projected/5d850953-c121-4229-939d-8156de224988-kube-api-access-qnntc\") pod \"test-curl-7611-f0c2a3\" (UID: \"5d850953-c121-4229-939d-8156de224988\") " pod="opendatahub/test-curl-7611-f0c2a3"
Sep 18 18:03:30.995943 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:30.995915    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnntc\" (UniqueName: \"kubernetes.io/projected/5d850953-c121-4229-939d-8156de224988-kube-api-access-qnntc\") pod \"test-curl-7611-f0c2a3\" (UID: \"5d850953-c121-4229-939d-8156de224988\") " pod="opendatahub/test-curl-7611-f0c2a3"
Sep 18 18:03:31.067065 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:31.067036    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7611-f0c2a3"
Sep 18 18:03:31.187052 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:31.187020    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d4a5000-57d4-4443-b5fc-cafd2ef0ede3-maas-api-tls\") pod \"maas-api-e2e-inf-a-502939-54d94b8c78-k54n7\" (UID: \"9d4a5000-57d4-4443-b5fc-cafd2ef0ede3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-502939-54d94b8c78-k54n7"
Sep 18 18:03:31.189662 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:31.189635    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d4a5000-57d4-4443-b5fc-cafd2ef0ede3-maas-api-tls\") pod \"maas-api-e2e-inf-a-502939-54d94b8c78-k54n7\" (UID: \"9d4a5000-57d4-4443-b5fc-cafd2ef0ede3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-502939-54d94b8c78-k54n7"
Sep 18 18:03:31.363196 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:31.363168    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7611-f0c2a3"]
Sep 18 18:03:31.363655 ip-10-0-140-54 kubenswrapper[2561]: W0918 18:03:31.363634    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5d850953_c121_4229_939d_8156de224988.slice/crio-9ebd9ca6e160b5ab1a779ccfcc4d7d6575ee434aeaafb7b01956fddc6cd0b0f9 WatchSource:0}: Error finding container 9ebd9ca6e160b5ab1a779ccfcc4d7d6575ee434aeaafb7b01956fddc6cd0b0f9: Status 404 returned error can't find the container with id 9ebd9ca6e160b5ab1a779ccfcc4d7d6575ee434aeaafb7b01956fddc6cd0b0f9
Sep 18 18:03:31.439109 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:31.439053    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-502939-54d94b8c78-k54n7"
Sep 18 18:03:31.627867 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:31.627835    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:31.654466 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:31.654427    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7611-f0c2a3" event={"ID":"5d850953-c121-4229-939d-8156de224988","Type":"ContainerStarted","Data":"ed87368025c7dd00dbe29d32f5c00cfd71192e5a63c9a7b0a557b5203719fbfc"}
Sep 18 18:03:31.654663 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:31.654477    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7611-f0c2a3" event={"ID":"5d850953-c121-4229-939d-8156de224988","Type":"ContainerStarted","Data":"9ebd9ca6e160b5ab1a779ccfcc4d7d6575ee434aeaafb7b01956fddc6cd0b0f9"}
Sep 18 18:03:31.654663 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:31.654474    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7611-d750ff" podUID="d0e0fdad-98e6-4fbd-a3fc-17d2ce86e0e0" containerName="test-curl-7611-d750ff" containerID="cri-o://ea7f326259a81b04b222e18ae59d98c381aae63a497de74533e104976622e968" gracePeriod=2
Sep 18 18:03:31.656783 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:03:31.656713    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7611-d750ff\" is forbidden: User \"system:node:ip-10-0-140-54.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-54.ec2.internal' and this object" podUID="d0e0fdad-98e6-4fbd-a3fc-17d2ce86e0e0" pod="opendatahub/test-curl-7611-d750ff"
Sep 18 18:03:31.694814 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:03:31.694719    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7611-d750ff\" is forbidden: User \"system:node:ip-10-0-140-54.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-54.ec2.internal' and this object" podUID="d0e0fdad-98e6-4fbd-a3fc-17d2ce86e0e0" pod="opendatahub/test-curl-7611-d750ff"
Sep 18 18:03:31.695655 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:31.695608    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7611-f0c2a3" podStartSLOduration=1.695592994 podStartE2EDuration="1.695592994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:03:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:03:31.691950244 +0000 UTC m=+816.319081866" watchObservedRunningTime="2026-09-18 18:03:31.695592994 +0000 UTC m=+816.322724614"
Sep 18 18:03:31.735675 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:31.735643    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-502939-54d94b8c78-k54n7"]
Sep 18 18:03:31.736868 ip-10-0-140-54 kubenswrapper[2561]: W0918 18:03:31.736839    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d4a5000_57d4_4443_b5fc_cafd2ef0ede3.slice/crio-77b883374f0eafc8b38de4a90eb1d5d2ef84f49b419534fa03ea3bcba56df370 WatchSource:0}: Error finding container 77b883374f0eafc8b38de4a90eb1d5d2ef84f49b419534fa03ea3bcba56df370: Status 404 returned error can't find the container with id 77b883374f0eafc8b38de4a90eb1d5d2ef84f49b419534fa03ea3bcba56df370
Sep 18 18:03:31.995672 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:31.995599    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:32.028764 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:32.028737    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-502939-56d954d58-zg4mx"]
Sep 18 18:03:32.034149 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:32.034107    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-502939-56d954d58-zg4mx"
Sep 18 18:03:32.037727 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:32.037708    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-502939-dockercfg-vljvp\""
Sep 18 18:03:32.037833 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:32.037764    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-502939\""
Sep 18 18:03:32.087737 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:32.087711    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-502939-56d954d58-zg4mx"]
Sep 18 18:03:32.095560 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:32.095538    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h58xf\" (UniqueName: \"kubernetes.io/projected/875d5495-8429-41d8-8625-e58b296b1886-kube-api-access-h58xf\") pod \"payload-pre-processing-e2e-inf-a-502939-56d954d58-zg4mx\" (UID: \"875d5495-8429-41d8-8625-e58b296b1886\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-502939-56d954d58-zg4mx"
Sep 18 18:03:32.095663 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:32.095622    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/875d5495-8429-41d8-8625-e58b296b1886-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-502939-56d954d58-zg4mx\" (UID: \"875d5495-8429-41d8-8625-e58b296b1886\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-502939-56d954d58-zg4mx"
Sep 18 18:03:32.196732 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:32.196706    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/875d5495-8429-41d8-8625-e58b296b1886-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-502939-56d954d58-zg4mx\" (UID: \"875d5495-8429-41d8-8625-e58b296b1886\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-502939-56d954d58-zg4mx"
Sep 18 18:03:32.196919 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:32.196775    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h58xf\" (UniqueName: \"kubernetes.io/projected/875d5495-8429-41d8-8625-e58b296b1886-kube-api-access-h58xf\") pod \"payload-pre-processing-e2e-inf-a-502939-56d954d58-zg4mx\" (UID: \"875d5495-8429-41d8-8625-e58b296b1886\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-502939-56d954d58-zg4mx"
Sep 18 18:03:32.197525 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:32.197501    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/875d5495-8429-41d8-8625-e58b296b1886-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-502939-56d954d58-zg4mx\" (UID: \"875d5495-8429-41d8-8625-e58b296b1886\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-502939-56d954d58-zg4mx"
Sep 18 18:03:32.226048 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:32.226017    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h58xf\" (UniqueName: \"kubernetes.io/projected/875d5495-8429-41d8-8625-e58b296b1886-kube-api-access-h58xf\") pod \"payload-pre-processing-e2e-inf-a-502939-56d954d58-zg4mx\" (UID: \"875d5495-8429-41d8-8625-e58b296b1886\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-502939-56d954d58-zg4mx"
Sep 18 18:03:32.272024 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:32.271964    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-502939-6f789b6949-kfsvn"]
Sep 18 18:03:32.275493 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:32.275471    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-502939-6f789b6949-kfsvn"
Sep 18 18:03:32.295920 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:32.295896    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-502939-6f789b6949-kfsvn"]
Sep 18 18:03:32.345276 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:32.345243    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-502939-56d954d58-zg4mx"
Sep 18 18:03:32.398379 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:32.398315    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1685ba89-9c41-45a5-8793-a9e7d8d9d1da-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-502939-6f789b6949-kfsvn\" (UID: \"1685ba89-9c41-45a5-8793-a9e7d8d9d1da\") " pod="openshift-ingress/payload-processing-e2e-inf-a-502939-6f789b6949-kfsvn"
Sep 18 18:03:32.398379 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:32.398358    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9zl8\" (UniqueName: \"kubernetes.io/projected/1685ba89-9c41-45a5-8793-a9e7d8d9d1da-kube-api-access-l9zl8\") pod \"payload-processing-e2e-inf-a-502939-6f789b6949-kfsvn\" (UID: \"1685ba89-9c41-45a5-8793-a9e7d8d9d1da\") " pod="openshift-ingress/payload-processing-e2e-inf-a-502939-6f789b6949-kfsvn"
Sep 18 18:03:32.499982 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:32.499955    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1685ba89-9c41-45a5-8793-a9e7d8d9d1da-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-502939-6f789b6949-kfsvn\" (UID: \"1685ba89-9c41-45a5-8793-a9e7d8d9d1da\") " pod="openshift-ingress/payload-processing-e2e-inf-a-502939-6f789b6949-kfsvn"
Sep 18 18:03:32.500130 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:32.499988    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9zl8\" (UniqueName: \"kubernetes.io/projected/1685ba89-9c41-45a5-8793-a9e7d8d9d1da-kube-api-access-l9zl8\") pod \"payload-processing-e2e-inf-a-502939-6f789b6949-kfsvn\" (UID: \"1685ba89-9c41-45a5-8793-a9e7d8d9d1da\") " pod="openshift-ingress/payload-processing-e2e-inf-a-502939-6f789b6949-kfsvn"
Sep 18 18:03:32.500682 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:32.500657    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1685ba89-9c41-45a5-8793-a9e7d8d9d1da-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-502939-6f789b6949-kfsvn\" (UID: \"1685ba89-9c41-45a5-8793-a9e7d8d9d1da\") " pod="openshift-ingress/payload-processing-e2e-inf-a-502939-6f789b6949-kfsvn"
Sep 18 18:03:32.516563 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:32.516535    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9zl8\" (UniqueName: \"kubernetes.io/projected/1685ba89-9c41-45a5-8793-a9e7d8d9d1da-kube-api-access-l9zl8\") pod \"payload-processing-e2e-inf-a-502939-6f789b6949-kfsvn\" (UID: \"1685ba89-9c41-45a5-8793-a9e7d8d9d1da\") " pod="openshift-ingress/payload-processing-e2e-inf-a-502939-6f789b6949-kfsvn"
Sep 18 18:03:32.587093 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:32.587011    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-502939-6f789b6949-kfsvn"
Sep 18 18:03:32.645378 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:32.645052    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-502939-56d954d58-zg4mx"]
Sep 18 18:03:32.648560 ip-10-0-140-54 kubenswrapper[2561]: W0918 18:03:32.648507    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod875d5495_8429_41d8_8625_e58b296b1886.slice/crio-22e02a6d64e2307a63c0cedd77fe1b80c5f8eb5466a2dca72f1770496f166d96 WatchSource:0}: Error finding container 22e02a6d64e2307a63c0cedd77fe1b80c5f8eb5466a2dca72f1770496f166d96: Status 404 returned error can't find the container with id 22e02a6d64e2307a63c0cedd77fe1b80c5f8eb5466a2dca72f1770496f166d96
Sep 18 18:03:32.661603 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:32.661576    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-502939-54d94b8c78-k54n7" event={"ID":"9d4a5000-57d4-4443-b5fc-cafd2ef0ede3","Type":"ContainerStarted","Data":"fa86a863390ee0e790994a567cf76e491082f54bf5c9dd0e389dca2dbfa37420"}
Sep 18 18:03:32.661714 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:32.661635    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-502939-54d94b8c78-k54n7" event={"ID":"9d4a5000-57d4-4443-b5fc-cafd2ef0ede3","Type":"ContainerStarted","Data":"77b883374f0eafc8b38de4a90eb1d5d2ef84f49b419534fa03ea3bcba56df370"}
Sep 18 18:03:32.662392 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:32.662367    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-502939-54d94b8c78-k54n7"
Sep 18 18:03:32.663257 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:32.663210    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-502939-56d954d58-zg4mx" event={"ID":"875d5495-8429-41d8-8625-e58b296b1886","Type":"ContainerStarted","Data":"22e02a6d64e2307a63c0cedd77fe1b80c5f8eb5466a2dca72f1770496f166d96"}
Sep 18 18:03:32.725081 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:32.725037    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-502939-54d94b8c78-k54n7" podStartSLOduration=2.080703198 podStartE2EDuration="2.725019909s" totalImagesPullingTime="644.316711ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:03:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:03:32.720083689 +0000 UTC m=+817.347215372" watchObservedRunningTime="2026-09-18 18:03:32.725019909 +0000 UTC m=+817.352151529"
Sep 18 18:03:32.800662 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:32.800630    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-03e512-7878686c45-mw6vc"]
Sep 18 18:03:32.804189 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:32.804168    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-03e512-7878686c45-mw6vc"
Sep 18 18:03:32.807281 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:32.807262    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-03e512-serving-cert\""
Sep 18 18:03:32.843648 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:32.843539    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-03e512-7878686c45-mw6vc"]
Sep 18 18:03:32.904176 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:32.904108    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a5f114b-5a1f-48bf-800a-f3d70cb2fd5b-maas-api-tls\") pod \"maas-api-e2e-worker-w4-03e512-7878686c45-mw6vc\" (UID: \"6a5f114b-5a1f-48bf-800a-f3d70cb2fd5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-03e512-7878686c45-mw6vc"
Sep 18 18:03:32.904362 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:32.904190    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a5f114b-5a1f-48bf-800a-f3d70cb2fd5b-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-03e512-7878686c45-mw6vc\" (UID: \"6a5f114b-5a1f-48bf-800a-f3d70cb2fd5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-03e512-7878686c45-mw6vc"
Sep 18 18:03:32.904362 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:32.904211    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4rmr\" (UniqueName: \"kubernetes.io/projected/6a5f114b-5a1f-48bf-800a-f3d70cb2fd5b-kube-api-access-c4rmr\") pod \"maas-api-e2e-worker-w4-03e512-7878686c45-mw6vc\" (UID: \"6a5f114b-5a1f-48bf-800a-f3d70cb2fd5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-03e512-7878686c45-mw6vc"
Sep 18 18:03:32.904362 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:32.904310    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a5f114b-5a1f-48bf-800a-f3d70cb2fd5b-metrics-certs\") pod \"maas-api-e2e-worker-w4-03e512-7878686c45-mw6vc\" (UID: \"6a5f114b-5a1f-48bf-800a-f3d70cb2fd5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-03e512-7878686c45-mw6vc"
Sep 18 18:03:32.907982 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:32.907957    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7611-f0c2a3"]
Sep 18 18:03:32.915389 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:32.915364    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-502939-6f789b6949-kfsvn"]
Sep 18 18:03:32.917746 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:32.917723    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7611-f0c2a3"]
Sep 18 18:03:32.919196 ip-10-0-140-54 kubenswrapper[2561]: W0918 18:03:32.919166    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1685ba89_9c41_45a5_8793_a9e7d8d9d1da.slice/crio-3c46ac5a34c115eb52015ad53a251c3abd31dcb7b5cf5d0e34aa2d8ea6b6b1ae WatchSource:0}: Error finding container 3c46ac5a34c115eb52015ad53a251c3abd31dcb7b5cf5d0e34aa2d8ea6b6b1ae: Status 404 returned error can't find the container with id 3c46ac5a34c115eb52015ad53a251c3abd31dcb7b5cf5d0e34aa2d8ea6b6b1ae
Sep 18 18:03:32.932951 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:32.932926    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:33.005231 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:33.005206    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a5f114b-5a1f-48bf-800a-f3d70cb2fd5b-maas-api-tls\") pod \"maas-api-e2e-worker-w4-03e512-7878686c45-mw6vc\" (UID: \"6a5f114b-5a1f-48bf-800a-f3d70cb2fd5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-03e512-7878686c45-mw6vc"
Sep 18 18:03:33.005365 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:33.005244    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a5f114b-5a1f-48bf-800a-f3d70cb2fd5b-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-03e512-7878686c45-mw6vc\" (UID: \"6a5f114b-5a1f-48bf-800a-f3d70cb2fd5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-03e512-7878686c45-mw6vc"
Sep 18 18:03:33.005365 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:33.005261    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4rmr\" (UniqueName: \"kubernetes.io/projected/6a5f114b-5a1f-48bf-800a-f3d70cb2fd5b-kube-api-access-c4rmr\") pod \"maas-api-e2e-worker-w4-03e512-7878686c45-mw6vc\" (UID: \"6a5f114b-5a1f-48bf-800a-f3d70cb2fd5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-03e512-7878686c45-mw6vc"
Sep 18 18:03:33.005365 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:33.005355    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a5f114b-5a1f-48bf-800a-f3d70cb2fd5b-metrics-certs\") pod \"maas-api-e2e-worker-w4-03e512-7878686c45-mw6vc\" (UID: \"6a5f114b-5a1f-48bf-800a-f3d70cb2fd5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-03e512-7878686c45-mw6vc"
Sep 18 18:03:33.005548 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:03:33.005366    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-03e512-serving-cert: secret "maas-api-e2e-worker-w4-03e512-serving-cert" not found
Sep 18 18:03:33.005548 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:03:33.005437    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a5f114b-5a1f-48bf-800a-f3d70cb2fd5b-maas-api-tls podName:6a5f114b-5a1f-48bf-800a-f3d70cb2fd5b nodeName:}" failed. No retries permitted until 2026-09-18 18:03:33.505412145 +0000 UTC m=+818.132543758 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6a5f114b-5a1f-48bf-800a-f3d70cb2fd5b-maas-api-tls") pod "maas-api-e2e-worker-w4-03e512-7878686c45-mw6vc" (UID: "6a5f114b-5a1f-48bf-800a-f3d70cb2fd5b") : secret "maas-api-e2e-worker-w4-03e512-serving-cert" not found
Sep 18 18:03:33.006028 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:33.006007    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a5f114b-5a1f-48bf-800a-f3d70cb2fd5b-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-03e512-7878686c45-mw6vc\" (UID: \"6a5f114b-5a1f-48bf-800a-f3d70cb2fd5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-03e512-7878686c45-mw6vc"
Sep 18 18:03:33.007782 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:33.007760    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a5f114b-5a1f-48bf-800a-f3d70cb2fd5b-metrics-certs\") pod \"maas-api-e2e-worker-w4-03e512-7878686c45-mw6vc\" (UID: \"6a5f114b-5a1f-48bf-800a-f3d70cb2fd5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-03e512-7878686c45-mw6vc"
Sep 18 18:03:33.031130 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:33.031083    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4rmr\" (UniqueName: \"kubernetes.io/projected/6a5f114b-5a1f-48bf-800a-f3d70cb2fd5b-kube-api-access-c4rmr\") pod \"maas-api-e2e-worker-w4-03e512-7878686c45-mw6vc\" (UID: \"6a5f114b-5a1f-48bf-800a-f3d70cb2fd5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-03e512-7878686c45-mw6vc"
Sep 18 18:03:33.080036 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:33.080002    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:33.497806 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:33.497777    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:33.511177 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:33.511149    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a5f114b-5a1f-48bf-800a-f3d70cb2fd5b-maas-api-tls\") pod \"maas-api-e2e-worker-w4-03e512-7878686c45-mw6vc\" (UID: \"6a5f114b-5a1f-48bf-800a-f3d70cb2fd5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-03e512-7878686c45-mw6vc"
Sep 18 18:03:33.511300 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:03:33.511286    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-03e512-serving-cert: secret "maas-api-e2e-worker-w4-03e512-serving-cert" not found
Sep 18 18:03:33.511353 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:03:33.511342    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a5f114b-5a1f-48bf-800a-f3d70cb2fd5b-maas-api-tls podName:6a5f114b-5a1f-48bf-800a-f3d70cb2fd5b nodeName:}" failed. No retries permitted until 2026-09-18 18:03:34.511328686 +0000 UTC m=+819.138460285 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6a5f114b-5a1f-48bf-800a-f3d70cb2fd5b-maas-api-tls") pod "maas-api-e2e-worker-w4-03e512-7878686c45-mw6vc" (UID: "6a5f114b-5a1f-48bf-800a-f3d70cb2fd5b") : secret "maas-api-e2e-worker-w4-03e512-serving-cert" not found
Sep 18 18:03:33.670973 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:33.670942    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-502939-56d954d58-zg4mx" event={"ID":"875d5495-8429-41d8-8625-e58b296b1886","Type":"ContainerStarted","Data":"6dc6510aafd2bb29b18dd668c1455e200d9f6b205b16a3e4e6efbdad157bd4c9"}
Sep 18 18:03:33.671418 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:33.671025    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-502939-56d954d58-zg4mx"
Sep 18 18:03:33.672862 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:33.672835    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-502939-6f789b6949-kfsvn" event={"ID":"1685ba89-9c41-45a5-8793-a9e7d8d9d1da","Type":"ContainerStarted","Data":"1e2a43de68b72462cb9b224089a5bab9b7c41930252ec56972d364c3cffa4317"}
Sep 18 18:03:33.672978 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:33.672857    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7611-f0c2a3" podUID="5d850953-c121-4229-939d-8156de224988" containerName="test-curl-7611-f0c2a3" containerID="cri-o://ed87368025c7dd00dbe29d32f5c00cfd71192e5a63c9a7b0a557b5203719fbfc" gracePeriod=2
Sep 18 18:03:33.672978 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:33.672885    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-502939-6f789b6949-kfsvn" event={"ID":"1685ba89-9c41-45a5-8793-a9e7d8d9d1da","Type":"ContainerStarted","Data":"3c46ac5a34c115eb52015ad53a251c3abd31dcb7b5cf5d0e34aa2d8ea6b6b1ae"}
Sep 18 18:03:33.735133 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:33.735065    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-502939-56d954d58-zg4mx" podStartSLOduration=2.735045067 podStartE2EDuration="2.735045067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:03:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:03:33.709077437 +0000 UTC m=+818.336209090" watchObservedRunningTime="2026-09-18 18:03:33.735045067 +0000 UTC m=+818.362176688"
Sep 18 18:03:33.735352 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:03:33.735327    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7611-f0c2a3\" is forbidden: User \"system:node:ip-10-0-140-54.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-54.ec2.internal' and this object" podUID="5d850953-c121-4229-939d-8156de224988" pod="opendatahub/test-curl-7611-f0c2a3"
Sep 18 18:03:33.735511 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:33.735484    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-502939-6f789b6949-kfsvn" podStartSLOduration=1.735474805 podStartE2EDuration="1.735474805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:03:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:03:33.732641391 +0000 UTC m=+818.359773013" watchObservedRunningTime="2026-09-18 18:03:33.735474805 +0000 UTC m=+818.362606466"
Sep 18 18:03:33.870080 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:33.870045    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7611-d750ff"
Sep 18 18:03:33.882939 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:33.882915    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:34.016064 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:34.016035    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qwwkv\" (UniqueName: \"kubernetes.io/projected/d0e0fdad-98e6-4fbd-a3fc-17d2ce86e0e0-kube-api-access-qwwkv\") pod \"d0e0fdad-98e6-4fbd-a3fc-17d2ce86e0e0\" (UID: \"d0e0fdad-98e6-4fbd-a3fc-17d2ce86e0e0\") "
Sep 18 18:03:34.017910 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:34.017880    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0e0fdad-98e6-4fbd-a3fc-17d2ce86e0e0-kube-api-access-qwwkv" (OuterVolumeSpecName: "kube-api-access-qwwkv") pod "d0e0fdad-98e6-4fbd-a3fc-17d2ce86e0e0" (UID: "d0e0fdad-98e6-4fbd-a3fc-17d2ce86e0e0"). InnerVolumeSpecName "kube-api-access-qwwkv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:03:34.098281 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:34.098187    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:34.116881 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:34.116861    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwwkv\" (UniqueName: \"kubernetes.io/projected/d0e0fdad-98e6-4fbd-a3fc-17d2ce86e0e0-kube-api-access-qwwkv\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:03:34.161968 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:34.161929    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-03e512-5bd997d694-gp4gc"]
Sep 18 18:03:34.162555 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:34.162540    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d850953-c121-4229-939d-8156de224988" containerName="test-curl-7611-f0c2a3"
Sep 18 18:03:34.162611 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:34.162559    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d850953-c121-4229-939d-8156de224988" containerName="test-curl-7611-f0c2a3"
Sep 18 18:03:34.162671 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:34.162661    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d850953-c121-4229-939d-8156de224988" containerName="test-curl-7611-f0c2a3"
Sep 18 18:03:34.166096 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:34.166078    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-03e512-5bd997d694-gp4gc"
Sep 18 18:03:34.168760 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:34.168743    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-03e512\""
Sep 18 18:03:34.204861 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:34.204837    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-03e512-5bd997d694-gp4gc"]
Sep 18 18:03:34.217498 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:34.217480    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgkxf\" (UniqueName: \"kubernetes.io/projected/33b62f5a-0f4d-4687-b901-05874f328161-kube-api-access-rgkxf\") pod \"payload-pre-processing-e2e-worker-w4-03e512-5bd997d694-gp4gc\" (UID: \"33b62f5a-0f4d-4687-b901-05874f328161\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-03e512-5bd997d694-gp4gc"
Sep 18 18:03:34.217596 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:34.217566    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33b62f5a-0f4d-4687-b901-05874f328161-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-03e512-5bd997d694-gp4gc\" (UID: \"33b62f5a-0f4d-4687-b901-05874f328161\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-03e512-5bd997d694-gp4gc"
Sep 18 18:03:34.250227 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:34.250208    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-03e512-85bdf7444c-vwgzw"]
Sep 18 18:03:34.253732 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:34.253716    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-03e512-85bdf7444c-vwgzw"
Sep 18 18:03:34.257658 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:34.257640    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-03e512-dockercfg-w8ks2\""
Sep 18 18:03:34.266692 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:34.266668    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-03e512-85bdf7444c-vwgzw"]
Sep 18 18:03:34.319003 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:34.318974    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4xcw\" (UniqueName: \"kubernetes.io/projected/9606a239-122e-4814-9048-ab09b406183f-kube-api-access-v4xcw\") pod \"payload-processing-e2e-worker-w4-03e512-85bdf7444c-vwgzw\" (UID: \"9606a239-122e-4814-9048-ab09b406183f\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-03e512-85bdf7444c-vwgzw"
Sep 18 18:03:34.319150 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:34.319011    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgkxf\" (UniqueName: \"kubernetes.io/projected/33b62f5a-0f4d-4687-b901-05874f328161-kube-api-access-rgkxf\") pod \"payload-pre-processing-e2e-worker-w4-03e512-5bd997d694-gp4gc\" (UID: \"33b62f5a-0f4d-4687-b901-05874f328161\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-03e512-5bd997d694-gp4gc"
Sep 18 18:03:34.319150 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:34.319073    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33b62f5a-0f4d-4687-b901-05874f328161-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-03e512-5bd997d694-gp4gc\" (UID: \"33b62f5a-0f4d-4687-b901-05874f328161\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-03e512-5bd997d694-gp4gc"
Sep 18 18:03:34.319305 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:34.319285    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9606a239-122e-4814-9048-ab09b406183f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-03e512-85bdf7444c-vwgzw\" (UID: \"9606a239-122e-4814-9048-ab09b406183f\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-03e512-85bdf7444c-vwgzw"
Sep 18 18:03:34.319824 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:34.319802    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33b62f5a-0f4d-4687-b901-05874f328161-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-03e512-5bd997d694-gp4gc\" (UID: \"33b62f5a-0f4d-4687-b901-05874f328161\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-03e512-5bd997d694-gp4gc"
Sep 18 18:03:34.340660 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:34.340637    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgkxf\" (UniqueName: \"kubernetes.io/projected/33b62f5a-0f4d-4687-b901-05874f328161-kube-api-access-rgkxf\") pod \"payload-pre-processing-e2e-worker-w4-03e512-5bd997d694-gp4gc\" (UID: \"33b62f5a-0f4d-4687-b901-05874f328161\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-03e512-5bd997d694-gp4gc"
Sep 18 18:03:34.420753 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:34.420721    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9606a239-122e-4814-9048-ab09b406183f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-03e512-85bdf7444c-vwgzw\" (UID: \"9606a239-122e-4814-9048-ab09b406183f\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-03e512-85bdf7444c-vwgzw"
Sep 18 18:03:34.420918 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:34.420762    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4xcw\" (UniqueName: \"kubernetes.io/projected/9606a239-122e-4814-9048-ab09b406183f-kube-api-access-v4xcw\") pod \"payload-processing-e2e-worker-w4-03e512-85bdf7444c-vwgzw\" (UID: \"9606a239-122e-4814-9048-ab09b406183f\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-03e512-85bdf7444c-vwgzw"
Sep 18 18:03:34.421700 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:34.421677    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9606a239-122e-4814-9048-ab09b406183f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-03e512-85bdf7444c-vwgzw\" (UID: \"9606a239-122e-4814-9048-ab09b406183f\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-03e512-85bdf7444c-vwgzw"
Sep 18 18:03:34.436376 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:34.436351    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4xcw\" (UniqueName: \"kubernetes.io/projected/9606a239-122e-4814-9048-ab09b406183f-kube-api-access-v4xcw\") pod \"payload-processing-e2e-worker-w4-03e512-85bdf7444c-vwgzw\" (UID: \"9606a239-122e-4814-9048-ab09b406183f\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-03e512-85bdf7444c-vwgzw"
Sep 18 18:03:34.476339 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:34.476309    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-03e512-5bd997d694-gp4gc"
Sep 18 18:03:34.521640 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:34.521606    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a5f114b-5a1f-48bf-800a-f3d70cb2fd5b-maas-api-tls\") pod \"maas-api-e2e-worker-w4-03e512-7878686c45-mw6vc\" (UID: \"6a5f114b-5a1f-48bf-800a-f3d70cb2fd5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-03e512-7878686c45-mw6vc"
Sep 18 18:03:34.524372 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:34.524337    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a5f114b-5a1f-48bf-800a-f3d70cb2fd5b-maas-api-tls\") pod \"maas-api-e2e-worker-w4-03e512-7878686c45-mw6vc\" (UID: \"6a5f114b-5a1f-48bf-800a-f3d70cb2fd5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-03e512-7878686c45-mw6vc"
Sep 18 18:03:34.563927 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:34.563895    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-03e512-85bdf7444c-vwgzw"
Sep 18 18:03:34.616364 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:34.616336    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-03e512-7878686c45-mw6vc"
Sep 18 18:03:34.683302 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:34.683256    2561 generic.go:356] "Generic (PLEG): container finished" podID="d0e0fdad-98e6-4fbd-a3fc-17d2ce86e0e0" containerID="ea7f326259a81b04b222e18ae59d98c381aae63a497de74533e104976622e968" exitCode=137
Sep 18 18:03:34.683676 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:34.683324    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7611-d750ff"
Sep 18 18:03:34.683676 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:34.683456    2561 scope.go:117] "RemoveContainer" containerID="ea7f326259a81b04b222e18ae59d98c381aae63a497de74533e104976622e968"
Sep 18 18:03:34.684773 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:34.684527    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-502939-6f789b6949-kfsvn"
Sep 18 18:03:34.699758 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:34.699728    2561 scope.go:117] "RemoveContainer" containerID="ea7f326259a81b04b222e18ae59d98c381aae63a497de74533e104976622e968"
Sep 18 18:03:34.700139 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:03:34.700077    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea7f326259a81b04b222e18ae59d98c381aae63a497de74533e104976622e968\": container with ID starting with ea7f326259a81b04b222e18ae59d98c381aae63a497de74533e104976622e968 not found: ID does not exist" containerID="ea7f326259a81b04b222e18ae59d98c381aae63a497de74533e104976622e968"
Sep 18 18:03:34.700229 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:34.700141    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea7f326259a81b04b222e18ae59d98c381aae63a497de74533e104976622e968"} err="failed to get container status \"ea7f326259a81b04b222e18ae59d98c381aae63a497de74533e104976622e968\": rpc error: code = NotFound desc = could not find container \"ea7f326259a81b04b222e18ae59d98c381aae63a497de74533e104976622e968\": container with ID starting with ea7f326259a81b04b222e18ae59d98c381aae63a497de74533e104976622e968 not found: ID does not exist"
Sep 18 18:03:34.702880 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:03:34.702857    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7611-d750ff\" is forbidden: User \"system:node:ip-10-0-140-54.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-54.ec2.internal' and this object" podUID="d0e0fdad-98e6-4fbd-a3fc-17d2ce86e0e0" pod="opendatahub/test-curl-7611-d750ff"
Sep 18 18:03:34.765580 ip-10-0-140-54 kubenswrapper[2561]: W0918 18:03:34.765468    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33b62f5a_0f4d_4687_b901_05874f328161.slice/crio-6427647716ec7ead848c9a2c5095426826ec276f9ee15056501f60d4b3f1a667 WatchSource:0}: Error finding container 6427647716ec7ead848c9a2c5095426826ec276f9ee15056501f60d4b3f1a667: Status 404 returned error can't find the container with id 6427647716ec7ead848c9a2c5095426826ec276f9ee15056501f60d4b3f1a667
Sep 18 18:03:34.768550 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:34.768382    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-03e512-5bd997d694-gp4gc"]
Sep 18 18:03:34.879178 ip-10-0-140-54 kubenswrapper[2561]: W0918 18:03:34.879149    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9606a239_122e_4814_9048_ab09b406183f.slice/crio-7189cc763e45ce09cdb1364c7d07af730b950fbb2107297cbcf435aef7987546 WatchSource:0}: Error finding container 7189cc763e45ce09cdb1364c7d07af730b950fbb2107297cbcf435aef7987546: Status 404 returned error can't find the container with id 7189cc763e45ce09cdb1364c7d07af730b950fbb2107297cbcf435aef7987546
Sep 18 18:03:34.888690 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:34.888663    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-03e512-85bdf7444c-vwgzw"]
Sep 18 18:03:34.901388 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:34.901339    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:34.940943 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:34.940917    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-03e512-7878686c45-mw6vc"]
Sep 18 18:03:34.943029 ip-10-0-140-54 kubenswrapper[2561]: W0918 18:03:34.943002    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a5f114b_5a1f_48bf_800a_f3d70cb2fd5b.slice/crio-43635579b90fcfbd10db01c5b21243ab09fc99eea9dd3fe5d259e0c4b7b22196 WatchSource:0}: Error finding container 43635579b90fcfbd10db01c5b21243ab09fc99eea9dd3fe5d259e0c4b7b22196: Status 404 returned error can't find the container with id 43635579b90fcfbd10db01c5b21243ab09fc99eea9dd3fe5d259e0c4b7b22196
Sep 18 18:03:35.148143 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:35.148072    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:35.424035 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:35.424006    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:35.692939 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:35.692909    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-03e512-85bdf7444c-vwgzw" event={"ID":"9606a239-122e-4814-9048-ab09b406183f","Type":"ContainerStarted","Data":"dccd6dd8ffe8845a0148f14408e5b413f3bd3399a7c7e028d9293ecc1eb9a2dc"}
Sep 18 18:03:35.693300 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:35.692963    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-03e512-85bdf7444c-vwgzw" event={"ID":"9606a239-122e-4814-9048-ab09b406183f","Type":"ContainerStarted","Data":"7189cc763e45ce09cdb1364c7d07af730b950fbb2107297cbcf435aef7987546"}
Sep 18 18:03:35.693300 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:35.693028    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-03e512-85bdf7444c-vwgzw"
Sep 18 18:03:35.694621 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:35.694592    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-03e512-5bd997d694-gp4gc" event={"ID":"33b62f5a-0f4d-4687-b901-05874f328161","Type":"ContainerStarted","Data":"86a11957ff99e383ff1f57570a531080a057392a18f2772cbdc70c13e32917ee"}
Sep 18 18:03:35.694732 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:35.694635    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-03e512-5bd997d694-gp4gc" event={"ID":"33b62f5a-0f4d-4687-b901-05874f328161","Type":"ContainerStarted","Data":"6427647716ec7ead848c9a2c5095426826ec276f9ee15056501f60d4b3f1a667"}
Sep 18 18:03:35.694732 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:35.694684    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-03e512-5bd997d694-gp4gc"
Sep 18 18:03:35.696217 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:35.696195    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-03e512-7878686c45-mw6vc" event={"ID":"6a5f114b-5a1f-48bf-800a-f3d70cb2fd5b","Type":"ContainerStarted","Data":"bb894e682b2e13a646d831c151c350404c07c28c9f4e78f8951b591a88ddd10e"}
Sep 18 18:03:35.696396 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:35.696385    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-03e512-7878686c45-mw6vc"
Sep 18 18:03:35.696436 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:35.696407    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-03e512-7878686c45-mw6vc" event={"ID":"6a5f114b-5a1f-48bf-800a-f3d70cb2fd5b","Type":"ContainerStarted","Data":"43635579b90fcfbd10db01c5b21243ab09fc99eea9dd3fe5d259e0c4b7b22196"}
Sep 18 18:03:35.723518 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:35.723476    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-03e512-85bdf7444c-vwgzw" podStartSLOduration=1.7234618560000001 podStartE2EDuration="1.723461856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:03:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:03:35.721835601 +0000 UTC m=+820.348967219" watchObservedRunningTime="2026-09-18 18:03:35.723461856 +0000 UTC m=+820.350593470"
Sep 18 18:03:35.723895 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:03:35.723866    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7611-d750ff\" is forbidden: User \"system:node:ip-10-0-140-54.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-54.ec2.internal' and this object" podUID="d0e0fdad-98e6-4fbd-a3fc-17d2ce86e0e0" pod="opendatahub/test-curl-7611-d750ff"
Sep 18 18:03:35.726881 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:03:35.726851    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7611-d750ff\" is forbidden: User \"system:node:ip-10-0-140-54.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-54.ec2.internal' and this object" podUID="d0e0fdad-98e6-4fbd-a3fc-17d2ce86e0e0" pod="opendatahub/test-curl-7611-d750ff"
Sep 18 18:03:35.759411 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:35.757756    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-03e512-5bd997d694-gp4gc" podStartSLOduration=1.7577328460000001 podStartE2EDuration="1.757732846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:03:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:03:35.752753176 +0000 UTC m=+820.379884797" watchObservedRunningTime="2026-09-18 18:03:35.757732846 +0000 UTC m=+820.384864467"
Sep 18 18:03:35.826656 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:35.826615    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:35.863641 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:03:35.863596    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7611-d750ff\" is forbidden: User \"system:node:ip-10-0-140-54.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-54.ec2.internal' and this object" podUID="d0e0fdad-98e6-4fbd-a3fc-17d2ce86e0e0" pod="opendatahub/test-curl-7611-d750ff"
Sep 18 18:03:35.863732 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:35.863638    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0e0fdad-98e6-4fbd-a3fc-17d2ce86e0e0" path="/var/lib/kubelet/pods/d0e0fdad-98e6-4fbd-a3fc-17d2ce86e0e0/volumes"
Sep 18 18:03:35.873913 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:35.873893    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7611-f0c2a3"
Sep 18 18:03:35.876852 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:03:35.876830    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7611-f0c2a3\" is forbidden: User \"system:node:ip-10-0-140-54.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-54.ec2.internal' and this object" podUID="5d850953-c121-4229-939d-8156de224988" pod="opendatahub/test-curl-7611-f0c2a3"
Sep 18 18:03:35.933580 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:35.933512    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qnntc\" (UniqueName: \"kubernetes.io/projected/5d850953-c121-4229-939d-8156de224988-kube-api-access-qnntc\") pod \"5d850953-c121-4229-939d-8156de224988\" (UID: \"5d850953-c121-4229-939d-8156de224988\") "
Sep 18 18:03:35.935457 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:35.935424    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d850953-c121-4229-939d-8156de224988-kube-api-access-qnntc" (OuterVolumeSpecName: "kube-api-access-qnntc") pod "5d850953-c121-4229-939d-8156de224988" (UID: "5d850953-c121-4229-939d-8156de224988"). InnerVolumeSpecName "kube-api-access-qnntc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:03:35.962486 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:35.962464    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:36.035034 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:36.035008    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnntc\" (UniqueName: \"kubernetes.io/projected/5d850953-c121-4229-939d-8156de224988-kube-api-access-qnntc\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:03:36.125931 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:36.125880    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-03e512-7878686c45-mw6vc" podStartSLOduration=3.688922454 podStartE2EDuration="4.125863615s" totalImagesPullingTime="436.941161ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:03:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:03:35.794860794 +0000 UTC m=+820.421992426" watchObservedRunningTime="2026-09-18 18:03:36.125863615 +0000 UTC m=+820.752995234"
Sep 18 18:03:36.126308 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:36.126289    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-609aba-58c9bb7b5c-w9mll"]
Sep 18 18:03:36.130003 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:36.129983    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-609aba-58c9bb7b5c-w9mll"
Sep 18 18:03:36.135174 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:03:36.135144    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7611-f0c2a3\" is forbidden: User \"system:node:ip-10-0-140-54.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-54.ec2.internal' and this object" podUID="5d850953-c121-4229-939d-8156de224988" pod="opendatahub/test-curl-7611-f0c2a3"
Sep 18 18:03:36.135174 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:36.135164    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-609aba-dockercfg-wc2tf\""
Sep 18 18:03:36.135346 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:36.135315    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-609aba\""
Sep 18 18:03:36.154914 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:36.154892    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-609aba-58c9bb7b5c-w9mll"]
Sep 18 18:03:36.236502 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:36.236427    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/222a170e-28a4-4f09-857e-20d2b245a957-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-609aba-58c9bb7b5c-w9mll\" (UID: \"222a170e-28a4-4f09-857e-20d2b245a957\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-609aba-58c9bb7b5c-w9mll"
Sep 18 18:03:36.236633 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:36.236561    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s25gc\" (UniqueName: \"kubernetes.io/projected/222a170e-28a4-4f09-857e-20d2b245a957-kube-api-access-s25gc\") pod \"payload-processing-e2e-worker-w0-609aba-58c9bb7b5c-w9mll\" (UID: \"222a170e-28a4-4f09-857e-20d2b245a957\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-609aba-58c9bb7b5c-w9mll"
Sep 18 18:03:36.337791 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:36.337760    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s25gc\" (UniqueName: \"kubernetes.io/projected/222a170e-28a4-4f09-857e-20d2b245a957-kube-api-access-s25gc\") pod \"payload-processing-e2e-worker-w0-609aba-58c9bb7b5c-w9mll\" (UID: \"222a170e-28a4-4f09-857e-20d2b245a957\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-609aba-58c9bb7b5c-w9mll"
Sep 18 18:03:36.337944 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:36.337809    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/222a170e-28a4-4f09-857e-20d2b245a957-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-609aba-58c9bb7b5c-w9mll\" (UID: \"222a170e-28a4-4f09-857e-20d2b245a957\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-609aba-58c9bb7b5c-w9mll"
Sep 18 18:03:36.338510 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:36.338492    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/222a170e-28a4-4f09-857e-20d2b245a957-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-609aba-58c9bb7b5c-w9mll\" (UID: \"222a170e-28a4-4f09-857e-20d2b245a957\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-609aba-58c9bb7b5c-w9mll"
Sep 18 18:03:36.360681 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:36.360650    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s25gc\" (UniqueName: \"kubernetes.io/projected/222a170e-28a4-4f09-857e-20d2b245a957-kube-api-access-s25gc\") pod \"payload-processing-e2e-worker-w0-609aba-58c9bb7b5c-w9mll\" (UID: \"222a170e-28a4-4f09-857e-20d2b245a957\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-609aba-58c9bb7b5c-w9mll"
Sep 18 18:03:36.439809 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:36.439780    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-609aba-58c9bb7b5c-w9mll"
Sep 18 18:03:36.703357 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:36.703316    2561 generic.go:356] "Generic (PLEG): container finished" podID="5d850953-c121-4229-939d-8156de224988" containerID="ed87368025c7dd00dbe29d32f5c00cfd71192e5a63c9a7b0a557b5203719fbfc" exitCode=137
Sep 18 18:03:36.703357 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:36.703363    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7611-f0c2a3"
Sep 18 18:03:36.703919 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:36.703413    2561 scope.go:117] "RemoveContainer" containerID="ed87368025c7dd00dbe29d32f5c00cfd71192e5a63c9a7b0a557b5203719fbfc"
Sep 18 18:03:36.706766 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:03:36.706742    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7611-f0c2a3\" is forbidden: User \"system:node:ip-10-0-140-54.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-54.ec2.internal' and this object" podUID="5d850953-c121-4229-939d-8156de224988" pod="opendatahub/test-curl-7611-f0c2a3"
Sep 18 18:03:36.712724 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:36.712557    2561 scope.go:117] "RemoveContainer" containerID="ed87368025c7dd00dbe29d32f5c00cfd71192e5a63c9a7b0a557b5203719fbfc"
Sep 18 18:03:36.712842 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:03:36.712824    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed87368025c7dd00dbe29d32f5c00cfd71192e5a63c9a7b0a557b5203719fbfc\": container with ID starting with ed87368025c7dd00dbe29d32f5c00cfd71192e5a63c9a7b0a557b5203719fbfc not found: ID does not exist" containerID="ed87368025c7dd00dbe29d32f5c00cfd71192e5a63c9a7b0a557b5203719fbfc"
Sep 18 18:03:36.712889 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:36.712849    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed87368025c7dd00dbe29d32f5c00cfd71192e5a63c9a7b0a557b5203719fbfc"} err="failed to get container status \"ed87368025c7dd00dbe29d32f5c00cfd71192e5a63c9a7b0a557b5203719fbfc\": rpc error: code = NotFound desc = could not find container \"ed87368025c7dd00dbe29d32f5c00cfd71192e5a63c9a7b0a557b5203719fbfc\": container with ID starting with ed87368025c7dd00dbe29d32f5c00cfd71192e5a63c9a7b0a557b5203719fbfc not found: ID does not exist"
Sep 18 18:03:36.715428 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:03:36.715406    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7611-f0c2a3\" is forbidden: User \"system:node:ip-10-0-140-54.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-54.ec2.internal' and this object" podUID="5d850953-c121-4229-939d-8156de224988" pod="opendatahub/test-curl-7611-f0c2a3"
Sep 18 18:03:36.770979 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:36.770825    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-609aba-58c9bb7b5c-w9mll"]
Sep 18 18:03:36.773815 ip-10-0-140-54 kubenswrapper[2561]: W0918 18:03:36.773789    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod222a170e_28a4_4f09_857e_20d2b245a957.slice/crio-9615c7a1475aabfcc3996630c14955897818fd83b4974a82bd30ffdc3551e896 WatchSource:0}: Error finding container 9615c7a1475aabfcc3996630c14955897818fd83b4974a82bd30ffdc3551e896: Status 404 returned error can't find the container with id 9615c7a1475aabfcc3996630c14955897818fd83b4974a82bd30ffdc3551e896
Sep 18 18:03:36.925299 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:36.925265    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:37.714666 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:37.714627    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-609aba-58c9bb7b5c-w9mll" event={"ID":"222a170e-28a4-4f09-857e-20d2b245a957","Type":"ContainerStarted","Data":"81d681646c2b39c89774fac7d899c644ddac7a24402d31ed66ade3ab4f3d75a4"}
Sep 18 18:03:37.715020 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:37.714678    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-609aba-58c9bb7b5c-w9mll" event={"ID":"222a170e-28a4-4f09-857e-20d2b245a957","Type":"ContainerStarted","Data":"9615c7a1475aabfcc3996630c14955897818fd83b4974a82bd30ffdc3551e896"}
Sep 18 18:03:37.715020 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:37.714709    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-609aba-58c9bb7b5c-w9mll"
Sep 18 18:03:37.716882 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:03:37.716858    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7611-f0c2a3\" is forbidden: User \"system:node:ip-10-0-140-54.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-54.ec2.internal' and this object" podUID="5d850953-c121-4229-939d-8156de224988" pod="opendatahub/test-curl-7611-f0c2a3"
Sep 18 18:03:37.743968 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:37.743922    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-609aba-58c9bb7b5c-w9mll" podStartSLOduration=1.74390752 podStartE2EDuration="1.74390752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:03:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:03:37.741877773 +0000 UTC m=+822.369009392" watchObservedRunningTime="2026-09-18 18:03:37.74390752 +0000 UTC m=+822.371039141"
Sep 18 18:03:37.861075 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:37.861043    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d850953-c121-4229-939d-8156de224988" path="/var/lib/kubelet/pods/5d850953-c121-4229-939d-8156de224988/volumes"
Sep 18 18:03:38.502742 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:38.502705    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:38.678884 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:38.678857    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-502939-54d94b8c78-k54n7"
Sep 18 18:03:39.010589 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:39.010536    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:39.124750 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:39.124724    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:39.290503 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:39.290421    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:39.711949 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:39.711915    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:39.838160 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:39.838099    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:40.479671 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:40.479629    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:40.613360 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:40.613330    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:41.170199 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:41.170169    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:41.709275 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:41.709249    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-03e512-7878686c45-mw6vc"
Sep 18 18:03:42.161336 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:42.161305    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:42.330743 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:42.330716    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:42.471735 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:42.471658    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:42.674188 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:42.674155    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:43.488371 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:43.488339    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:43.805112 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:43.805030    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:43.914423 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:43.914393    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:44.317826 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:44.317790    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:44.686463 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:44.686436    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-502939-56d954d58-zg4mx"
Sep 18 18:03:44.729497 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:44.729469    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:45.226627 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:45.226589    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:45.698251 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:45.698221    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-502939-6f789b6949-kfsvn"
Sep 18 18:03:45.854408 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:45.854372    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:46.148656 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:46.148619    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:46.275076 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:46.275037    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:46.389705 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:46.389674    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:46.706237 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:46.706209    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-03e512-5bd997d694-gp4gc"
Sep 18 18:03:46.706782 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:46.706289    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-03e512-85bdf7444c-vwgzw"
Sep 18 18:03:46.829951 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:46.829920    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:46.925268 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:46.925236    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:47.328301 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:47.328264    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:47.970395 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:47.970367    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:48.262222 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:48.262149    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:48.569388 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:48.569304    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:48.723286 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:48.723256    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-609aba-58c9bb7b5c-w9mll"
Sep 18 18:03:48.781053 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:48.781021    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:48.848713 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:48.848631    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-63da2c01-5cd49dfc47-v72st"]
Sep 18 18:03:48.857229 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:48.857203    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-63da2c01-5cd49dfc47-v72st"
Sep 18 18:03:48.865641 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:48.865618    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-63da2c01\""
Sep 18 18:03:48.871193 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:48.871170    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-63da2c01-5cd49dfc47-v72st"]
Sep 18 18:03:48.961563 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:48.961529    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8a2c661-284a-4fa3-98cd-7179f930d3df-plugins-config-volume\") pod \"payload-processing-e2e-ait-63da2c01-5cd49dfc47-v72st\" (UID: \"f8a2c661-284a-4fa3-98cd-7179f930d3df\") " pod="openshift-ingress/payload-processing-e2e-ait-63da2c01-5cd49dfc47-v72st"
Sep 18 18:03:48.961738 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:48.961575    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cglgj\" (UniqueName: \"kubernetes.io/projected/f8a2c661-284a-4fa3-98cd-7179f930d3df-kube-api-access-cglgj\") pod \"payload-processing-e2e-ait-63da2c01-5cd49dfc47-v72st\" (UID: \"f8a2c661-284a-4fa3-98cd-7179f930d3df\") " pod="openshift-ingress/payload-processing-e2e-ait-63da2c01-5cd49dfc47-v72st"
Sep 18 18:03:49.063319 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:49.062565    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8a2c661-284a-4fa3-98cd-7179f930d3df-plugins-config-volume\") pod \"payload-processing-e2e-ait-63da2c01-5cd49dfc47-v72st\" (UID: \"f8a2c661-284a-4fa3-98cd-7179f930d3df\") " pod="openshift-ingress/payload-processing-e2e-ait-63da2c01-5cd49dfc47-v72st"
Sep 18 18:03:49.063319 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:49.062617    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cglgj\" (UniqueName: \"kubernetes.io/projected/f8a2c661-284a-4fa3-98cd-7179f930d3df-kube-api-access-cglgj\") pod \"payload-processing-e2e-ait-63da2c01-5cd49dfc47-v72st\" (UID: \"f8a2c661-284a-4fa3-98cd-7179f930d3df\") " pod="openshift-ingress/payload-processing-e2e-ait-63da2c01-5cd49dfc47-v72st"
Sep 18 18:03:49.064068 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:49.063594    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8a2c661-284a-4fa3-98cd-7179f930d3df-plugins-config-volume\") pod \"payload-processing-e2e-ait-63da2c01-5cd49dfc47-v72st\" (UID: \"f8a2c661-284a-4fa3-98cd-7179f930d3df\") " pod="openshift-ingress/payload-processing-e2e-ait-63da2c01-5cd49dfc47-v72st"
Sep 18 18:03:49.078893 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:49.078867    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cglgj\" (UniqueName: \"kubernetes.io/projected/f8a2c661-284a-4fa3-98cd-7179f930d3df-kube-api-access-cglgj\") pod \"payload-processing-e2e-ait-63da2c01-5cd49dfc47-v72st\" (UID: \"f8a2c661-284a-4fa3-98cd-7179f930d3df\") " pod="openshift-ingress/payload-processing-e2e-ait-63da2c01-5cd49dfc47-v72st"
Sep 18 18:03:49.123293 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:49.123198    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:49.170648 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:49.170616    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-63da2c01-5cd49dfc47-v72st"
Sep 18 18:03:49.342650 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:49.342616    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:49.484086 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:49.484056    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-a39784-c6f678db4-7cs9h"]
Sep 18 18:03:49.489462 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:49.489442    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a39784-c6f678db4-7cs9h"
Sep 18 18:03:49.498949 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:49.498929    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-63da2c01-5cd49dfc47-v72st"]
Sep 18 18:03:49.499059 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:49.498954    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-a39784-serving-cert\""
Sep 18 18:03:49.499975 ip-10-0-140-54 kubenswrapper[2561]: W0918 18:03:49.499958    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8a2c661_284a_4fa3_98cd_7179f930d3df.slice/crio-342723663ce665a335d27f577971db37d0662d0673fd39aed9b1fb654b9d9ee9 WatchSource:0}: Error finding container 342723663ce665a335d27f577971db37d0662d0673fd39aed9b1fb654b9d9ee9: Status 404 returned error can't find the container with id 342723663ce665a335d27f577971db37d0662d0673fd39aed9b1fb654b9d9ee9
Sep 18 18:03:49.508801 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:49.508782    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-a39784-c6f678db4-7cs9h"]
Sep 18 18:03:49.523089 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:49.523068    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:49.656106 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:49.656069    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:49.668438 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:49.668410    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f071103-1087-4b52-b526-99219526e36c-maas-api-tls\") pod \"maas-api-e2e-autores-a39784-c6f678db4-7cs9h\" (UID: \"2f071103-1087-4b52-b526-99219526e36c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a39784-c6f678db4-7cs9h"
Sep 18 18:03:49.668572 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:49.668456    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64h7q\" (UniqueName: \"kubernetes.io/projected/2f071103-1087-4b52-b526-99219526e36c-kube-api-access-64h7q\") pod \"maas-api-e2e-autores-a39784-c6f678db4-7cs9h\" (UID: \"2f071103-1087-4b52-b526-99219526e36c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a39784-c6f678db4-7cs9h"
Sep 18 18:03:49.668572 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:49.668492    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f071103-1087-4b52-b526-99219526e36c-metrics-certs\") pod \"maas-api-e2e-autores-a39784-c6f678db4-7cs9h\" (UID: \"2f071103-1087-4b52-b526-99219526e36c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a39784-c6f678db4-7cs9h"
Sep 18 18:03:49.668671 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:49.668607    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f071103-1087-4b52-b526-99219526e36c-openshift-service-ca\") pod \"maas-api-e2e-autores-a39784-c6f678db4-7cs9h\" (UID: \"2f071103-1087-4b52-b526-99219526e36c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a39784-c6f678db4-7cs9h"
Sep 18 18:03:49.769965 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:49.769891    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f071103-1087-4b52-b526-99219526e36c-maas-api-tls\") pod \"maas-api-e2e-autores-a39784-c6f678db4-7cs9h\" (UID: \"2f071103-1087-4b52-b526-99219526e36c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a39784-c6f678db4-7cs9h"
Sep 18 18:03:49.769965 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:49.769928    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64h7q\" (UniqueName: \"kubernetes.io/projected/2f071103-1087-4b52-b526-99219526e36c-kube-api-access-64h7q\") pod \"maas-api-e2e-autores-a39784-c6f678db4-7cs9h\" (UID: \"2f071103-1087-4b52-b526-99219526e36c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a39784-c6f678db4-7cs9h"
Sep 18 18:03:49.769965 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:49.769947    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f071103-1087-4b52-b526-99219526e36c-metrics-certs\") pod \"maas-api-e2e-autores-a39784-c6f678db4-7cs9h\" (UID: \"2f071103-1087-4b52-b526-99219526e36c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a39784-c6f678db4-7cs9h"
Sep 18 18:03:49.770241 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:03:49.770046    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-a39784-serving-cert: secret "maas-api-e2e-autores-a39784-serving-cert" not found
Sep 18 18:03:49.770241 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:03:49.770136    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f071103-1087-4b52-b526-99219526e36c-maas-api-tls podName:2f071103-1087-4b52-b526-99219526e36c nodeName:}" failed. No retries permitted until 2026-09-18 18:03:50.270101659 +0000 UTC m=+834.897233263 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2f071103-1087-4b52-b526-99219526e36c-maas-api-tls") pod "maas-api-e2e-autores-a39784-c6f678db4-7cs9h" (UID: "2f071103-1087-4b52-b526-99219526e36c") : secret "maas-api-e2e-autores-a39784-serving-cert" not found
Sep 18 18:03:49.770241 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:49.770159    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f071103-1087-4b52-b526-99219526e36c-openshift-service-ca\") pod \"maas-api-e2e-autores-a39784-c6f678db4-7cs9h\" (UID: \"2f071103-1087-4b52-b526-99219526e36c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a39784-c6f678db4-7cs9h"
Sep 18 18:03:49.770870 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:49.770848    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f071103-1087-4b52-b526-99219526e36c-openshift-service-ca\") pod \"maas-api-e2e-autores-a39784-c6f678db4-7cs9h\" (UID: \"2f071103-1087-4b52-b526-99219526e36c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a39784-c6f678db4-7cs9h"
Sep 18 18:03:49.772397 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:49.772380    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f071103-1087-4b52-b526-99219526e36c-metrics-certs\") pod \"maas-api-e2e-autores-a39784-c6f678db4-7cs9h\" (UID: \"2f071103-1087-4b52-b526-99219526e36c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a39784-c6f678db4-7cs9h"
Sep 18 18:03:49.780582 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:49.780553    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-63da2c01-5cd49dfc47-v72st" event={"ID":"f8a2c661-284a-4fa3-98cd-7179f930d3df","Type":"ContainerStarted","Data":"d93dc1d5f16617a47d07e7c1c1fdcc4c8437fa1d8994547557236884679da8bc"}
Sep 18 18:03:49.780692 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:49.780646    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-63da2c01-5cd49dfc47-v72st"
Sep 18 18:03:49.780692 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:49.780670    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-63da2c01-5cd49dfc47-v72st" event={"ID":"f8a2c661-284a-4fa3-98cd-7179f930d3df","Type":"ContainerStarted","Data":"342723663ce665a335d27f577971db37d0662d0673fd39aed9b1fb654b9d9ee9"}
Sep 18 18:03:49.783108 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:49.783083    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64h7q\" (UniqueName: \"kubernetes.io/projected/2f071103-1087-4b52-b526-99219526e36c-kube-api-access-64h7q\") pod \"maas-api-e2e-autores-a39784-c6f678db4-7cs9h\" (UID: \"2f071103-1087-4b52-b526-99219526e36c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a39784-c6f678db4-7cs9h"
Sep 18 18:03:49.808951 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:49.808902    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-63da2c01-5cd49dfc47-v72st" podStartSLOduration=1.8088846950000002 podStartE2EDuration="1.808884695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:03:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:03:49.807534953 +0000 UTC m=+834.434666573" watchObservedRunningTime="2026-09-18 18:03:49.808884695 +0000 UTC m=+834.436016318"
Sep 18 18:03:50.010994 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:50.010963    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:50.274973 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:50.274940    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f071103-1087-4b52-b526-99219526e36c-maas-api-tls\") pod \"maas-api-e2e-autores-a39784-c6f678db4-7cs9h\" (UID: \"2f071103-1087-4b52-b526-99219526e36c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a39784-c6f678db4-7cs9h"
Sep 18 18:03:50.277422 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:50.277399    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f071103-1087-4b52-b526-99219526e36c-maas-api-tls\") pod \"maas-api-e2e-autores-a39784-c6f678db4-7cs9h\" (UID: \"2f071103-1087-4b52-b526-99219526e36c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a39784-c6f678db4-7cs9h"
Sep 18 18:03:50.367983 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:50.367952    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:50.401706 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:50.401677    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a39784-c6f678db4-7cs9h"
Sep 18 18:03:50.494794 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:50.494769    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:50.719005 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:50.718973    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-a39784-c6f678db4-7cs9h"]
Sep 18 18:03:50.721966 ip-10-0-140-54 kubenswrapper[2561]: W0918 18:03:50.721934    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f071103_1087_4b52_b526_99219526e36c.slice/crio-9528f947e96b67f1d032dfe4d3fe0a3b0f7ca9a0c1af5a879662df5c5a9a79ec WatchSource:0}: Error finding container 9528f947e96b67f1d032dfe4d3fe0a3b0f7ca9a0c1af5a879662df5c5a9a79ec: Status 404 returned error can't find the container with id 9528f947e96b67f1d032dfe4d3fe0a3b0f7ca9a0c1af5a879662df5c5a9a79ec
Sep 18 18:03:50.787638 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:50.787601    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a39784-c6f678db4-7cs9h" event={"ID":"2f071103-1087-4b52-b526-99219526e36c","Type":"ContainerStarted","Data":"9528f947e96b67f1d032dfe4d3fe0a3b0f7ca9a0c1af5a879662df5c5a9a79ec"}
Sep 18 18:03:51.177665 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:51.177637    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:51.794890 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:51.794854    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a39784-c6f678db4-7cs9h" event={"ID":"2f071103-1087-4b52-b526-99219526e36c","Type":"ContainerStarted","Data":"240ef21cf11dc7ff673f87d8fa4232d11b3a75efeff7e35d41b94619e7e846c4"}
Sep 18 18:03:51.795279 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:51.794913    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a39784-c6f678db4-7cs9h"
Sep 18 18:03:51.839847 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:51.839803    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a39784-c6f678db4-7cs9h" podStartSLOduration=2.267041193 podStartE2EDuration="2.839787583s" totalImagesPullingTime="572.74639ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:03:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:03:51.838839996 +0000 UTC m=+836.465971611" watchObservedRunningTime="2026-09-18 18:03:51.839787583 +0000 UTC m=+836.466919203"
Sep 18 18:03:52.612600 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:52.612570    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:52.911840 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:52.911802    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:53.342515 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:53.342443    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:53.499192 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:53.499160    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:53.651612 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:53.651579    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:54.463783 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:54.463754    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ee3049-7cc8b98769-s9jj5"]
Sep 18 18:03:54.469934 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:54.469914    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ee3049-7cc8b98769-s9jj5"
Sep 18 18:03:54.473242 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:54.473220    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-ee3049-serving-cert\""
Sep 18 18:03:54.499522 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:54.499500    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ee3049-7cc8b98769-s9jj5"]
Sep 18 18:03:54.517501 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:54.517478    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/843c8a1a-4277-4e30-9fe3-1d726d6f0efb-maas-api-tls\") pod \"maas-api-e2e-inf-b-ee3049-7cc8b98769-s9jj5\" (UID: \"843c8a1a-4277-4e30-9fe3-1d726d6f0efb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ee3049-7cc8b98769-s9jj5"
Sep 18 18:03:54.517601 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:54.517504    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njtkb\" (UniqueName: \"kubernetes.io/projected/843c8a1a-4277-4e30-9fe3-1d726d6f0efb-kube-api-access-njtkb\") pod \"maas-api-e2e-inf-b-ee3049-7cc8b98769-s9jj5\" (UID: \"843c8a1a-4277-4e30-9fe3-1d726d6f0efb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ee3049-7cc8b98769-s9jj5"
Sep 18 18:03:54.517649 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:54.517599    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/843c8a1a-4277-4e30-9fe3-1d726d6f0efb-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ee3049-7cc8b98769-s9jj5\" (UID: \"843c8a1a-4277-4e30-9fe3-1d726d6f0efb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ee3049-7cc8b98769-s9jj5"
Sep 18 18:03:54.517685 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:54.517647    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/843c8a1a-4277-4e30-9fe3-1d726d6f0efb-metrics-certs\") pod \"maas-api-e2e-inf-b-ee3049-7cc8b98769-s9jj5\" (UID: \"843c8a1a-4277-4e30-9fe3-1d726d6f0efb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ee3049-7cc8b98769-s9jj5"
Sep 18 18:03:54.572723 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:54.572695    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:54.618431 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:54.618397    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/843c8a1a-4277-4e30-9fe3-1d726d6f0efb-maas-api-tls\") pod \"maas-api-e2e-inf-b-ee3049-7cc8b98769-s9jj5\" (UID: \"843c8a1a-4277-4e30-9fe3-1d726d6f0efb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ee3049-7cc8b98769-s9jj5"
Sep 18 18:03:54.618431 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:54.618434    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njtkb\" (UniqueName: \"kubernetes.io/projected/843c8a1a-4277-4e30-9fe3-1d726d6f0efb-kube-api-access-njtkb\") pod \"maas-api-e2e-inf-b-ee3049-7cc8b98769-s9jj5\" (UID: \"843c8a1a-4277-4e30-9fe3-1d726d6f0efb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ee3049-7cc8b98769-s9jj5"
Sep 18 18:03:54.618665 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:54.618510    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/843c8a1a-4277-4e30-9fe3-1d726d6f0efb-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ee3049-7cc8b98769-s9jj5\" (UID: \"843c8a1a-4277-4e30-9fe3-1d726d6f0efb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ee3049-7cc8b98769-s9jj5"
Sep 18 18:03:54.618665 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:54.618550    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/843c8a1a-4277-4e30-9fe3-1d726d6f0efb-metrics-certs\") pod \"maas-api-e2e-inf-b-ee3049-7cc8b98769-s9jj5\" (UID: \"843c8a1a-4277-4e30-9fe3-1d726d6f0efb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ee3049-7cc8b98769-s9jj5"
Sep 18 18:03:54.618665 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:03:54.618554    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-ee3049-serving-cert: secret "maas-api-e2e-inf-b-ee3049-serving-cert" not found
Sep 18 18:03:54.618816 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:03:54.618703    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/843c8a1a-4277-4e30-9fe3-1d726d6f0efb-maas-api-tls podName:843c8a1a-4277-4e30-9fe3-1d726d6f0efb nodeName:}" failed. No retries permitted until 2026-09-18 18:03:55.118683658 +0000 UTC m=+839.745815261 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/843c8a1a-4277-4e30-9fe3-1d726d6f0efb-maas-api-tls") pod "maas-api-e2e-inf-b-ee3049-7cc8b98769-s9jj5" (UID: "843c8a1a-4277-4e30-9fe3-1d726d6f0efb") : secret "maas-api-e2e-inf-b-ee3049-serving-cert" not found
Sep 18 18:03:54.619195 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:54.619171    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/843c8a1a-4277-4e30-9fe3-1d726d6f0efb-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ee3049-7cc8b98769-s9jj5\" (UID: \"843c8a1a-4277-4e30-9fe3-1d726d6f0efb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ee3049-7cc8b98769-s9jj5"
Sep 18 18:03:54.620928 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:54.620909    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/843c8a1a-4277-4e30-9fe3-1d726d6f0efb-metrics-certs\") pod \"maas-api-e2e-inf-b-ee3049-7cc8b98769-s9jj5\" (UID: \"843c8a1a-4277-4e30-9fe3-1d726d6f0efb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ee3049-7cc8b98769-s9jj5"
Sep 18 18:03:54.672755 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:54.672726    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njtkb\" (UniqueName: \"kubernetes.io/projected/843c8a1a-4277-4e30-9fe3-1d726d6f0efb-kube-api-access-njtkb\") pod \"maas-api-e2e-inf-b-ee3049-7cc8b98769-s9jj5\" (UID: \"843c8a1a-4277-4e30-9fe3-1d726d6f0efb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ee3049-7cc8b98769-s9jj5"
Sep 18 18:03:54.734604 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:54.734528    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:54.861411 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:54.861381    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:54.978527 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:54.978496    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:55.123393 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:55.123307    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/843c8a1a-4277-4e30-9fe3-1d726d6f0efb-maas-api-tls\") pod \"maas-api-e2e-inf-b-ee3049-7cc8b98769-s9jj5\" (UID: \"843c8a1a-4277-4e30-9fe3-1d726d6f0efb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ee3049-7cc8b98769-s9jj5"
Sep 18 18:03:55.125591 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:55.125563    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/843c8a1a-4277-4e30-9fe3-1d726d6f0efb-maas-api-tls\") pod \"maas-api-e2e-inf-b-ee3049-7cc8b98769-s9jj5\" (UID: \"843c8a1a-4277-4e30-9fe3-1d726d6f0efb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ee3049-7cc8b98769-s9jj5"
Sep 18 18:03:55.380320 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:55.380225    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ee3049-7cc8b98769-s9jj5"
Sep 18 18:03:55.506473 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:55.506441    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:55.668304 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:55.668282    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ee3049-7cc8b98769-s9jj5"]
Sep 18 18:03:55.669469 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:55.669446    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:55.670263 ip-10-0-140-54 kubenswrapper[2561]: W0918 18:03:55.670237    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod843c8a1a_4277_4e30_9fe3_1d726d6f0efb.slice/crio-4ddb541ef86e290c912983130e29cde48b7e337827bdc42b050c239e201db098 WatchSource:0}: Error finding container 4ddb541ef86e290c912983130e29cde48b7e337827bdc42b050c239e201db098: Status 404 returned error can't find the container with id 4ddb541ef86e290c912983130e29cde48b7e337827bdc42b050c239e201db098
Sep 18 18:03:55.814900 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:55.814864    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ee3049-7cc8b98769-s9jj5" event={"ID":"843c8a1a-4277-4e30-9fe3-1d726d6f0efb","Type":"ContainerStarted","Data":"4ddb541ef86e290c912983130e29cde48b7e337827bdc42b050c239e201db098"}
Sep 18 18:03:56.821296 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:56.821256    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ee3049-7cc8b98769-s9jj5" event={"ID":"843c8a1a-4277-4e30-9fe3-1d726d6f0efb","Type":"ContainerStarted","Data":"fd61df0b726e0d3c517d80ea3ec7c9e6734091033a74683315828603e28bc237"}
Sep 18 18:03:56.821651 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:56.821334    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ee3049-7cc8b98769-s9jj5"
Sep 18 18:03:56.864721 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:56.864669    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ee3049-7cc8b98769-s9jj5" podStartSLOduration=2.5351942750000003 podStartE2EDuration="2.864653554s" totalImagesPullingTime="329.459279ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:03:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:03:56.863343447 +0000 UTC m=+841.490475098" watchObservedRunningTime="2026-09-18 18:03:56.864653554 +0000 UTC m=+841.491785174"
Sep 18 18:03:57.118228 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:57.118141    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:57.805418 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:57.805388    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a39784-c6f678db4-7cs9h"
Sep 18 18:03:57.915960 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:57.915929    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:03:59.323399 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:03:59.323364    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:00.441221 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:00.441185    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:00.790797 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:00.790711    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-63da2c01-5cd49dfc47-v72st"
Sep 18 18:04:01.852131 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:01.852089    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:02.832096 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:02.832065    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ee3049-7cc8b98769-s9jj5"
Sep 18 18:04:02.842197 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:02.842173    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:03.771218 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:03.771189    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:04.711727 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:04.711697    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:05.629738 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:05.629705    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:05.818065 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:05.818039    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:06.011308 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:06.011270    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:06.164758 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:06.164726    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:06.289894 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:06.289808    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:07.494074 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:07.494046    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:07.808691 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:07.808619    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:08.632236 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:08.632206    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:09.581539 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:09.581507    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:10.695995 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:10.695966    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:11.881084 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:11.881050    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:13.088761 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:13.088730    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:13.947306 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:13.947266    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:13.981527 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:13.981492    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-63da2c01-5cd49dfc47-v72st"]
Sep 18 18:04:13.981767 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:13.981712    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-63da2c01-5cd49dfc47-v72st" podUID="f8a2c661-284a-4fa3-98cd-7179f930d3df" containerName="payload-processing" containerID="cri-o://d93dc1d5f16617a47d07e7c1c1fdcc4c8437fa1d8994547557236884679da8bc" gracePeriod=30
Sep 18 18:04:14.063311 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:14.063271    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:14.248651 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:14.248622    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-63da2c01-5cd49dfc47-v72st"
Sep 18 18:04:14.321471 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:14.321429    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8a2c661-284a-4fa3-98cd-7179f930d3df-plugins-config-volume\") pod \"f8a2c661-284a-4fa3-98cd-7179f930d3df\" (UID: \"f8a2c661-284a-4fa3-98cd-7179f930d3df\") "
Sep 18 18:04:14.321638 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:14.321550    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cglgj\" (UniqueName: \"kubernetes.io/projected/f8a2c661-284a-4fa3-98cd-7179f930d3df-kube-api-access-cglgj\") pod \"f8a2c661-284a-4fa3-98cd-7179f930d3df\" (UID: \"f8a2c661-284a-4fa3-98cd-7179f930d3df\") "
Sep 18 18:04:14.321864 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:14.321833    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8a2c661-284a-4fa3-98cd-7179f930d3df-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f8a2c661-284a-4fa3-98cd-7179f930d3df" (UID: "f8a2c661-284a-4fa3-98cd-7179f930d3df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:04:14.323798 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:14.323779    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8a2c661-284a-4fa3-98cd-7179f930d3df-kube-api-access-cglgj" (OuterVolumeSpecName: "kube-api-access-cglgj") pod "f8a2c661-284a-4fa3-98cd-7179f930d3df" (UID: "f8a2c661-284a-4fa3-98cd-7179f930d3df"). InnerVolumeSpecName "kube-api-access-cglgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:04:14.381252 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:14.381224    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:14.422865 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:14.422836    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cglgj\" (UniqueName: \"kubernetes.io/projected/f8a2c661-284a-4fa3-98cd-7179f930d3df-kube-api-access-cglgj\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:04:14.422865 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:14.422858    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8a2c661-284a-4fa3-98cd-7179f930d3df-plugins-config-volume\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:04:14.591339 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:14.591262    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:14.720294 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:14.720256    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:14.917589 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:14.917553    2561 generic.go:356] "Generic (PLEG): container finished" podID="f8a2c661-284a-4fa3-98cd-7179f930d3df" containerID="d93dc1d5f16617a47d07e7c1c1fdcc4c8437fa1d8994547557236884679da8bc" exitCode=0
Sep 18 18:04:14.917751 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:14.917596    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-63da2c01-5cd49dfc47-v72st"
Sep 18 18:04:14.917751 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:14.917636    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-63da2c01-5cd49dfc47-v72st" event={"ID":"f8a2c661-284a-4fa3-98cd-7179f930d3df","Type":"ContainerDied","Data":"d93dc1d5f16617a47d07e7c1c1fdcc4c8437fa1d8994547557236884679da8bc"}
Sep 18 18:04:14.917751 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:14.917684    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-63da2c01-5cd49dfc47-v72st" event={"ID":"f8a2c661-284a-4fa3-98cd-7179f930d3df","Type":"ContainerDied","Data":"342723663ce665a335d27f577971db37d0662d0673fd39aed9b1fb654b9d9ee9"}
Sep 18 18:04:14.917751 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:14.917706    2561 scope.go:117] "RemoveContainer" containerID="d93dc1d5f16617a47d07e7c1c1fdcc4c8437fa1d8994547557236884679da8bc"
Sep 18 18:04:14.927504 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:14.927487    2561 scope.go:117] "RemoveContainer" containerID="d93dc1d5f16617a47d07e7c1c1fdcc4c8437fa1d8994547557236884679da8bc"
Sep 18 18:04:14.927761 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:04:14.927744    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d93dc1d5f16617a47d07e7c1c1fdcc4c8437fa1d8994547557236884679da8bc\": container with ID starting with d93dc1d5f16617a47d07e7c1c1fdcc4c8437fa1d8994547557236884679da8bc not found: ID does not exist" containerID="d93dc1d5f16617a47d07e7c1c1fdcc4c8437fa1d8994547557236884679da8bc"
Sep 18 18:04:14.927823 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:14.927767    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d93dc1d5f16617a47d07e7c1c1fdcc4c8437fa1d8994547557236884679da8bc"} err="failed to get container status \"d93dc1d5f16617a47d07e7c1c1fdcc4c8437fa1d8994547557236884679da8bc\": rpc error: code = NotFound desc = could not find container \"d93dc1d5f16617a47d07e7c1c1fdcc4c8437fa1d8994547557236884679da8bc\": container with ID starting with d93dc1d5f16617a47d07e7c1c1fdcc4c8437fa1d8994547557236884679da8bc not found: ID does not exist"
Sep 18 18:04:14.980548 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:14.980519    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-63da2c01-5cd49dfc47-v72st"]
Sep 18 18:04:14.987238 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:14.987214    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-63da2c01-5cd49dfc47-v72st"]
Sep 18 18:04:15.050283 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:15.050246    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:15.175232 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:15.175161    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:15.325262 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:15.325231    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:15.612366 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:15.612290    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:15.790223 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:15.790193    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:15.863623 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:15.863548    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8a2c661-284a-4fa3-98cd-7179f930d3df" path="/var/lib/kubelet/pods/f8a2c661-284a-4fa3-98cd-7179f930d3df/volumes"
Sep 18 18:04:16.210269 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:16.210235    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:16.944557 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:16.944531    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:18.133783 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:18.133755    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:18.284988 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:18.284961    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:18.600736 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:18.600704    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:19.899180 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:19.899153    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:21.249013 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:21.248983    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:22.703861 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:22.703813    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:23.755737 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:23.755707    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:24.051009 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:24.050927    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:24.353278 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:24.353198    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:24.773905 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:24.773876    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:24.911492 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:24.911464    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:25.551635 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:25.551592    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:26.840713 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:26.840685    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:27.639424 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:27.639379    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:35.176929 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:35.176895    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-03e512/simulator-w4-03e512-kserve-645699d677-9nddq"]
Sep 18 18:04:35.177350 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:35.177336    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8a2c661-284a-4fa3-98cd-7179f930d3df" containerName="payload-processing"
Sep 18 18:04:35.177394 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:35.177351    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8a2c661-284a-4fa3-98cd-7179f930d3df" containerName="payload-processing"
Sep 18 18:04:35.177437 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:35.177428    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8a2c661-284a-4fa3-98cd-7179f930d3df" containerName="payload-processing"
Sep 18 18:04:35.181403 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:35.181386    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-03e512/simulator-w4-03e512-kserve-645699d677-9nddq"
Sep 18 18:04:35.190324 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:35.190303    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-03e512\"/\"default-dockercfg-wf7xj\""
Sep 18 18:04:35.190407 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:35.190330    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-03e512\"/\"simulator-w4-03e512-kserve-self-signed-certs\""
Sep 18 18:04:35.190407 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:35.190331    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-03e512\"/\"openshift-service-ca.crt\""
Sep 18 18:04:35.190997 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:35.190983    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-03e512\"/\"kube-root-ca.crt\""
Sep 18 18:04:35.224001 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:35.223969    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-03e512/simulator-w4-03e512-kserve-645699d677-9nddq"]
Sep 18 18:04:35.225472 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:35.225450    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qv52\" (UniqueName: \"kubernetes.io/projected/0097f764-4e35-4d2d-af6c-8d2a0f496b7e-kube-api-access-5qv52\") pod \"simulator-w4-03e512-kserve-645699d677-9nddq\" (UID: \"0097f764-4e35-4d2d-af6c-8d2a0f496b7e\") " pod="e2e-models-e2e-worker-w4-03e512/simulator-w4-03e512-kserve-645699d677-9nddq"
Sep 18 18:04:35.225569 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:35.225502    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0097f764-4e35-4d2d-af6c-8d2a0f496b7e-home\") pod \"simulator-w4-03e512-kserve-645699d677-9nddq\" (UID: \"0097f764-4e35-4d2d-af6c-8d2a0f496b7e\") " pod="e2e-models-e2e-worker-w4-03e512/simulator-w4-03e512-kserve-645699d677-9nddq"
Sep 18 18:04:35.225569 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:35.225526    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0097f764-4e35-4d2d-af6c-8d2a0f496b7e-tls-certs\") pod \"simulator-w4-03e512-kserve-645699d677-9nddq\" (UID: \"0097f764-4e35-4d2d-af6c-8d2a0f496b7e\") " pod="e2e-models-e2e-worker-w4-03e512/simulator-w4-03e512-kserve-645699d677-9nddq"
Sep 18 18:04:35.225569 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:35.225556    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0097f764-4e35-4d2d-af6c-8d2a0f496b7e-dshm\") pod \"simulator-w4-03e512-kserve-645699d677-9nddq\" (UID: \"0097f764-4e35-4d2d-af6c-8d2a0f496b7e\") " pod="e2e-models-e2e-worker-w4-03e512/simulator-w4-03e512-kserve-645699d677-9nddq"
Sep 18 18:04:35.225727 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:35.225579    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0097f764-4e35-4d2d-af6c-8d2a0f496b7e-tmp-dir\") pod \"simulator-w4-03e512-kserve-645699d677-9nddq\" (UID: \"0097f764-4e35-4d2d-af6c-8d2a0f496b7e\") " pod="e2e-models-e2e-worker-w4-03e512/simulator-w4-03e512-kserve-645699d677-9nddq"
Sep 18 18:04:35.225727 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:35.225646    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0097f764-4e35-4d2d-af6c-8d2a0f496b7e-model-cache\") pod \"simulator-w4-03e512-kserve-645699d677-9nddq\" (UID: \"0097f764-4e35-4d2d-af6c-8d2a0f496b7e\") " pod="e2e-models-e2e-worker-w4-03e512/simulator-w4-03e512-kserve-645699d677-9nddq"
Sep 18 18:04:35.326502 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:35.326470    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0097f764-4e35-4d2d-af6c-8d2a0f496b7e-home\") pod \"simulator-w4-03e512-kserve-645699d677-9nddq\" (UID: \"0097f764-4e35-4d2d-af6c-8d2a0f496b7e\") " pod="e2e-models-e2e-worker-w4-03e512/simulator-w4-03e512-kserve-645699d677-9nddq"
Sep 18 18:04:35.326502 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:35.326500    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0097f764-4e35-4d2d-af6c-8d2a0f496b7e-tls-certs\") pod \"simulator-w4-03e512-kserve-645699d677-9nddq\" (UID: \"0097f764-4e35-4d2d-af6c-8d2a0f496b7e\") " pod="e2e-models-e2e-worker-w4-03e512/simulator-w4-03e512-kserve-645699d677-9nddq"
Sep 18 18:04:35.326726 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:35.326515    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0097f764-4e35-4d2d-af6c-8d2a0f496b7e-dshm\") pod \"simulator-w4-03e512-kserve-645699d677-9nddq\" (UID: \"0097f764-4e35-4d2d-af6c-8d2a0f496b7e\") " pod="e2e-models-e2e-worker-w4-03e512/simulator-w4-03e512-kserve-645699d677-9nddq"
Sep 18 18:04:35.326726 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:35.326537    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0097f764-4e35-4d2d-af6c-8d2a0f496b7e-tmp-dir\") pod \"simulator-w4-03e512-kserve-645699d677-9nddq\" (UID: \"0097f764-4e35-4d2d-af6c-8d2a0f496b7e\") " pod="e2e-models-e2e-worker-w4-03e512/simulator-w4-03e512-kserve-645699d677-9nddq"
Sep 18 18:04:35.326726 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:35.326664    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0097f764-4e35-4d2d-af6c-8d2a0f496b7e-model-cache\") pod \"simulator-w4-03e512-kserve-645699d677-9nddq\" (UID: \"0097f764-4e35-4d2d-af6c-8d2a0f496b7e\") " pod="e2e-models-e2e-worker-w4-03e512/simulator-w4-03e512-kserve-645699d677-9nddq"
Sep 18 18:04:35.326894 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:35.326764    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qv52\" (UniqueName: \"kubernetes.io/projected/0097f764-4e35-4d2d-af6c-8d2a0f496b7e-kube-api-access-5qv52\") pod \"simulator-w4-03e512-kserve-645699d677-9nddq\" (UID: \"0097f764-4e35-4d2d-af6c-8d2a0f496b7e\") " pod="e2e-models-e2e-worker-w4-03e512/simulator-w4-03e512-kserve-645699d677-9nddq"
Sep 18 18:04:35.326894 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:35.326881    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0097f764-4e35-4d2d-af6c-8d2a0f496b7e-tmp-dir\") pod \"simulator-w4-03e512-kserve-645699d677-9nddq\" (UID: \"0097f764-4e35-4d2d-af6c-8d2a0f496b7e\") " pod="e2e-models-e2e-worker-w4-03e512/simulator-w4-03e512-kserve-645699d677-9nddq"
Sep 18 18:04:35.326994 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:35.326984    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0097f764-4e35-4d2d-af6c-8d2a0f496b7e-model-cache\") pod \"simulator-w4-03e512-kserve-645699d677-9nddq\" (UID: \"0097f764-4e35-4d2d-af6c-8d2a0f496b7e\") " pod="e2e-models-e2e-worker-w4-03e512/simulator-w4-03e512-kserve-645699d677-9nddq"
Sep 18 18:04:35.327049 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:35.326985    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0097f764-4e35-4d2d-af6c-8d2a0f496b7e-home\") pod \"simulator-w4-03e512-kserve-645699d677-9nddq\" (UID: \"0097f764-4e35-4d2d-af6c-8d2a0f496b7e\") " pod="e2e-models-e2e-worker-w4-03e512/simulator-w4-03e512-kserve-645699d677-9nddq"
Sep 18 18:04:35.328817 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:35.328793    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0097f764-4e35-4d2d-af6c-8d2a0f496b7e-dshm\") pod \"simulator-w4-03e512-kserve-645699d677-9nddq\" (UID: \"0097f764-4e35-4d2d-af6c-8d2a0f496b7e\") " pod="e2e-models-e2e-worker-w4-03e512/simulator-w4-03e512-kserve-645699d677-9nddq"
Sep 18 18:04:35.329058 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:35.329042    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0097f764-4e35-4d2d-af6c-8d2a0f496b7e-tls-certs\") pod \"simulator-w4-03e512-kserve-645699d677-9nddq\" (UID: \"0097f764-4e35-4d2d-af6c-8d2a0f496b7e\") " pod="e2e-models-e2e-worker-w4-03e512/simulator-w4-03e512-kserve-645699d677-9nddq"
Sep 18 18:04:35.342630 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:35.342601    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qv52\" (UniqueName: \"kubernetes.io/projected/0097f764-4e35-4d2d-af6c-8d2a0f496b7e-kube-api-access-5qv52\") pod \"simulator-w4-03e512-kserve-645699d677-9nddq\" (UID: \"0097f764-4e35-4d2d-af6c-8d2a0f496b7e\") " pod="e2e-models-e2e-worker-w4-03e512/simulator-w4-03e512-kserve-645699d677-9nddq"
Sep 18 18:04:35.485726 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:35.485648    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:35.490699 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:35.490676    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-03e512/simulator-w4-03e512-kserve-645699d677-9nddq"
Sep 18 18:04:35.772300 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:35.772272    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-03e512/simulator-w4-03e512-kserve-645699d677-9nddq"]
Sep 18 18:04:35.773488 ip-10-0-140-54 kubenswrapper[2561]: W0918 18:04:35.773460    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0097f764_4e35_4d2d_af6c_8d2a0f496b7e.slice/crio-32ff466c81cea38349634f6c167e9f4905e9207ed1167b6f85d6b9579bf1a1b9 WatchSource:0}: Error finding container 32ff466c81cea38349634f6c167e9f4905e9207ed1167b6f85d6b9579bf1a1b9: Status 404 returned error can't find the container with id 32ff466c81cea38349634f6c167e9f4905e9207ed1167b6f85d6b9579bf1a1b9
Sep 18 18:04:36.037141 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:36.037028    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-03e512/simulator-w4-03e512-kserve-645699d677-9nddq" event={"ID":"0097f764-4e35-4d2d-af6c-8d2a0f496b7e","Type":"ContainerStarted","Data":"ed1aa5ae9f0965a7b3f5862be8455ad9726c058bf25f45103772ef7ba12aaadd"}
Sep 18 18:04:36.037141 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:36.037074    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-03e512/simulator-w4-03e512-kserve-645699d677-9nddq" event={"ID":"0097f764-4e35-4d2d-af6c-8d2a0f496b7e","Type":"ContainerStarted","Data":"32ff466c81cea38349634f6c167e9f4905e9207ed1167b6f85d6b9579bf1a1b9"}
Sep 18 18:04:36.069361 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:36.069317    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-03e512/simulator-w4-03e512-kserve-645699d677-9nddq" podStartSLOduration=1.069300932 podStartE2EDuration="1.069300932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:04:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:04:36.066841104 +0000 UTC m=+880.693972724" watchObservedRunningTime="2026-09-18 18:04:36.069300932 +0000 UTC m=+880.696432552"
Sep 18 18:04:45.491740 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:45.491709    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-03e512/simulator-w4-03e512-kserve-645699d677-9nddq"
Sep 18 18:04:45.492222 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:45.492025    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-03e512/simulator-w4-03e512-kserve-645699d677-9nddq"
Sep 18 18:04:45.504536 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:45.504512    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-03e512/simulator-w4-03e512-kserve-645699d677-9nddq"
Sep 18 18:04:46.102695 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:46.102657    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-03e512/simulator-w4-03e512-kserve-645699d677-9nddq"
Sep 18 18:04:46.941724 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:46.941690    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:47.900133 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:47.900079    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:49.070761 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:49.070727    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:50.176955 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:50.176917    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:51.159331 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:51.159300    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:52.319820 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:52.319792    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:53.222481 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:53.222448    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:54.106886 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:54.106857    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:55.108276 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:55.108249    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:55.541019 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:55.540988    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:04:55.803383 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:55.803299    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-5kjlz_75f2a1f7-95a7-462e-9ea7-23c51dd29599/console-operator/1.log"
Sep 18 18:04:55.805346 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:55.805311    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-5kjlz_75f2a1f7-95a7-462e-9ea7-23c51dd29599/console-operator/1.log"
Sep 18 18:04:55.809957 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:55.809938    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkcgk_c5d026b7-af6b-4899-85f4-6554c0f7cd52/ovn-acl-logging/0.log"
Sep 18 18:04:55.812218 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:55.812199    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkcgk_c5d026b7-af6b-4899-85f4-6554c0f7cd52/ovn-acl-logging/0.log"
Sep 18 18:04:59.166800 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:04:59.166772    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:00.104801 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:00.104766    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:00.289844 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:00.289812    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:00.388969 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:00.388892    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:00.499793 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:00.499764    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:00.633027 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:00.632977    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:00.811168 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:00.811107    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:01.105008 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:01.104922    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:01.286335 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:01.286309    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:01.647712 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:01.647682    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:02.101188 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:02.101082    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:03.101549 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:03.101523    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:03.523765 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:03.523733    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:04.706106 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:04.706073    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:04.876713 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:04.876679    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:06.288284 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:06.288251    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:07.426015 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:07.425984    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:08.506348 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:08.506314    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:08.628282 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:08.628220    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:09.389541 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:09.389504    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:09.724334 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:09.724292    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:10.111560 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:10.111465    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:11.360035 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:11.359921    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:12.302420 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:12.302391    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:13.212364 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:13.212336    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:26.338263 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:26.338229    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-a39784/autores-model-a39784-kserve-6b499d5459-d9rr2"]
Sep 18 18:05:26.341735 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:26.341712    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-a39784/autores-model-a39784-kserve-6b499d5459-d9rr2"
Sep 18 18:05:26.344217 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:26.344194    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-a39784\"/\"kube-root-ca.crt\""
Sep 18 18:05:26.344868 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:26.344850    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-a39784\"/\"openshift-service-ca.crt\""
Sep 18 18:05:26.345178 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:26.345161    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-a39784\"/\"default-dockercfg-nwxgb\""
Sep 18 18:05:26.345178 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:26.345174    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-a39784\"/\"autores-model-a39784-kserve-self-signed-certs\""
Sep 18 18:05:26.356177 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:26.356153    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-a39784/autores-model-a39784-kserve-6b499d5459-d9rr2"]
Sep 18 18:05:26.424318 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:26.424282    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b3a110f-8419-42cc-88a4-00d77bf40766-tmp-dir\") pod \"autores-model-a39784-kserve-6b499d5459-d9rr2\" (UID: \"7b3a110f-8419-42cc-88a4-00d77bf40766\") " pod="ai-tenant-e2e-autores-a39784/autores-model-a39784-kserve-6b499d5459-d9rr2"
Sep 18 18:05:26.424496 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:26.424330    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b3a110f-8419-42cc-88a4-00d77bf40766-dshm\") pod \"autores-model-a39784-kserve-6b499d5459-d9rr2\" (UID: \"7b3a110f-8419-42cc-88a4-00d77bf40766\") " pod="ai-tenant-e2e-autores-a39784/autores-model-a39784-kserve-6b499d5459-d9rr2"
Sep 18 18:05:26.424496 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:26.424451    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b3a110f-8419-42cc-88a4-00d77bf40766-home\") pod \"autores-model-a39784-kserve-6b499d5459-d9rr2\" (UID: \"7b3a110f-8419-42cc-88a4-00d77bf40766\") " pod="ai-tenant-e2e-autores-a39784/autores-model-a39784-kserve-6b499d5459-d9rr2"
Sep 18 18:05:26.424578 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:26.424505    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b3a110f-8419-42cc-88a4-00d77bf40766-model-cache\") pod \"autores-model-a39784-kserve-6b499d5459-d9rr2\" (UID: \"7b3a110f-8419-42cc-88a4-00d77bf40766\") " pod="ai-tenant-e2e-autores-a39784/autores-model-a39784-kserve-6b499d5459-d9rr2"
Sep 18 18:05:26.424578 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:26.424562    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b3a110f-8419-42cc-88a4-00d77bf40766-tls-certs\") pod \"autores-model-a39784-kserve-6b499d5459-d9rr2\" (UID: \"7b3a110f-8419-42cc-88a4-00d77bf40766\") " pod="ai-tenant-e2e-autores-a39784/autores-model-a39784-kserve-6b499d5459-d9rr2"
Sep 18 18:05:26.424645 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:26.424596    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmhlq\" (UniqueName: \"kubernetes.io/projected/7b3a110f-8419-42cc-88a4-00d77bf40766-kube-api-access-gmhlq\") pod \"autores-model-a39784-kserve-6b499d5459-d9rr2\" (UID: \"7b3a110f-8419-42cc-88a4-00d77bf40766\") " pod="ai-tenant-e2e-autores-a39784/autores-model-a39784-kserve-6b499d5459-d9rr2"
Sep 18 18:05:26.525166 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:26.525104    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b3a110f-8419-42cc-88a4-00d77bf40766-home\") pod \"autores-model-a39784-kserve-6b499d5459-d9rr2\" (UID: \"7b3a110f-8419-42cc-88a4-00d77bf40766\") " pod="ai-tenant-e2e-autores-a39784/autores-model-a39784-kserve-6b499d5459-d9rr2"
Sep 18 18:05:26.525337 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:26.525211    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b3a110f-8419-42cc-88a4-00d77bf40766-model-cache\") pod \"autores-model-a39784-kserve-6b499d5459-d9rr2\" (UID: \"7b3a110f-8419-42cc-88a4-00d77bf40766\") " pod="ai-tenant-e2e-autores-a39784/autores-model-a39784-kserve-6b499d5459-d9rr2"
Sep 18 18:05:26.525337 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:26.525235    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b3a110f-8419-42cc-88a4-00d77bf40766-tls-certs\") pod \"autores-model-a39784-kserve-6b499d5459-d9rr2\" (UID: \"7b3a110f-8419-42cc-88a4-00d77bf40766\") " pod="ai-tenant-e2e-autores-a39784/autores-model-a39784-kserve-6b499d5459-d9rr2"
Sep 18 18:05:26.525337 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:26.525260    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmhlq\" (UniqueName: \"kubernetes.io/projected/7b3a110f-8419-42cc-88a4-00d77bf40766-kube-api-access-gmhlq\") pod \"autores-model-a39784-kserve-6b499d5459-d9rr2\" (UID: \"7b3a110f-8419-42cc-88a4-00d77bf40766\") " pod="ai-tenant-e2e-autores-a39784/autores-model-a39784-kserve-6b499d5459-d9rr2"
Sep 18 18:05:26.525510 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:26.525366    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b3a110f-8419-42cc-88a4-00d77bf40766-tmp-dir\") pod \"autores-model-a39784-kserve-6b499d5459-d9rr2\" (UID: \"7b3a110f-8419-42cc-88a4-00d77bf40766\") " pod="ai-tenant-e2e-autores-a39784/autores-model-a39784-kserve-6b499d5459-d9rr2"
Sep 18 18:05:26.525510 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:26.525412    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b3a110f-8419-42cc-88a4-00d77bf40766-dshm\") pod \"autores-model-a39784-kserve-6b499d5459-d9rr2\" (UID: \"7b3a110f-8419-42cc-88a4-00d77bf40766\") " pod="ai-tenant-e2e-autores-a39784/autores-model-a39784-kserve-6b499d5459-d9rr2"
Sep 18 18:05:26.525620 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:26.525599    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:26.525681 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:26.525618    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b3a110f-8419-42cc-88a4-00d77bf40766-home\") pod \"autores-model-a39784-kserve-6b499d5459-d9rr2\" (UID: \"7b3a110f-8419-42cc-88a4-00d77bf40766\") " pod="ai-tenant-e2e-autores-a39784/autores-model-a39784-kserve-6b499d5459-d9rr2"
Sep 18 18:05:26.525739 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:26.525681    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b3a110f-8419-42cc-88a4-00d77bf40766-model-cache\") pod \"autores-model-a39784-kserve-6b499d5459-d9rr2\" (UID: \"7b3a110f-8419-42cc-88a4-00d77bf40766\") " pod="ai-tenant-e2e-autores-a39784/autores-model-a39784-kserve-6b499d5459-d9rr2"
Sep 18 18:05:26.525794 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:26.525752    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b3a110f-8419-42cc-88a4-00d77bf40766-tmp-dir\") pod \"autores-model-a39784-kserve-6b499d5459-d9rr2\" (UID: \"7b3a110f-8419-42cc-88a4-00d77bf40766\") " pod="ai-tenant-e2e-autores-a39784/autores-model-a39784-kserve-6b499d5459-d9rr2"
Sep 18 18:05:26.527582 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:26.527557    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b3a110f-8419-42cc-88a4-00d77bf40766-dshm\") pod \"autores-model-a39784-kserve-6b499d5459-d9rr2\" (UID: \"7b3a110f-8419-42cc-88a4-00d77bf40766\") " pod="ai-tenant-e2e-autores-a39784/autores-model-a39784-kserve-6b499d5459-d9rr2"
Sep 18 18:05:26.527831 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:26.527807    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b3a110f-8419-42cc-88a4-00d77bf40766-tls-certs\") pod \"autores-model-a39784-kserve-6b499d5459-d9rr2\" (UID: \"7b3a110f-8419-42cc-88a4-00d77bf40766\") " pod="ai-tenant-e2e-autores-a39784/autores-model-a39784-kserve-6b499d5459-d9rr2"
Sep 18 18:05:26.534242 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:26.534224    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmhlq\" (UniqueName: \"kubernetes.io/projected/7b3a110f-8419-42cc-88a4-00d77bf40766-kube-api-access-gmhlq\") pod \"autores-model-a39784-kserve-6b499d5459-d9rr2\" (UID: \"7b3a110f-8419-42cc-88a4-00d77bf40766\") " pod="ai-tenant-e2e-autores-a39784/autores-model-a39784-kserve-6b499d5459-d9rr2"
Sep 18 18:05:26.650984 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:26.650961    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-a39784/autores-model-a39784-kserve-6b499d5459-d9rr2"
Sep 18 18:05:26.940410 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:26.940385    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-a39784/autores-model-a39784-kserve-6b499d5459-d9rr2"]
Sep 18 18:05:26.943205 ip-10-0-140-54 kubenswrapper[2561]: W0918 18:05:26.943177    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b3a110f_8419_42cc_88a4_00d77bf40766.slice/crio-043d17acffdaa096d6c4a167517b9d695b7308d332dcb5f867e7c4fefca5fa15 WatchSource:0}: Error finding container 043d17acffdaa096d6c4a167517b9d695b7308d332dcb5f867e7c4fefca5fa15: Status 404 returned error can't find the container with id 043d17acffdaa096d6c4a167517b9d695b7308d332dcb5f867e7c4fefca5fa15
Sep 18 18:05:27.300741 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:27.300652    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-a39784/autores-model-a39784-kserve-6b499d5459-d9rr2" event={"ID":"7b3a110f-8419-42cc-88a4-00d77bf40766","Type":"ContainerStarted","Data":"2a2a9d152fe5d428e2b760a92d41b6eac21b8fbeb5d41737ab75a0ec9113593d"}
Sep 18 18:05:27.300741 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:27.300694    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-a39784/autores-model-a39784-kserve-6b499d5459-d9rr2" event={"ID":"7b3a110f-8419-42cc-88a4-00d77bf40766","Type":"ContainerStarted","Data":"043d17acffdaa096d6c4a167517b9d695b7308d332dcb5f867e7c4fefca5fa15"}
Sep 18 18:05:27.326130 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:27.326065    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-a39784/autores-model-a39784-kserve-6b499d5459-d9rr2" podStartSLOduration=1.326050572 podStartE2EDuration="1.326050572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:05:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:05:27.323400871 +0000 UTC m=+931.950532490" watchObservedRunningTime="2026-09-18 18:05:27.326050572 +0000 UTC m=+931.953182193"
Sep 18 18:05:31.800438 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:31.800402    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:33.035969 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:33.035933    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:34.334882 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:34.334844    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:35.707695 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:35.707666    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:36.651217 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:36.651179    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-a39784/autores-model-a39784-kserve-6b499d5459-d9rr2"
Sep 18 18:05:36.651425 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:36.651253    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-a39784/autores-model-a39784-kserve-6b499d5459-d9rr2"
Sep 18 18:05:36.664011 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:36.663990    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-a39784/autores-model-a39784-kserve-6b499d5459-d9rr2"
Sep 18 18:05:36.996822 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:36.996733    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:37.364554 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:37.364467    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-a39784/autores-model-a39784-kserve-6b499d5459-d9rr2"
Sep 18 18:05:38.772832 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:38.772791    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:39.856102 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:39.856069    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:40.896657 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:40.896628    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:41.066605 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:41.066571    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:42.084207 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:42.084174    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:43.156433 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:43.156402    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:44.179627 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:44.179597    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:46.211690 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:46.211660    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:47.359260 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:47.359233    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:47.502577 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:47.502549    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:47.626195 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:47.626107    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:47.746457 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:47.746421    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:47.916260 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:47.916228    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:48.697595 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:48.697563    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:49.116319 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:49.116244    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:49.532857 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:49.532828    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:49.801110 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:49.801035    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:49.979858 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:49.979827    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:50.085974 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:50.085891    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:50.797093 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:50.797053    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:51.000829 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:51.000789    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:52.220692 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:52.220663    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:52.920910 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:52.920880    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:53.071509 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:53.071483    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:53.710743 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:53.710713    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:55.073509 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:55.073479    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:56.431363 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:56.431331    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:56.615398 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:56.615365    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:57.193319 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:57.193291    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:57.400089 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:57.400044    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:58.000665 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:58.000636    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:05:59.054123 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:05:59.054096    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:00.284887 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:00.284850    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:01.381012 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:01.380979    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:02.708293 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:02.708264    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:04.203647 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:04.203620    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:05.681317 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:05.681281    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:06.891533 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:06.891498    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:08.725814 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:08.725783    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-a39784-c6f678db4-7cs9h"]
Sep 18 18:06:08.726285 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:08.726040    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a39784-c6f678db4-7cs9h" podUID="2f071103-1087-4b52-b526-99219526e36c" containerName="maas-api" containerID="cri-o://240ef21cf11dc7ff673f87d8fa4232d11b3a75efeff7e35d41b94619e7e846c4" gracePeriod=30
Sep 18 18:06:08.742292 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:08.742267    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:08.849688 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:08.849659    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:08.957675 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:08.957641    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:08.987780 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:08.987723    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a39784-c6f678db4-7cs9h"
Sep 18 18:06:09.132345 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:09.132316    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:09.139424 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:09.139397    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f071103-1087-4b52-b526-99219526e36c-metrics-certs\") pod \"2f071103-1087-4b52-b526-99219526e36c\" (UID: \"2f071103-1087-4b52-b526-99219526e36c\") "
Sep 18 18:06:09.139424 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:09.139426    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f071103-1087-4b52-b526-99219526e36c-maas-api-tls\") pod \"2f071103-1087-4b52-b526-99219526e36c\" (UID: \"2f071103-1087-4b52-b526-99219526e36c\") "
Sep 18 18:06:09.139652 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:09.139539    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f071103-1087-4b52-b526-99219526e36c-openshift-service-ca\") pod \"2f071103-1087-4b52-b526-99219526e36c\" (UID: \"2f071103-1087-4b52-b526-99219526e36c\") "
Sep 18 18:06:09.139652 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:09.139581    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-64h7q\" (UniqueName: \"kubernetes.io/projected/2f071103-1087-4b52-b526-99219526e36c-kube-api-access-64h7q\") pod \"2f071103-1087-4b52-b526-99219526e36c\" (UID: \"2f071103-1087-4b52-b526-99219526e36c\") "
Sep 18 18:06:09.140058 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:09.140027    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f071103-1087-4b52-b526-99219526e36c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2f071103-1087-4b52-b526-99219526e36c" (UID: "2f071103-1087-4b52-b526-99219526e36c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:06:09.141691 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:09.141665    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f071103-1087-4b52-b526-99219526e36c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2f071103-1087-4b52-b526-99219526e36c" (UID: "2f071103-1087-4b52-b526-99219526e36c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:06:09.141691 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:09.141673    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f071103-1087-4b52-b526-99219526e36c-kube-api-access-64h7q" (OuterVolumeSpecName: "kube-api-access-64h7q") pod "2f071103-1087-4b52-b526-99219526e36c" (UID: "2f071103-1087-4b52-b526-99219526e36c"). InnerVolumeSpecName "kube-api-access-64h7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:06:09.142168 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:09.142143    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f071103-1087-4b52-b526-99219526e36c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2f071103-1087-4b52-b526-99219526e36c" (UID: "2f071103-1087-4b52-b526-99219526e36c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:06:09.241273 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:09.241206    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f071103-1087-4b52-b526-99219526e36c-metrics-certs\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:06:09.241273 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:09.241229    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f071103-1087-4b52-b526-99219526e36c-maas-api-tls\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:06:09.241273 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:09.241237    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f071103-1087-4b52-b526-99219526e36c-openshift-service-ca\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:06:09.241273 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:09.241246    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64h7q\" (UniqueName: \"kubernetes.io/projected/2f071103-1087-4b52-b526-99219526e36c-kube-api-access-64h7q\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:06:09.486542 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:09.486508    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:09.526044 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:09.525958    2561 generic.go:356] "Generic (PLEG): container finished" podID="2f071103-1087-4b52-b526-99219526e36c" containerID="240ef21cf11dc7ff673f87d8fa4232d11b3a75efeff7e35d41b94619e7e846c4" exitCode=0
Sep 18 18:06:09.526044 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:09.526004    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a39784-c6f678db4-7cs9h"
Sep 18 18:06:09.526304 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:09.526044    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a39784-c6f678db4-7cs9h" event={"ID":"2f071103-1087-4b52-b526-99219526e36c","Type":"ContainerDied","Data":"240ef21cf11dc7ff673f87d8fa4232d11b3a75efeff7e35d41b94619e7e846c4"}
Sep 18 18:06:09.526304 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:09.526096    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a39784-c6f678db4-7cs9h" event={"ID":"2f071103-1087-4b52-b526-99219526e36c","Type":"ContainerDied","Data":"9528f947e96b67f1d032dfe4d3fe0a3b0f7ca9a0c1af5a879662df5c5a9a79ec"}
Sep 18 18:06:09.526304 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:09.526131    2561 scope.go:117] "RemoveContainer" containerID="240ef21cf11dc7ff673f87d8fa4232d11b3a75efeff7e35d41b94619e7e846c4"
Sep 18 18:06:09.537622 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:09.537606    2561 scope.go:117] "RemoveContainer" containerID="240ef21cf11dc7ff673f87d8fa4232d11b3a75efeff7e35d41b94619e7e846c4"
Sep 18 18:06:09.538034 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:06:09.538009    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"240ef21cf11dc7ff673f87d8fa4232d11b3a75efeff7e35d41b94619e7e846c4\": container with ID starting with 240ef21cf11dc7ff673f87d8fa4232d11b3a75efeff7e35d41b94619e7e846c4 not found: ID does not exist" containerID="240ef21cf11dc7ff673f87d8fa4232d11b3a75efeff7e35d41b94619e7e846c4"
Sep 18 18:06:09.538110 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:09.538045    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"240ef21cf11dc7ff673f87d8fa4232d11b3a75efeff7e35d41b94619e7e846c4"} err="failed to get container status \"240ef21cf11dc7ff673f87d8fa4232d11b3a75efeff7e35d41b94619e7e846c4\": rpc error: code = NotFound desc = could not find container \"240ef21cf11dc7ff673f87d8fa4232d11b3a75efeff7e35d41b94619e7e846c4\": container with ID starting with 240ef21cf11dc7ff673f87d8fa4232d11b3a75efeff7e35d41b94619e7e846c4 not found: ID does not exist"
Sep 18 18:06:09.550911 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:09.550883    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-a39784-c6f678db4-7cs9h"]
Sep 18 18:06:09.556763 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:09.556744    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-a39784-c6f678db4-7cs9h"]
Sep 18 18:06:09.657011 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:09.656984    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:09.862135 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:09.862042    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f071103-1087-4b52-b526-99219526e36c" path="/var/lib/kubelet/pods/2f071103-1087-4b52-b526-99219526e36c/volumes"
Sep 18 18:06:10.982690 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:10.982657    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:12.409369 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:12.409340    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:13.552619 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:13.552591    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:14.463328 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:14.463295    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-a39784/autores-model-a39784-kserve-6b499d5459-d9rr2"]
Sep 18 18:06:14.588800 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:14.588769    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:14.699079 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:14.699048    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:15.648869 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:15.648806    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:16.893540 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:16.893507    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:18.218751 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:18.218721    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:19.382551 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:19.382519    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:20.395069 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:20.395036    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:21.659105 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:21.659072    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:22.822286 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:22.822253    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:24.086526 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:24.086495    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:25.025673 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:25.025633    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:25.817462 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:25.817426    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:25.981936 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:25.981890    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:26.200780 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:26.200750    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:26.420899 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:26.420869    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:26.601309 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:26.601231    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:26.877229 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:26.877151    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:27.005501 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:27.005472    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:27.416637 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:27.416610    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:27.525715 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:27.525675    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:28.444391 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:28.444363    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:29.006537 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:29.006506    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:29.482378 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:29.482341    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-a39784/autores-model-a39784-kserve-6b499d5459-d9rr2" podUID="7b3a110f-8419-42cc-88a4-00d77bf40766" containerName="main" containerID="cri-o://2a2a9d152fe5d428e2b760a92d41b6eac21b8fbeb5d41737ab75a0ec9113593d" gracePeriod=45
Sep 18 18:06:29.634481 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:29.634450    2561 generic.go:356] "Generic (PLEG): container finished" podID="7b3a110f-8419-42cc-88a4-00d77bf40766" containerID="2a2a9d152fe5d428e2b760a92d41b6eac21b8fbeb5d41737ab75a0ec9113593d" exitCode=0
Sep 18 18:06:29.634630 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:29.634538    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-a39784/autores-model-a39784-kserve-6b499d5459-d9rr2" event={"ID":"7b3a110f-8419-42cc-88a4-00d77bf40766","Type":"ContainerDied","Data":"2a2a9d152fe5d428e2b760a92d41b6eac21b8fbeb5d41737ab75a0ec9113593d"}
Sep 18 18:06:29.648988 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:29.648959    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:29.746645 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:29.746591    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-a39784/autores-model-a39784-kserve-6b499d5459-d9rr2"
Sep 18 18:06:29.846734 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:29.846703    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b3a110f-8419-42cc-88a4-00d77bf40766-tls-certs\") pod \"7b3a110f-8419-42cc-88a4-00d77bf40766\" (UID: \"7b3a110f-8419-42cc-88a4-00d77bf40766\") "
Sep 18 18:06:29.846911 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:29.846739    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gmhlq\" (UniqueName: \"kubernetes.io/projected/7b3a110f-8419-42cc-88a4-00d77bf40766-kube-api-access-gmhlq\") pod \"7b3a110f-8419-42cc-88a4-00d77bf40766\" (UID: \"7b3a110f-8419-42cc-88a4-00d77bf40766\") "
Sep 18 18:06:29.846911 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:29.846821    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b3a110f-8419-42cc-88a4-00d77bf40766-model-cache\") pod \"7b3a110f-8419-42cc-88a4-00d77bf40766\" (UID: \"7b3a110f-8419-42cc-88a4-00d77bf40766\") "
Sep 18 18:06:29.846911 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:29.846876    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b3a110f-8419-42cc-88a4-00d77bf40766-home\") pod \"7b3a110f-8419-42cc-88a4-00d77bf40766\" (UID: \"7b3a110f-8419-42cc-88a4-00d77bf40766\") "
Sep 18 18:06:29.847083 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:29.846923    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b3a110f-8419-42cc-88a4-00d77bf40766-dshm\") pod \"7b3a110f-8419-42cc-88a4-00d77bf40766\" (UID: \"7b3a110f-8419-42cc-88a4-00d77bf40766\") "
Sep 18 18:06:29.847083 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:29.846977    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b3a110f-8419-42cc-88a4-00d77bf40766-tmp-dir\") pod \"7b3a110f-8419-42cc-88a4-00d77bf40766\" (UID: \"7b3a110f-8419-42cc-88a4-00d77bf40766\") "
Sep 18 18:06:29.847209 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:29.847174    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b3a110f-8419-42cc-88a4-00d77bf40766-model-cache" (OuterVolumeSpecName: "model-cache") pod "7b3a110f-8419-42cc-88a4-00d77bf40766" (UID: "7b3a110f-8419-42cc-88a4-00d77bf40766"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:06:29.847209 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:29.847188    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b3a110f-8419-42cc-88a4-00d77bf40766-home" (OuterVolumeSpecName: "home") pod "7b3a110f-8419-42cc-88a4-00d77bf40766" (UID: "7b3a110f-8419-42cc-88a4-00d77bf40766"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:06:29.847406 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:29.847358    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b3a110f-8419-42cc-88a4-00d77bf40766-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7b3a110f-8419-42cc-88a4-00d77bf40766" (UID: "7b3a110f-8419-42cc-88a4-00d77bf40766"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:06:29.847406 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:29.847376    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b3a110f-8419-42cc-88a4-00d77bf40766-model-cache\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:06:29.847510 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:29.847417    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b3a110f-8419-42cc-88a4-00d77bf40766-home\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:06:29.849214 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:29.849166    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b3a110f-8419-42cc-88a4-00d77bf40766-dshm" (OuterVolumeSpecName: "dshm") pod "7b3a110f-8419-42cc-88a4-00d77bf40766" (UID: "7b3a110f-8419-42cc-88a4-00d77bf40766"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:06:29.849330 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:29.849235    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b3a110f-8419-42cc-88a4-00d77bf40766-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7b3a110f-8419-42cc-88a4-00d77bf40766" (UID: "7b3a110f-8419-42cc-88a4-00d77bf40766"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:06:29.849330 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:29.849277    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b3a110f-8419-42cc-88a4-00d77bf40766-kube-api-access-gmhlq" (OuterVolumeSpecName: "kube-api-access-gmhlq") pod "7b3a110f-8419-42cc-88a4-00d77bf40766" (UID: "7b3a110f-8419-42cc-88a4-00d77bf40766"). InnerVolumeSpecName "kube-api-access-gmhlq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:06:29.948922 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:29.948891    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b3a110f-8419-42cc-88a4-00d77bf40766-dshm\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:06:29.948922 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:29.948919    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b3a110f-8419-42cc-88a4-00d77bf40766-tmp-dir\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:06:29.948922 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:29.948927    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b3a110f-8419-42cc-88a4-00d77bf40766-tls-certs\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:06:29.949174 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:29.948936    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmhlq\" (UniqueName: \"kubernetes.io/projected/7b3a110f-8419-42cc-88a4-00d77bf40766-kube-api-access-gmhlq\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:06:30.641205 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:30.641173    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-a39784/autores-model-a39784-kserve-6b499d5459-d9rr2"
Sep 18 18:06:30.641205 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:30.641181    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-a39784/autores-model-a39784-kserve-6b499d5459-d9rr2" event={"ID":"7b3a110f-8419-42cc-88a4-00d77bf40766","Type":"ContainerDied","Data":"043d17acffdaa096d6c4a167517b9d695b7308d332dcb5f867e7c4fefca5fa15"}
Sep 18 18:06:30.641690 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:30.641226    2561 scope.go:117] "RemoveContainer" containerID="2a2a9d152fe5d428e2b760a92d41b6eac21b8fbeb5d41737ab75a0ec9113593d"
Sep 18 18:06:30.665485 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:30.665452    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-a39784/autores-model-a39784-kserve-6b499d5459-d9rr2"]
Sep 18 18:06:30.668454 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:30.668433    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-a39784/autores-model-a39784-kserve-6b499d5459-d9rr2"]
Sep 18 18:06:30.704687 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:30.704628    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:31.831462 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:31.831411    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:31.861800 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:31.861767    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b3a110f-8419-42cc-88a4-00d77bf40766" path="/var/lib/kubelet/pods/7b3a110f-8419-42cc-88a4-00d77bf40766/volumes"
Sep 18 18:06:36.548982 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:36.548945    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:36.708545 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:36.708515    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:36.842625 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:36.842533    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:37.322635 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:37.322603    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:47.253724 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:47.253686    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:47.382246 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:47.382213    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:48.053998 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:48.053968    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:52.594243 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:52.594212    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:52.904217 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:52.904187    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:53.551430 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:53.551396    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:53.779902 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:53.779842    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:54.140895 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:54.140857    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:56.895492 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:56.895465    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:57.122892 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:57.122863    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:57.299328 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:57.299299    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:58.483176 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:58.483144    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:06:59.564971 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:06:59.564937    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:07:00.905595 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:07:00.905550    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:07:01.816589 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:07:01.816559    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:07:01.914636 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:07:01.914606    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:07:02.072160 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:07:02.072047    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:07:02.560628 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:07:02.560571    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:07:02.721816 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:07:02.721787    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:07:03.343860 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:07:03.343828    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:07:11.495403 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:07:11.495370    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:07:11.674410 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:07:11.674376    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:07:22.838297 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:07:22.838262    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:07:24.975518 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:07:24.975485    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:07:25.086439 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:07:25.086411    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:07:26.036052 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:07:26.036022    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:07:26.842068 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:07:26.842036    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:07:26.950143 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:07:26.950091    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:07:27.615042 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:07:27.615010    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:07:28.190505 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:07:28.190475    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:07:29.217289 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:07:29.217259    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:07:30.323988 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:07:30.323955    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:07:34.771981 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:07:34.771881    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:07:34.972431 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:07:34.972399    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:07:35.266927 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:07:35.266898    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:07:41.645879 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:07:41.645846    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:07:41.754256 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:07:41.754225    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:07:43.866256 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:07:43.866226    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-609aba/premium-w0-609aba-kserve-897b66447-b9bzg"]
Sep 18 18:07:43.866661 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:07:43.866648    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f071103-1087-4b52-b526-99219526e36c" containerName="maas-api"
Sep 18 18:07:43.866661 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:07:43.866662    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f071103-1087-4b52-b526-99219526e36c" containerName="maas-api"
Sep 18 18:07:43.866743 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:07:43.866682    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b3a110f-8419-42cc-88a4-00d77bf40766" containerName="main"
Sep 18 18:07:43.866743 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:07:43.866688    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b3a110f-8419-42cc-88a4-00d77bf40766" containerName="main"
Sep 18 18:07:43.866811 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:07:43.866752    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f071103-1087-4b52-b526-99219526e36c" containerName="maas-api"
Sep 18 18:07:43.866811 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:07:43.866760    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b3a110f-8419-42cc-88a4-00d77bf40766" containerName="main"
Sep 18 18:07:43.870045 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:07:43.870029    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-609aba/premium-w0-609aba-kserve-897b66447-b9bzg"
Sep 18 18:07:43.877553 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:07:43.877536    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-609aba\"/\"openshift-service-ca.crt\""
Sep 18 18:07:43.877662 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:07:43.877639    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-609aba\"/\"default-dockercfg-4hgmq\""
Sep 18 18:07:43.877723 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:07:43.877648    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-609aba\"/\"premium-w0-609aba-kserve-self-signed-certs\""
Sep 18 18:07:43.877784 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:07:43.877774    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-609aba\"/\"kube-root-ca.crt\""
Sep 18 18:07:43.892794 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:07:43.892771    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-609aba/premium-w0-609aba-kserve-897b66447-b9bzg"]
Sep 18 18:07:43.925449 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:07:43.925425    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:07:43.949309 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:07:43.949281    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a870971-8984-4e51-822a-1726f7f1a85d-tls-certs\") pod \"premium-w0-609aba-kserve-897b66447-b9bzg\" (UID: \"3a870971-8984-4e51-822a-1726f7f1a85d\") " pod="e2e-models-e2e-worker-w0-609aba/premium-w0-609aba-kserve-897b66447-b9bzg"
Sep 18 18:07:43.949413 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:07:43.949319    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a870971-8984-4e51-822a-1726f7f1a85d-tmp-dir\") pod \"premium-w0-609aba-kserve-897b66447-b9bzg\" (UID: \"3a870971-8984-4e51-822a-1726f7f1a85d\") " pod="e2e-models-e2e-worker-w0-609aba/premium-w0-609aba-kserve-897b66447-b9bzg"
Sep 18 18:07:43.949454 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:07:43.949417    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a870971-8984-4e51-822a-1726f7f1a85d-dshm\") pod \"premium-w0-609aba-kserve-897b66447-b9bzg\" (UID: \"3a870971-8984-4e51-822a-1726f7f1a85d\") " pod="e2e-models-e2e-worker-w0-609aba/premium-w0-609aba-kserve-897b66447-b9bzg"
Sep 18 18:07:43.949454 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:07:43.949441    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a870971-8984-4e51-822a-1726f7f1a85d-model-cache\") pod \"premium-w0-609aba-kserve-897b66447-b9bzg\" (UID: \"3a870971-8984-4e51-822a-1726f7f1a85d\") " pod="e2e-models-e2e-worker-w0-609aba/premium-w0-609aba-kserve-897b66447-b9bzg"
Sep 18 18:07:43.949525 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:07:43.949460    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a870971-8984-4e51-822a-1726f7f1a85d-home\") pod \"premium-w0-609aba-kserve-897b66447-b9bzg\" (UID: \"3a870971-8984-4e51-822a-1726f7f1a85d\") " pod="e2e-models-e2e-worker-w0-609aba/premium-w0-609aba-kserve-897b66447-b9bzg"
Sep 18 18:07:43.949559 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:07:43.949534    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chn2x\" (UniqueName: \"kubernetes.io/projected/3a870971-8984-4e51-822a-1726f7f1a85d-kube-api-access-chn2x\") pod \"premium-w0-609aba-kserve-897b66447-b9bzg\" (UID: \"3a870971-8984-4e51-822a-1726f7f1a85d\") " pod="e2e-models-e2e-worker-w0-609aba/premium-w0-609aba-kserve-897b66447-b9bzg"
Sep 18 18:07:44.050110 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:07:44.050076    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a870971-8984-4e51-822a-1726f7f1a85d-dshm\") pod \"premium-w0-609aba-kserve-897b66447-b9bzg\" (UID: \"3a870971-8984-4e51-822a-1726f7f1a85d\") " pod="e2e-models-e2e-worker-w0-609aba/premium-w0-609aba-kserve-897b66447-b9bzg"
Sep 18 18:07:44.050110 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:07:44.050112    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a870971-8984-4e51-822a-1726f7f1a85d-model-cache\") pod \"premium-w0-609aba-kserve-897b66447-b9bzg\" (UID: \"3a870971-8984-4e51-822a-1726f7f1a85d\") " pod="e2e-models-e2e-worker-w0-609aba/premium-w0-609aba-kserve-897b66447-b9bzg"
Sep 18 18:07:44.050382 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:07:44.050142    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a870971-8984-4e51-822a-1726f7f1a85d-home\") pod \"premium-w0-609aba-kserve-897b66447-b9bzg\" (UID: \"3a870971-8984-4e51-822a-1726f7f1a85d\") " pod="e2e-models-e2e-worker-w0-609aba/premium-w0-609aba-kserve-897b66447-b9bzg"
Sep 18 18:07:44.050382 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:07:44.050174    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chn2x\" (UniqueName: \"kubernetes.io/projected/3a870971-8984-4e51-822a-1726f7f1a85d-kube-api-access-chn2x\") pod \"premium-w0-609aba-kserve-897b66447-b9bzg\" (UID: \"3a870971-8984-4e51-822a-1726f7f1a85d\") " pod="e2e-models-e2e-worker-w0-609aba/premium-w0-609aba-kserve-897b66447-b9bzg"
Sep 18 18:07:44.050382 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:07:44.050240    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a870971-8984-4e51-822a-1726f7f1a85d-tls-certs\") pod \"premium-w0-609aba-kserve-897b66447-b9bzg\" (UID: \"3a870971-8984-4e51-822a-1726f7f1a85d\") " pod="e2e-models-e2e-worker-w0-609aba/premium-w0-609aba-kserve-897b66447-b9bzg"
Sep 18 18:07:44.050382 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:07:44.050274    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a870971-8984-4e51-822a-1726f7f1a85d-tmp-dir\") pod \"premium-w0-609aba-kserve-897b66447-b9bzg\" (UID: \"3a870971-8984-4e51-822a-1726f7f1a85d\") " pod="e2e-models-e2e-worker-w0-609aba/premium-w0-609aba-kserve-897b66447-b9bzg"
Sep 18 18:07:44.050604 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:07:44.050571    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a870971-8984-4e51-822a-1726f7f1a85d-model-cache\") pod \"premium-w0-609aba-kserve-897b66447-b9bzg\" (UID: \"3a870971-8984-4e51-822a-1726f7f1a85d\") " pod="e2e-models-e2e-worker-w0-609aba/premium-w0-609aba-kserve-897b66447-b9bzg"
Sep 18 18:07:44.050656 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:07:44.050606    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a870971-8984-4e51-822a-1726f7f1a85d-home\") pod \"premium-w0-609aba-kserve-897b66447-b9bzg\" (UID: \"3a870971-8984-4e51-822a-1726f7f1a85d\") " pod="e2e-models-e2e-worker-w0-609aba/premium-w0-609aba-kserve-897b66447-b9bzg"
Sep 18 18:07:44.050714 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:07:44.050695    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a870971-8984-4e51-822a-1726f7f1a85d-tmp-dir\") pod \"premium-w0-609aba-kserve-897b66447-b9bzg\" (UID: \"3a870971-8984-4e51-822a-1726f7f1a85d\") " pod="e2e-models-e2e-worker-w0-609aba/premium-w0-609aba-kserve-897b66447-b9bzg"
Sep 18 18:07:44.052317 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:07:44.052293    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a870971-8984-4e51-822a-1726f7f1a85d-dshm\") pod \"premium-w0-609aba-kserve-897b66447-b9bzg\" (UID: \"3a870971-8984-4e51-822a-1726f7f1a85d\") " pod="e2e-models-e2e-worker-w0-609aba/premium-w0-609aba-kserve-897b66447-b9bzg"
Sep 18 18:07:44.052637 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:07:44.052618    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a870971-8984-4e51-822a-1726f7f1a85d-tls-certs\") pod \"premium-w0-609aba-kserve-897b66447-b9bzg\" (UID: \"3a870971-8984-4e51-822a-1726f7f1a85d\") " pod="e2e-models-e2e-worker-w0-609aba/premium-w0-609aba-kserve-897b66447-b9bzg"
Sep 18 18:07:44.062811 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:07:44.062793    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chn2x\" (UniqueName: \"kubernetes.io/projected/3a870971-8984-4e51-822a-1726f7f1a85d-kube-api-access-chn2x\") pod \"premium-w0-609aba-kserve-897b66447-b9bzg\" (UID: \"3a870971-8984-4e51-822a-1726f7f1a85d\") " pod="e2e-models-e2e-worker-w0-609aba/premium-w0-609aba-kserve-897b66447-b9bzg"
Sep 18 18:07:44.179735 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:07:44.179708    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-609aba/premium-w0-609aba-kserve-897b66447-b9bzg"
Sep 18 18:07:44.457403 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:07:44.457327    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-609aba/premium-w0-609aba-kserve-897b66447-b9bzg"]
Sep 18 18:07:44.460613 ip-10-0-140-54 kubenswrapper[2561]: W0918 18:07:44.460586    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3a870971_8984_4e51_822a_1726f7f1a85d.slice/crio-e2d0792925252c176f4b143bba28b6c0c64339eebdcf8b5e3e01547a70392413 WatchSource:0}: Error finding container e2d0792925252c176f4b143bba28b6c0c64339eebdcf8b5e3e01547a70392413: Status 404 returned error can't find the container with id e2d0792925252c176f4b143bba28b6c0c64339eebdcf8b5e3e01547a70392413
Sep 18 18:07:45.021930 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:07:45.021892    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-609aba/premium-w0-609aba-kserve-897b66447-b9bzg" event={"ID":"3a870971-8984-4e51-822a-1726f7f1a85d","Type":"ContainerStarted","Data":"88907f86a40babfeaff18093b90f5a1cf0f1ca179967d39bf2e58c5e94f58752"}
Sep 18 18:07:45.022375 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:07:45.021940    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-609aba/premium-w0-609aba-kserve-897b66447-b9bzg" event={"ID":"3a870971-8984-4e51-822a-1726f7f1a85d","Type":"ContainerStarted","Data":"e2d0792925252c176f4b143bba28b6c0c64339eebdcf8b5e3e01547a70392413"}
Sep 18 18:07:45.044677 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:07:45.044619    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-609aba/premium-w0-609aba-kserve-897b66447-b9bzg" podStartSLOduration=2.044601206 podStartE2EDuration="2.044601206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:07:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:07:45.044097442 +0000 UTC m=+1069.671229062" watchObservedRunningTime="2026-09-18 18:07:45.044601206 +0000 UTC m=+1069.671732825"
Sep 18 18:07:45.539670 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:07:45.539638    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:07:46.352573 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:07:46.352540    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:07:54.180314 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:07:54.180190    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-609aba/premium-w0-609aba-kserve-897b66447-b9bzg"
Sep 18 18:07:54.180314 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:07:54.180286    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-609aba/premium-w0-609aba-kserve-897b66447-b9bzg"
Sep 18 18:07:54.192532 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:07:54.192510    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-609aba/premium-w0-609aba-kserve-897b66447-b9bzg"
Sep 18 18:07:54.254585 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:07:54.254542    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:07:54.471140 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:07:54.471042    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:07:54.604751 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:07:54.604721    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:07:55.083293 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:07:55.083267    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-609aba/premium-w0-609aba-kserve-897b66447-b9bzg"
Sep 18 18:07:55.356432 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:07:55.356311    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:07:55.551029 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:07:55.550995    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:07:57.153255 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:07:57.153221    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:08:03.859431 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:08:03.859396    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:08:04.879990 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:08:04.879950    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:08:05.054470 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:08:05.054426    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:08:05.611474 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:08:05.611405    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:08:07.738760 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:08:07.738726    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:08:14.143441 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:08:14.143409    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:08:14.320127 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:08:14.320097    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:08:14.464476 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:08:14.464446    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:08:15.016740 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:08:15.016699    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:08:16.050011 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:08:16.049982    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:08:28.083060 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:08:28.083028    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:08:28.322545 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:08:28.322512    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:08:28.502331 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:08:28.502300    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:08:28.715891 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:08:28.715860    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:08:30.658419 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:08:30.658353    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:08:32.603333 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:08:32.603289    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:08:34.005459 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:08:34.005433    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:08:35.509798 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:08:35.509766    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:08:36.827325 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:08:36.827288    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:08:36.989076 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:08:36.989034    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:08:37.477959 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:08:37.477928    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:08:37.890567 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:08:37.890488    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:08:38.160771 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:08:38.160739    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3ffa634f-984fdd89-p5rq4"]
Sep 18 18:08:38.164512 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:08:38.164496    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3ffa634f-984fdd89-p5rq4"
Sep 18 18:08:38.167050 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:08:38.167026    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-3ffa634f\""
Sep 18 18:08:38.167405 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:08:38.167389    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-3ffa634f-dockercfg-nz6dm\""
Sep 18 18:08:38.191473 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:08:38.191449    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3ffa634f-984fdd89-p5rq4"]
Sep 18 18:08:38.200982 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:08:38.200938    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:08:38.281425 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:08:38.281395    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/839fdbe8-fdae-4fe3-957a-482b1a996927-plugins-config-volume\") pod \"payload-processing-e2e-mt-3ffa634f-984fdd89-p5rq4\" (UID: \"839fdbe8-fdae-4fe3-957a-482b1a996927\") " pod="openshift-ingress/payload-processing-e2e-mt-3ffa634f-984fdd89-p5rq4"
Sep 18 18:08:38.281571 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:08:38.281444    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r4mq\" (UniqueName: \"kubernetes.io/projected/839fdbe8-fdae-4fe3-957a-482b1a996927-kube-api-access-7r4mq\") pod \"payload-processing-e2e-mt-3ffa634f-984fdd89-p5rq4\" (UID: \"839fdbe8-fdae-4fe3-957a-482b1a996927\") " pod="openshift-ingress/payload-processing-e2e-mt-3ffa634f-984fdd89-p5rq4"
Sep 18 18:08:38.382284 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:08:38.382252    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/839fdbe8-fdae-4fe3-957a-482b1a996927-plugins-config-volume\") pod \"payload-processing-e2e-mt-3ffa634f-984fdd89-p5rq4\" (UID: \"839fdbe8-fdae-4fe3-957a-482b1a996927\") " pod="openshift-ingress/payload-processing-e2e-mt-3ffa634f-984fdd89-p5rq4"
Sep 18 18:08:38.382436 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:08:38.382309    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r4mq\" (UniqueName: \"kubernetes.io/projected/839fdbe8-fdae-4fe3-957a-482b1a996927-kube-api-access-7r4mq\") pod \"payload-processing-e2e-mt-3ffa634f-984fdd89-p5rq4\" (UID: \"839fdbe8-fdae-4fe3-957a-482b1a996927\") " pod="openshift-ingress/payload-processing-e2e-mt-3ffa634f-984fdd89-p5rq4"
Sep 18 18:08:38.382953 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:08:38.382935    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/839fdbe8-fdae-4fe3-957a-482b1a996927-plugins-config-volume\") pod \"payload-processing-e2e-mt-3ffa634f-984fdd89-p5rq4\" (UID: \"839fdbe8-fdae-4fe3-957a-482b1a996927\") " pod="openshift-ingress/payload-processing-e2e-mt-3ffa634f-984fdd89-p5rq4"
Sep 18 18:08:38.394806 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:08:38.394777    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r4mq\" (UniqueName: \"kubernetes.io/projected/839fdbe8-fdae-4fe3-957a-482b1a996927-kube-api-access-7r4mq\") pod \"payload-processing-e2e-mt-3ffa634f-984fdd89-p5rq4\" (UID: \"839fdbe8-fdae-4fe3-957a-482b1a996927\") " pod="openshift-ingress/payload-processing-e2e-mt-3ffa634f-984fdd89-p5rq4"
Sep 18 18:08:38.474779 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:08:38.474697    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3ffa634f-984fdd89-p5rq4"
Sep 18 18:08:38.783876 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:08:38.783794    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3ffa634f-984fdd89-p5rq4"]
Sep 18 18:08:38.790761 ip-10-0-140-54 kubenswrapper[2561]: W0918 18:08:38.790735    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod839fdbe8_fdae_4fe3_957a_482b1a996927.slice/crio-93776dcd6513e3f7367ecd2660d0deeab31838c1cc0022ca97fe419c44a603b8 WatchSource:0}: Error finding container 93776dcd6513e3f7367ecd2660d0deeab31838c1cc0022ca97fe419c44a603b8: Status 404 returned error can't find the container with id 93776dcd6513e3f7367ecd2660d0deeab31838c1cc0022ca97fe419c44a603b8
Sep 18 18:08:39.321960 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:08:39.321928    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3ffa634f-984fdd89-p5rq4" event={"ID":"839fdbe8-fdae-4fe3-957a-482b1a996927","Type":"ContainerStarted","Data":"5c40ee258dde37b5544be07de96d95456c383e9ff31b19b8476e6ba4ecb26e87"}
Sep 18 18:08:39.322435 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:08:39.321985    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3ffa634f-984fdd89-p5rq4" event={"ID":"839fdbe8-fdae-4fe3-957a-482b1a996927","Type":"ContainerStarted","Data":"93776dcd6513e3f7367ecd2660d0deeab31838c1cc0022ca97fe419c44a603b8"}
Sep 18 18:08:39.322435 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:08:39.322054    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-3ffa634f-984fdd89-p5rq4"
Sep 18 18:08:50.329757 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:08:50.329724    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-3ffa634f-984fdd89-p5rq4"
Sep 18 18:08:50.364446 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:08:50.364393    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-3ffa634f-984fdd89-p5rq4" podStartSLOduration=12.364377654 podStartE2EDuration="12.364377654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:08:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:08:39.354681552 +0000 UTC m=+1123.981813172" watchObservedRunningTime="2026-09-18 18:08:50.364377654 +0000 UTC m=+1134.991509273"
Sep 18 18:08:53.190753 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:08:53.190716    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-ef5655/explicit-model-ef5655-kserve-6546d44897-pll2z"]
Sep 18 18:08:53.194580 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:08:53.194564    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-ef5655/explicit-model-ef5655-kserve-6546d44897-pll2z"
Sep 18 18:08:53.200298 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:08:53.200270    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-140-54.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"ai-tenant-e2e-explicit-ef5655\": no relationship found between node 'ip-10-0-140-54.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"ai-tenant-e2e-explicit-ef5655\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap"
Sep 18 18:08:53.200399 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:08:53.200328    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"default-dockercfg-cpbb8\" is forbidden: User \"system:node:ip-10-0-140-54.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"ai-tenant-e2e-explicit-ef5655\": no relationship found between node 'ip-10-0-140-54.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"ai-tenant-e2e-explicit-ef5655\"/\"default-dockercfg-cpbb8\"" type="*v1.Secret"
Sep 18 18:08:53.201141 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:08:53.201105    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-ef5655\"/\"explicit-model-ef5655-kserve-self-signed-certs\""
Sep 18 18:08:53.201250 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:08:53.201184    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-ef5655\"/\"openshift-service-ca.crt\""
Sep 18 18:08:53.224507 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:08:53.224486    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p8pj\" (UniqueName: \"kubernetes.io/projected/81850d0b-1734-4646-b3f0-50fe3e1dbd08-kube-api-access-5p8pj\") pod \"explicit-model-ef5655-kserve-6546d44897-pll2z\" (UID: \"81850d0b-1734-4646-b3f0-50fe3e1dbd08\") " pod="ai-tenant-e2e-explicit-ef5655/explicit-model-ef5655-kserve-6546d44897-pll2z"
Sep 18 18:08:53.224655 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:08:53.224521    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81850d0b-1734-4646-b3f0-50fe3e1dbd08-dshm\") pod \"explicit-model-ef5655-kserve-6546d44897-pll2z\" (UID: \"81850d0b-1734-4646-b3f0-50fe3e1dbd08\") " pod="ai-tenant-e2e-explicit-ef5655/explicit-model-ef5655-kserve-6546d44897-pll2z"
Sep 18 18:08:53.224655 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:08:53.224549    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81850d0b-1734-4646-b3f0-50fe3e1dbd08-tmp-dir\") pod \"explicit-model-ef5655-kserve-6546d44897-pll2z\" (UID: \"81850d0b-1734-4646-b3f0-50fe3e1dbd08\") " pod="ai-tenant-e2e-explicit-ef5655/explicit-model-ef5655-kserve-6546d44897-pll2z"
Sep 18 18:08:53.224655 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:08:53.224617    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81850d0b-1734-4646-b3f0-50fe3e1dbd08-tls-certs\") pod \"explicit-model-ef5655-kserve-6546d44897-pll2z\" (UID: \"81850d0b-1734-4646-b3f0-50fe3e1dbd08\") " pod="ai-tenant-e2e-explicit-ef5655/explicit-model-ef5655-kserve-6546d44897-pll2z"
Sep 18 18:08:53.224934 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:08:53.224713    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81850d0b-1734-4646-b3f0-50fe3e1dbd08-model-cache\") pod \"explicit-model-ef5655-kserve-6546d44897-pll2z\" (UID: \"81850d0b-1734-4646-b3f0-50fe3e1dbd08\") " pod="ai-tenant-e2e-explicit-ef5655/explicit-model-ef5655-kserve-6546d44897-pll2z"
Sep 18 18:08:53.224934 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:08:53.224769    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81850d0b-1734-4646-b3f0-50fe3e1dbd08-home\") pod \"explicit-model-ef5655-kserve-6546d44897-pll2z\" (UID: \"81850d0b-1734-4646-b3f0-50fe3e1dbd08\") " pod="ai-tenant-e2e-explicit-ef5655/explicit-model-ef5655-kserve-6546d44897-pll2z"
Sep 18 18:08:53.229607 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:08:53.229221    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-ef5655/explicit-model-ef5655-kserve-6546d44897-pll2z"]
Sep 18 18:08:53.325699 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:08:53.325659    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p8pj\" (UniqueName: \"kubernetes.io/projected/81850d0b-1734-4646-b3f0-50fe3e1dbd08-kube-api-access-5p8pj\") pod \"explicit-model-ef5655-kserve-6546d44897-pll2z\" (UID: \"81850d0b-1734-4646-b3f0-50fe3e1dbd08\") " pod="ai-tenant-e2e-explicit-ef5655/explicit-model-ef5655-kserve-6546d44897-pll2z"
Sep 18 18:08:53.325699 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:08:53.325698    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81850d0b-1734-4646-b3f0-50fe3e1dbd08-dshm\") pod \"explicit-model-ef5655-kserve-6546d44897-pll2z\" (UID: \"81850d0b-1734-4646-b3f0-50fe3e1dbd08\") " pod="ai-tenant-e2e-explicit-ef5655/explicit-model-ef5655-kserve-6546d44897-pll2z"
Sep 18 18:08:53.325941 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:08:53.325719    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81850d0b-1734-4646-b3f0-50fe3e1dbd08-tmp-dir\") pod \"explicit-model-ef5655-kserve-6546d44897-pll2z\" (UID: \"81850d0b-1734-4646-b3f0-50fe3e1dbd08\") " pod="ai-tenant-e2e-explicit-ef5655/explicit-model-ef5655-kserve-6546d44897-pll2z"
Sep 18 18:08:53.325941 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:08:53.325750    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81850d0b-1734-4646-b3f0-50fe3e1dbd08-tls-certs\") pod \"explicit-model-ef5655-kserve-6546d44897-pll2z\" (UID: \"81850d0b-1734-4646-b3f0-50fe3e1dbd08\") " pod="ai-tenant-e2e-explicit-ef5655/explicit-model-ef5655-kserve-6546d44897-pll2z"
Sep 18 18:08:53.325941 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:08:53.325812    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81850d0b-1734-4646-b3f0-50fe3e1dbd08-model-cache\") pod \"explicit-model-ef5655-kserve-6546d44897-pll2z\" (UID: \"81850d0b-1734-4646-b3f0-50fe3e1dbd08\") " pod="ai-tenant-e2e-explicit-ef5655/explicit-model-ef5655-kserve-6546d44897-pll2z"
Sep 18 18:08:53.325941 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:08:53.325865    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81850d0b-1734-4646-b3f0-50fe3e1dbd08-home\") pod \"explicit-model-ef5655-kserve-6546d44897-pll2z\" (UID: \"81850d0b-1734-4646-b3f0-50fe3e1dbd08\") " pod="ai-tenant-e2e-explicit-ef5655/explicit-model-ef5655-kserve-6546d44897-pll2z"
Sep 18 18:08:53.326175 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:08:53.326085    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81850d0b-1734-4646-b3f0-50fe3e1dbd08-tmp-dir\") pod \"explicit-model-ef5655-kserve-6546d44897-pll2z\" (UID: \"81850d0b-1734-4646-b3f0-50fe3e1dbd08\") " pod="ai-tenant-e2e-explicit-ef5655/explicit-model-ef5655-kserve-6546d44897-pll2z"
Sep 18 18:08:53.326264 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:08:53.326245    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81850d0b-1734-4646-b3f0-50fe3e1dbd08-home\") pod \"explicit-model-ef5655-kserve-6546d44897-pll2z\" (UID: \"81850d0b-1734-4646-b3f0-50fe3e1dbd08\") " pod="ai-tenant-e2e-explicit-ef5655/explicit-model-ef5655-kserve-6546d44897-pll2z"
Sep 18 18:08:53.326320 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:08:53.326286    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81850d0b-1734-4646-b3f0-50fe3e1dbd08-model-cache\") pod \"explicit-model-ef5655-kserve-6546d44897-pll2z\" (UID: \"81850d0b-1734-4646-b3f0-50fe3e1dbd08\") " pod="ai-tenant-e2e-explicit-ef5655/explicit-model-ef5655-kserve-6546d44897-pll2z"
Sep 18 18:08:53.327975 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:08:53.327956    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81850d0b-1734-4646-b3f0-50fe3e1dbd08-dshm\") pod \"explicit-model-ef5655-kserve-6546d44897-pll2z\" (UID: \"81850d0b-1734-4646-b3f0-50fe3e1dbd08\") " pod="ai-tenant-e2e-explicit-ef5655/explicit-model-ef5655-kserve-6546d44897-pll2z"
Sep 18 18:08:53.328164 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:08:53.328144    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81850d0b-1734-4646-b3f0-50fe3e1dbd08-tls-certs\") pod \"explicit-model-ef5655-kserve-6546d44897-pll2z\" (UID: \"81850d0b-1734-4646-b3f0-50fe3e1dbd08\") " pod="ai-tenant-e2e-explicit-ef5655/explicit-model-ef5655-kserve-6546d44897-pll2z"
Sep 18 18:08:53.360737 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:08:53.360714    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:08:54.272341 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:08:54.272300    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-ef5655\"/\"default-dockercfg-cpbb8\""
Sep 18 18:08:54.335809 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:08:54.335777    2561 projected.go:291] Couldn't get configMap ai-tenant-e2e-explicit-ef5655/kube-root-ca.crt: failed to sync configmap cache: timed out waiting for the condition
Sep 18 18:08:54.335809 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:08:54.335813    2561 projected.go:196] Error preparing data for projected volume kube-api-access-5p8pj for pod ai-tenant-e2e-explicit-ef5655/explicit-model-ef5655-kserve-6546d44897-pll2z: failed to sync configmap cache: timed out waiting for the condition
Sep 18 18:08:54.336008 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:08:54.335874    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/81850d0b-1734-4646-b3f0-50fe3e1dbd08-kube-api-access-5p8pj podName:81850d0b-1734-4646-b3f0-50fe3e1dbd08 nodeName:}" failed. No retries permitted until 2026-09-18 18:08:54.835857328 +0000 UTC m=+1139.462988926 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-5p8pj" (UniqueName: "kubernetes.io/projected/81850d0b-1734-4646-b3f0-50fe3e1dbd08-kube-api-access-5p8pj") pod "explicit-model-ef5655-kserve-6546d44897-pll2z" (UID: "81850d0b-1734-4646-b3f0-50fe3e1dbd08") : failed to sync configmap cache: timed out waiting for the condition
Sep 18 18:08:54.477347 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:08:54.477315    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:08:54.793292 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:08:54.793265    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-ef5655\"/\"kube-root-ca.crt\""
Sep 18 18:08:54.840427 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:08:54.840393    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p8pj\" (UniqueName: \"kubernetes.io/projected/81850d0b-1734-4646-b3f0-50fe3e1dbd08-kube-api-access-5p8pj\") pod \"explicit-model-ef5655-kserve-6546d44897-pll2z\" (UID: \"81850d0b-1734-4646-b3f0-50fe3e1dbd08\") " pod="ai-tenant-e2e-explicit-ef5655/explicit-model-ef5655-kserve-6546d44897-pll2z"
Sep 18 18:08:54.842860 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:08:54.842831    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p8pj\" (UniqueName: \"kubernetes.io/projected/81850d0b-1734-4646-b3f0-50fe3e1dbd08-kube-api-access-5p8pj\") pod \"explicit-model-ef5655-kserve-6546d44897-pll2z\" (UID: \"81850d0b-1734-4646-b3f0-50fe3e1dbd08\") " pod="ai-tenant-e2e-explicit-ef5655/explicit-model-ef5655-kserve-6546d44897-pll2z"
Sep 18 18:08:55.003867 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:08:55.003839    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-ef5655/explicit-model-ef5655-kserve-6546d44897-pll2z"
Sep 18 18:08:55.295032 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:08:55.293056    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-ef5655/explicit-model-ef5655-kserve-6546d44897-pll2z"]
Sep 18 18:08:55.297098 ip-10-0-140-54 kubenswrapper[2561]: W0918 18:08:55.297065    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod81850d0b_1734_4646_b3f0_50fe3e1dbd08.slice/crio-8ff720a8a8dcac605efd5883f0d8a8c7be5fab59ab14fdfb41d9c5291fcdfa6d WatchSource:0}: Error finding container 8ff720a8a8dcac605efd5883f0d8a8c7be5fab59ab14fdfb41d9c5291fcdfa6d: Status 404 returned error can't find the container with id 8ff720a8a8dcac605efd5883f0d8a8c7be5fab59ab14fdfb41d9c5291fcdfa6d
Sep 18 18:08:55.409129 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:08:55.409087    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-ef5655/explicit-model-ef5655-kserve-6546d44897-pll2z" event={"ID":"81850d0b-1734-4646-b3f0-50fe3e1dbd08","Type":"ContainerStarted","Data":"f4ae06adbaf315adab46fe754a756ce938c6ea7f474ec08ec674af3f419a5117"}
Sep 18 18:08:55.409288 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:08:55.409144    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-ef5655/explicit-model-ef5655-kserve-6546d44897-pll2z" event={"ID":"81850d0b-1734-4646-b3f0-50fe3e1dbd08","Type":"ContainerStarted","Data":"8ff720a8a8dcac605efd5883f0d8a8c7be5fab59ab14fdfb41d9c5291fcdfa6d"}
Sep 18 18:08:55.441863 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:08:55.441802    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-ef5655/explicit-model-ef5655-kserve-6546d44897-pll2z" podStartSLOduration=2.441785523 podStartE2EDuration="2.441785523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:08:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:08:55.440528642 +0000 UTC m=+1140.067660270" watchObservedRunningTime="2026-09-18 18:08:55.441785523 +0000 UTC m=+1140.068917142"
Sep 18 18:08:57.829101 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:08:57.829066    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:00.387343 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:00.387312    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:02.161822 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:02.161794    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:03.279621 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:03.279594    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:03.771356 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:03.771319    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:04.223565 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:04.223536    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:04.459298 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:04.459266    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:04.560201 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:04.560103    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:04.920165 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:04.920109    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:05.004059 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:05.004023    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-ef5655/explicit-model-ef5655-kserve-6546d44897-pll2z"
Sep 18 18:09:05.004265 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:05.004077    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-ef5655/explicit-model-ef5655-kserve-6546d44897-pll2z"
Sep 18 18:09:05.017254 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:05.017229    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-ef5655/explicit-model-ef5655-kserve-6546d44897-pll2z"
Sep 18 18:09:05.474781 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:05.474754    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-ef5655/explicit-model-ef5655-kserve-6546d44897-pll2z"
Sep 18 18:09:05.486560 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:05.486536    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:07.219878 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:07.219846    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:08.156468 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:08.156434    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:08.929625 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:08.929592    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:11.465445 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:11.465413    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:11.567535 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:11.567505    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:12.927630 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:12.927596    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:13.976557 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:13.976528    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:14.859823 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:14.859785    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:15.860617 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:15.860581    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:16.714963 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:16.714917    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:17.620484 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:17.620452    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:18.714301 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:18.714215    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:22.583959 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:22.583927    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:23.329377 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:23.329343    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:23.370052 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:23.370023    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3ffa634f-984fdd89-p5rq4"]
Sep 18 18:09:23.370268 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:23.370242    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-3ffa634f-984fdd89-p5rq4" podUID="839fdbe8-fdae-4fe3-957a-482b1a996927" containerName="payload-processing" containerID="cri-o://5c40ee258dde37b5544be07de96d95456c383e9ff31b19b8476e6ba4ecb26e87" gracePeriod=30
Sep 18 18:09:23.434349 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:23.434319    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:23.540665 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:23.540637    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:23.561927 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:23.561899    2561 generic.go:356] "Generic (PLEG): container finished" podID="839fdbe8-fdae-4fe3-957a-482b1a996927" containerID="5c40ee258dde37b5544be07de96d95456c383e9ff31b19b8476e6ba4ecb26e87" exitCode=0
Sep 18 18:09:23.562100 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:23.561949    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3ffa634f-984fdd89-p5rq4" event={"ID":"839fdbe8-fdae-4fe3-957a-482b1a996927","Type":"ContainerDied","Data":"5c40ee258dde37b5544be07de96d95456c383e9ff31b19b8476e6ba4ecb26e87"}
Sep 18 18:09:23.630742 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:23.630715    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3ffa634f-984fdd89-p5rq4"
Sep 18 18:09:23.661725 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:23.661699    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:23.726526 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:23.726495    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/839fdbe8-fdae-4fe3-957a-482b1a996927-plugins-config-volume\") pod \"839fdbe8-fdae-4fe3-957a-482b1a996927\" (UID: \"839fdbe8-fdae-4fe3-957a-482b1a996927\") "
Sep 18 18:09:23.726672 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:23.726615    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7r4mq\" (UniqueName: \"kubernetes.io/projected/839fdbe8-fdae-4fe3-957a-482b1a996927-kube-api-access-7r4mq\") pod \"839fdbe8-fdae-4fe3-957a-482b1a996927\" (UID: \"839fdbe8-fdae-4fe3-957a-482b1a996927\") "
Sep 18 18:09:23.726912 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:23.726882    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/839fdbe8-fdae-4fe3-957a-482b1a996927-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "839fdbe8-fdae-4fe3-957a-482b1a996927" (UID: "839fdbe8-fdae-4fe3-957a-482b1a996927"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:09:23.728678 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:23.728657    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/839fdbe8-fdae-4fe3-957a-482b1a996927-kube-api-access-7r4mq" (OuterVolumeSpecName: "kube-api-access-7r4mq") pod "839fdbe8-fdae-4fe3-957a-482b1a996927" (UID: "839fdbe8-fdae-4fe3-957a-482b1a996927"). InnerVolumeSpecName "kube-api-access-7r4mq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:09:23.828060 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:23.828026    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7r4mq\" (UniqueName: \"kubernetes.io/projected/839fdbe8-fdae-4fe3-957a-482b1a996927-kube-api-access-7r4mq\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:09:23.828060 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:23.828053    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/839fdbe8-fdae-4fe3-957a-482b1a996927-plugins-config-volume\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:09:24.048805 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:24.048774    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:24.568274 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:24.568244    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3ffa634f-984fdd89-p5rq4"
Sep 18 18:09:24.568274 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:24.568261    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3ffa634f-984fdd89-p5rq4" event={"ID":"839fdbe8-fdae-4fe3-957a-482b1a996927","Type":"ContainerDied","Data":"93776dcd6513e3f7367ecd2660d0deeab31838c1cc0022ca97fe419c44a603b8"}
Sep 18 18:09:24.568494 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:24.568321    2561 scope.go:117] "RemoveContainer" containerID="5c40ee258dde37b5544be07de96d95456c383e9ff31b19b8476e6ba4ecb26e87"
Sep 18 18:09:24.600126 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:24.600093    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3ffa634f-984fdd89-p5rq4"]
Sep 18 18:09:24.605387 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:24.605364    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3ffa634f-984fdd89-p5rq4"]
Sep 18 18:09:24.881999 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:24.881922    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:25.867150 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:25.864245    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="839fdbe8-fdae-4fe3-957a-482b1a996927" path="/var/lib/kubelet/pods/839fdbe8-fdae-4fe3-957a-482b1a996927/volumes"
Sep 18 18:09:26.050332 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:26.050298    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:27.072622 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:27.072592    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:27.906368 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:27.906340    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:28.779638 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:28.779607    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:29.672001 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:29.671974    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:30.462034 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:30.462001    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:30.564999 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:30.564960    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:30.663165 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:30.663134    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:30.780302 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:30.780218    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:31.204976 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:31.204941    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:31.349741 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:31.349715    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:31.497242 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:31.497163    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:33.536821 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:33.536788    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:35.611394 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:35.611364    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:36.530932 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:36.530881    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:36.683608 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:36.683580    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:36.872741 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:36.872656    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:37.031351 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:37.031321    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:37.290169 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:37.290138    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:37.407183 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:37.407143    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:38.624362 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:38.624324    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:39.741391 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:39.741360    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:40.736966 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:40.736934    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:40.868923 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:40.868890    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:40.988556 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:40.988472    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:41.084139 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:41.084093    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:41.690858 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:41.690826    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:42.242139 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:42.242094    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-ef5655/explicit-model-ef5655-kserve-6546d44897-pll2z"]
Sep 18 18:09:42.334032 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:42.334006    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:48.043739 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:48.043705    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:48.233692 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:48.233658    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:49.103164 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:49.103110    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:49.455490 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:49.455462    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:49.726358 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:49.726229    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:53.848626 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:53.848595    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:54.458492 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:54.458450    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:54.592182 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:54.592152    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:55.004464 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:55.004433    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:55.840612 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:55.840587    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-5kjlz_75f2a1f7-95a7-462e-9ea7-23c51dd29599/console-operator/1.log"
Sep 18 18:09:55.840899 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:55.840879    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-5kjlz_75f2a1f7-95a7-462e-9ea7-23c51dd29599/console-operator/1.log"
Sep 18 18:09:55.848007 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:55.847981    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkcgk_c5d026b7-af6b-4899-85f4-6554c0f7cd52/ovn-acl-logging/0.log"
Sep 18 18:09:55.848155 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:55.848083    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkcgk_c5d026b7-af6b-4899-85f4-6554c0f7cd52/ovn-acl-logging/0.log"
Sep 18 18:09:56.175188 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:56.175155    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:56.937380 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:56.937332    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:57.261289 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:57.261192    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-ef5655/explicit-model-ef5655-kserve-6546d44897-pll2z" podUID="81850d0b-1734-4646-b3f0-50fe3e1dbd08" containerName="main" containerID="cri-o://f4ae06adbaf315adab46fe754a756ce938c6ea7f474ec08ec674af3f419a5117" gracePeriod=45
Sep 18 18:09:57.524009 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:57.523952    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-ef5655/explicit-model-ef5655-kserve-6546d44897-pll2z"
Sep 18 18:09:57.671105 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:57.671072    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81850d0b-1734-4646-b3f0-50fe3e1dbd08-dshm\") pod \"81850d0b-1734-4646-b3f0-50fe3e1dbd08\" (UID: \"81850d0b-1734-4646-b3f0-50fe3e1dbd08\") "
Sep 18 18:09:57.671317 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:57.671140    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81850d0b-1734-4646-b3f0-50fe3e1dbd08-tls-certs\") pod \"81850d0b-1734-4646-b3f0-50fe3e1dbd08\" (UID: \"81850d0b-1734-4646-b3f0-50fe3e1dbd08\") "
Sep 18 18:09:57.671317 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:57.671229    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81850d0b-1734-4646-b3f0-50fe3e1dbd08-tmp-dir\") pod \"81850d0b-1734-4646-b3f0-50fe3e1dbd08\" (UID: \"81850d0b-1734-4646-b3f0-50fe3e1dbd08\") "
Sep 18 18:09:57.671317 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:57.671263    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5p8pj\" (UniqueName: \"kubernetes.io/projected/81850d0b-1734-4646-b3f0-50fe3e1dbd08-kube-api-access-5p8pj\") pod \"81850d0b-1734-4646-b3f0-50fe3e1dbd08\" (UID: \"81850d0b-1734-4646-b3f0-50fe3e1dbd08\") "
Sep 18 18:09:57.671317 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:57.671285    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81850d0b-1734-4646-b3f0-50fe3e1dbd08-home\") pod \"81850d0b-1734-4646-b3f0-50fe3e1dbd08\" (UID: \"81850d0b-1734-4646-b3f0-50fe3e1dbd08\") "
Sep 18 18:09:57.671528 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:57.671391    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81850d0b-1734-4646-b3f0-50fe3e1dbd08-model-cache\") pod \"81850d0b-1734-4646-b3f0-50fe3e1dbd08\" (UID: \"81850d0b-1734-4646-b3f0-50fe3e1dbd08\") "
Sep 18 18:09:57.671528 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:57.671432    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81850d0b-1734-4646-b3f0-50fe3e1dbd08-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "81850d0b-1734-4646-b3f0-50fe3e1dbd08" (UID: "81850d0b-1734-4646-b3f0-50fe3e1dbd08"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:09:57.671631 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:57.671590    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81850d0b-1734-4646-b3f0-50fe3e1dbd08-home" (OuterVolumeSpecName: "home") pod "81850d0b-1734-4646-b3f0-50fe3e1dbd08" (UID: "81850d0b-1734-4646-b3f0-50fe3e1dbd08"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:09:57.671753 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:57.671711    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81850d0b-1734-4646-b3f0-50fe3e1dbd08-model-cache" (OuterVolumeSpecName: "model-cache") pod "81850d0b-1734-4646-b3f0-50fe3e1dbd08" (UID: "81850d0b-1734-4646-b3f0-50fe3e1dbd08"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:09:57.671889 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:57.671791    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81850d0b-1734-4646-b3f0-50fe3e1dbd08-tmp-dir\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:09:57.671889 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:57.671806    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81850d0b-1734-4646-b3f0-50fe3e1dbd08-home\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:09:57.673525 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:57.673502    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81850d0b-1734-4646-b3f0-50fe3e1dbd08-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "81850d0b-1734-4646-b3f0-50fe3e1dbd08" (UID: "81850d0b-1734-4646-b3f0-50fe3e1dbd08"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:09:57.673593 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:57.673522    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81850d0b-1734-4646-b3f0-50fe3e1dbd08-dshm" (OuterVolumeSpecName: "dshm") pod "81850d0b-1734-4646-b3f0-50fe3e1dbd08" (UID: "81850d0b-1734-4646-b3f0-50fe3e1dbd08"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:09:57.673593 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:57.673553    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81850d0b-1734-4646-b3f0-50fe3e1dbd08-kube-api-access-5p8pj" (OuterVolumeSpecName: "kube-api-access-5p8pj") pod "81850d0b-1734-4646-b3f0-50fe3e1dbd08" (UID: "81850d0b-1734-4646-b3f0-50fe3e1dbd08"). InnerVolumeSpecName "kube-api-access-5p8pj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:09:57.742961 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:57.742928    2561 generic.go:356] "Generic (PLEG): container finished" podID="81850d0b-1734-4646-b3f0-50fe3e1dbd08" containerID="f4ae06adbaf315adab46fe754a756ce938c6ea7f474ec08ec674af3f419a5117" exitCode=0
Sep 18 18:09:57.743086 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:57.742971    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-ef5655/explicit-model-ef5655-kserve-6546d44897-pll2z"
Sep 18 18:09:57.743086 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:57.743010    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-ef5655/explicit-model-ef5655-kserve-6546d44897-pll2z" event={"ID":"81850d0b-1734-4646-b3f0-50fe3e1dbd08","Type":"ContainerDied","Data":"f4ae06adbaf315adab46fe754a756ce938c6ea7f474ec08ec674af3f419a5117"}
Sep 18 18:09:57.743086 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:57.743055    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-ef5655/explicit-model-ef5655-kserve-6546d44897-pll2z" event={"ID":"81850d0b-1734-4646-b3f0-50fe3e1dbd08","Type":"ContainerDied","Data":"8ff720a8a8dcac605efd5883f0d8a8c7be5fab59ab14fdfb41d9c5291fcdfa6d"}
Sep 18 18:09:57.743086 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:57.743077    2561 scope.go:117] "RemoveContainer" containerID="f4ae06adbaf315adab46fe754a756ce938c6ea7f474ec08ec674af3f419a5117"
Sep 18 18:09:57.752870 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:57.752842    2561 scope.go:117] "RemoveContainer" containerID="f4ae06adbaf315adab46fe754a756ce938c6ea7f474ec08ec674af3f419a5117"
Sep 18 18:09:57.753250 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:09:57.753225    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4ae06adbaf315adab46fe754a756ce938c6ea7f474ec08ec674af3f419a5117\": container with ID starting with f4ae06adbaf315adab46fe754a756ce938c6ea7f474ec08ec674af3f419a5117 not found: ID does not exist" containerID="f4ae06adbaf315adab46fe754a756ce938c6ea7f474ec08ec674af3f419a5117"
Sep 18 18:09:57.753367 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:57.753260    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4ae06adbaf315adab46fe754a756ce938c6ea7f474ec08ec674af3f419a5117"} err="failed to get container status \"f4ae06adbaf315adab46fe754a756ce938c6ea7f474ec08ec674af3f419a5117\": rpc error: code = NotFound desc = could not find container \"f4ae06adbaf315adab46fe754a756ce938c6ea7f474ec08ec674af3f419a5117\": container with ID starting with f4ae06adbaf315adab46fe754a756ce938c6ea7f474ec08ec674af3f419a5117 not found: ID does not exist"
Sep 18 18:09:57.772199 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:57.772174    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-ef5655/explicit-model-ef5655-kserve-6546d44897-pll2z"]
Sep 18 18:09:57.772548 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:57.772529    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81850d0b-1734-4646-b3f0-50fe3e1dbd08-model-cache\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:09:57.772604 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:57.772551    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81850d0b-1734-4646-b3f0-50fe3e1dbd08-dshm\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:09:57.772604 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:57.772559    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81850d0b-1734-4646-b3f0-50fe3e1dbd08-tls-certs\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:09:57.772604 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:57.772567    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5p8pj\" (UniqueName: \"kubernetes.io/projected/81850d0b-1734-4646-b3f0-50fe3e1dbd08-kube-api-access-5p8pj\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:09:57.775418 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:57.775371    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-ef5655/explicit-model-ef5655-kserve-6546d44897-pll2z"]
Sep 18 18:09:57.861700 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:57.861663    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:57.862377 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:57.862356    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81850d0b-1734-4646-b3f0-50fe3e1dbd08" path="/var/lib/kubelet/pods/81850d0b-1734-4646-b3f0-50fe3e1dbd08/volumes"
Sep 18 18:09:58.617497 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:58.617464    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:58.702566 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:58.702532    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:58.800067 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:58.800037    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:59.156235 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:59.156200    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:09:59.260029 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:09:59.260003    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:10:09.663073 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:09.663041    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:10:12.246174 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:12.246141    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:10:19.530363 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:19.530330    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:10:20.969679 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:20.969647    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:10:21.150211 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:21.150174    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:10:21.840370 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:21.840310    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:10:21.947053 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:21.947019    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:10:22.047733 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:22.047705    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:10:22.147828 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:22.147801    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:10:22.949859 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:22.949822    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:10:23.050145 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:23.050104    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:10:23.199664 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:23.199637    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:10:24.253764 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:24.253731    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:10:24.568751 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:24.568665    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:10:24.887641 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:24.887566    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:10:25.281717 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:25.281689    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:10:25.505097 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:25.505066    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:10:25.774238 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:25.774197    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:10:26.349249 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:26.349217    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:10:27.481630 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:27.481591    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:10:28.150969 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:28.150935    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:10:28.241928 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:28.241897    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:10:28.424622 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:28.424542    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:10:29.187517 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:29.187485    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:10:29.999001 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:29.998960    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:10:31.053498 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:31.053463    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:10:31.843400 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:31.843371    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:10:32.613737 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:32.613709    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-130a80-777c8489f4-tdchm"]
Sep 18 18:10:32.614184 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:32.614169    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="839fdbe8-fdae-4fe3-957a-482b1a996927" containerName="payload-processing"
Sep 18 18:10:32.614184 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:32.614188    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="839fdbe8-fdae-4fe3-957a-482b1a996927" containerName="payload-processing"
Sep 18 18:10:32.614307 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:32.614208    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81850d0b-1734-4646-b3f0-50fe3e1dbd08" containerName="main"
Sep 18 18:10:32.614307 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:32.614214    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="81850d0b-1734-4646-b3f0-50fe3e1dbd08" containerName="main"
Sep 18 18:10:32.614381 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:32.614313    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="839fdbe8-fdae-4fe3-957a-482b1a996927" containerName="payload-processing"
Sep 18 18:10:32.614381 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:32.614320    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="81850d0b-1734-4646-b3f0-50fe3e1dbd08" containerName="main"
Sep 18 18:10:32.617538 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:32.617519    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-130a80-777c8489f4-tdchm"
Sep 18 18:10:32.620143 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:32.620096    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-130a80-serving-cert\""
Sep 18 18:10:32.631593 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:32.631570    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-130a80-777c8489f4-tdchm"]
Sep 18 18:10:32.647889 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:32.647869    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:10:32.715520 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:32.715489    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:10:32.720548 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:32.720527    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2846\" (UniqueName: \"kubernetes.io/projected/b9e1f09c-7857-49ae-8efe-78b2bad8cf8d-kube-api-access-j2846\") pod \"maas-api-e2e-shared-b-w2-130a80-777c8489f4-tdchm\" (UID: \"b9e1f09c-7857-49ae-8efe-78b2bad8cf8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-130a80-777c8489f4-tdchm"
Sep 18 18:10:32.720668 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:32.720571    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9e1f09c-7857-49ae-8efe-78b2bad8cf8d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-130a80-777c8489f4-tdchm\" (UID: \"b9e1f09c-7857-49ae-8efe-78b2bad8cf8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-130a80-777c8489f4-tdchm"
Sep 18 18:10:32.720668 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:32.720629    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9e1f09c-7857-49ae-8efe-78b2bad8cf8d-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-130a80-777c8489f4-tdchm\" (UID: \"b9e1f09c-7857-49ae-8efe-78b2bad8cf8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-130a80-777c8489f4-tdchm"
Sep 18 18:10:32.720668 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:32.720650    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9e1f09c-7857-49ae-8efe-78b2bad8cf8d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-130a80-777c8489f4-tdchm\" (UID: \"b9e1f09c-7857-49ae-8efe-78b2bad8cf8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-130a80-777c8489f4-tdchm"
Sep 18 18:10:32.821758 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:32.821727    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9e1f09c-7857-49ae-8efe-78b2bad8cf8d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-130a80-777c8489f4-tdchm\" (UID: \"b9e1f09c-7857-49ae-8efe-78b2bad8cf8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-130a80-777c8489f4-tdchm"
Sep 18 18:10:32.821911 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:32.821783    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9e1f09c-7857-49ae-8efe-78b2bad8cf8d-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-130a80-777c8489f4-tdchm\" (UID: \"b9e1f09c-7857-49ae-8efe-78b2bad8cf8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-130a80-777c8489f4-tdchm"
Sep 18 18:10:32.821911 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:32.821878    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9e1f09c-7857-49ae-8efe-78b2bad8cf8d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-130a80-777c8489f4-tdchm\" (UID: \"b9e1f09c-7857-49ae-8efe-78b2bad8cf8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-130a80-777c8489f4-tdchm"
Sep 18 18:10:32.822034 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:10:32.821903    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-130a80-serving-cert: secret "maas-api-e2e-shared-b-w2-130a80-serving-cert" not found
Sep 18 18:10:32.822034 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:32.821933    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2846\" (UniqueName: \"kubernetes.io/projected/b9e1f09c-7857-49ae-8efe-78b2bad8cf8d-kube-api-access-j2846\") pod \"maas-api-e2e-shared-b-w2-130a80-777c8489f4-tdchm\" (UID: \"b9e1f09c-7857-49ae-8efe-78b2bad8cf8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-130a80-777c8489f4-tdchm"
Sep 18 18:10:32.822034 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:10:32.821991    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9e1f09c-7857-49ae-8efe-78b2bad8cf8d-maas-api-tls podName:b9e1f09c-7857-49ae-8efe-78b2bad8cf8d nodeName:}" failed. No retries permitted until 2026-09-18 18:10:33.321966966 +0000 UTC m=+1237.949098581 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b9e1f09c-7857-49ae-8efe-78b2bad8cf8d-maas-api-tls") pod "maas-api-e2e-shared-b-w2-130a80-777c8489f4-tdchm" (UID: "b9e1f09c-7857-49ae-8efe-78b2bad8cf8d") : secret "maas-api-e2e-shared-b-w2-130a80-serving-cert" not found
Sep 18 18:10:32.822663 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:32.822628    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9e1f09c-7857-49ae-8efe-78b2bad8cf8d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-130a80-777c8489f4-tdchm\" (UID: \"b9e1f09c-7857-49ae-8efe-78b2bad8cf8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-130a80-777c8489f4-tdchm"
Sep 18 18:10:32.824320 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:32.824291    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9e1f09c-7857-49ae-8efe-78b2bad8cf8d-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-130a80-777c8489f4-tdchm\" (UID: \"b9e1f09c-7857-49ae-8efe-78b2bad8cf8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-130a80-777c8489f4-tdchm"
Sep 18 18:10:32.831851 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:32.831828    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2846\" (UniqueName: \"kubernetes.io/projected/b9e1f09c-7857-49ae-8efe-78b2bad8cf8d-kube-api-access-j2846\") pod \"maas-api-e2e-shared-b-w2-130a80-777c8489f4-tdchm\" (UID: \"b9e1f09c-7857-49ae-8efe-78b2bad8cf8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-130a80-777c8489f4-tdchm"
Sep 18 18:10:32.914508 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:32.914478    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:10:33.210434 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:33.210350    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:10:33.327638 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:33.327608    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9e1f09c-7857-49ae-8efe-78b2bad8cf8d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-130a80-777c8489f4-tdchm\" (UID: \"b9e1f09c-7857-49ae-8efe-78b2bad8cf8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-130a80-777c8489f4-tdchm"
Sep 18 18:10:33.330409 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:33.330377    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:10:33.330553 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:33.330484    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9e1f09c-7857-49ae-8efe-78b2bad8cf8d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-130a80-777c8489f4-tdchm\" (UID: \"b9e1f09c-7857-49ae-8efe-78b2bad8cf8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-130a80-777c8489f4-tdchm"
Sep 18 18:10:33.528547 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:33.528465    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-130a80-777c8489f4-tdchm"
Sep 18 18:10:33.804370 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:33.804343    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-130a80-777c8489f4-tdchm"]
Sep 18 18:10:33.806939 ip-10-0-140-54 kubenswrapper[2561]: W0918 18:10:33.806914    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9e1f09c_7857_49ae_8efe_78b2bad8cf8d.slice/crio-9909cf46822293bba750cc962384fcfd1253c7b2c2a7cd0ccce26041de7b8ffc WatchSource:0}: Error finding container 9909cf46822293bba750cc962384fcfd1253c7b2c2a7cd0ccce26041de7b8ffc: Status 404 returned error can't find the container with id 9909cf46822293bba750cc962384fcfd1253c7b2c2a7cd0ccce26041de7b8ffc
Sep 18 18:10:33.808312 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:33.808282    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 18:10:33.945838 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:33.945790    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-130a80-777c8489f4-tdchm" event={"ID":"b9e1f09c-7857-49ae-8efe-78b2bad8cf8d","Type":"ContainerStarted","Data":"9909cf46822293bba750cc962384fcfd1253c7b2c2a7cd0ccce26041de7b8ffc"}
Sep 18 18:10:34.953686 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:34.953655    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-130a80-777c8489f4-tdchm" event={"ID":"b9e1f09c-7857-49ae-8efe-78b2bad8cf8d","Type":"ContainerStarted","Data":"b2905502d00bb7cef9ac03136f2f516815de4c44603fd5ab2d922652bb1db806"}
Sep 18 18:10:34.954234 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:34.953800    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-130a80-777c8489f4-tdchm"
Sep 18 18:10:34.974931 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:34.974889    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-130a80-777c8489f4-tdchm" podStartSLOduration=2.577475795 podStartE2EDuration="2.974874038s" totalImagesPullingTime="397.398243ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:10:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:10:34.972915466 +0000 UTC m=+1239.600047086" watchObservedRunningTime="2026-09-18 18:10:34.974874038 +0000 UTC m=+1239.602005658"
Sep 18 18:10:39.811327 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:39.811295    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:10:40.965417 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:40.965385    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-130a80-777c8489f4-tdchm"
Sep 18 18:10:54.864054 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:54.863980    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:10:56.537416 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:56.537383    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:10:57.256241 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:57.256205    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:10:57.989175 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:57.989141    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:10:58.234298 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:58.234266    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:10:58.863723 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:58.863656    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:10:59.030594 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:59.030557    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7599-33545b"]
Sep 18 18:10:59.035723 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:59.035704    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7599-33545b"
Sep 18 18:10:59.038356 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:59.038338    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-fz228\""
Sep 18 18:10:59.043434 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:59.043266    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7599-33545b"]
Sep 18 18:10:59.072353 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:59.072321    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tzm8\" (UniqueName: \"kubernetes.io/projected/a7d1f1d1-2621-4114-a7de-c427e5961b08-kube-api-access-7tzm8\") pod \"test-curl-7599-33545b\" (UID: \"a7d1f1d1-2621-4114-a7de-c427e5961b08\") " pod="opendatahub/test-curl-7599-33545b"
Sep 18 18:10:59.173108 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:59.173072    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tzm8\" (UniqueName: \"kubernetes.io/projected/a7d1f1d1-2621-4114-a7de-c427e5961b08-kube-api-access-7tzm8\") pod \"test-curl-7599-33545b\" (UID: \"a7d1f1d1-2621-4114-a7de-c427e5961b08\") " pod="opendatahub/test-curl-7599-33545b"
Sep 18 18:10:59.182029 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:59.182004    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tzm8\" (UniqueName: \"kubernetes.io/projected/a7d1f1d1-2621-4114-a7de-c427e5961b08-kube-api-access-7tzm8\") pod \"test-curl-7599-33545b\" (UID: \"a7d1f1d1-2621-4114-a7de-c427e5961b08\") " pod="opendatahub/test-curl-7599-33545b"
Sep 18 18:10:59.345694 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:59.345666    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7599-33545b"
Sep 18 18:10:59.622764 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:59.622738    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7599-33545b"]
Sep 18 18:10:59.624911 ip-10-0-140-54 kubenswrapper[2561]: W0918 18:10:59.624881    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda7d1f1d1_2621_4114_a7de_c427e5961b08.slice/crio-c9217e29fa48a1c62e3c8045adb7e6c89c9a77859d1e144712e7cc68cb97280a WatchSource:0}: Error finding container c9217e29fa48a1c62e3c8045adb7e6c89c9a77859d1e144712e7cc68cb97280a: Status 404 returned error can't find the container with id c9217e29fa48a1c62e3c8045adb7e6c89c9a77859d1e144712e7cc68cb97280a
Sep 18 18:10:59.707965 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:10:59.707943    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:11:00.084273 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:00.084237    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7599-33545b" event={"ID":"a7d1f1d1-2621-4114-a7de-c427e5961b08","Type":"ContainerStarted","Data":"5f21c096bb8fe3fdcc163a118edf0ab64da5a5f8bacc95ce3f08189340a00b59"}
Sep 18 18:11:00.084273 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:00.084283    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7599-33545b" event={"ID":"a7d1f1d1-2621-4114-a7de-c427e5961b08","Type":"ContainerStarted","Data":"c9217e29fa48a1c62e3c8045adb7e6c89c9a77859d1e144712e7cc68cb97280a"}
Sep 18 18:11:00.104164 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:00.104101    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7599-33545b" podStartSLOduration=1.104084331 podStartE2EDuration="1.104084331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:10:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:11:00.101920682 +0000 UTC m=+1264.729052303" watchObservedRunningTime="2026-09-18 18:11:00.104084331 +0000 UTC m=+1264.731215958"
Sep 18 18:11:00.649663 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:00.649630    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:11:00.821457 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:00.821425    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7599-33545b"]
Sep 18 18:11:00.824882 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:00.824851    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7599-33545b"]
Sep 18 18:11:00.943796 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:00.943763    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7599-a22a68"]
Sep 18 18:11:00.944245 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:00.944232    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7d1f1d1-2621-4114-a7de-c427e5961b08" containerName="test-curl-7599-33545b"
Sep 18 18:11:00.944299 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:00.944247    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7d1f1d1-2621-4114-a7de-c427e5961b08" containerName="test-curl-7599-33545b"
Sep 18 18:11:00.944335 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:00.944314    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7d1f1d1-2621-4114-a7de-c427e5961b08" containerName="test-curl-7599-33545b"
Sep 18 18:11:00.947749 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:00.947732    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7599-a22a68"
Sep 18 18:11:00.953473 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:00.953449    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7599-a22a68"]
Sep 18 18:11:00.990473 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:00.990447    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h56kn\" (UniqueName: \"kubernetes.io/projected/8d54b9b7-f526-4607-80a6-e8f8caafc260-kube-api-access-h56kn\") pod \"test-curl-7599-a22a68\" (UID: \"8d54b9b7-f526-4607-80a6-e8f8caafc260\") " pod="opendatahub/test-curl-7599-a22a68"
Sep 18 18:11:01.089410 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:11:01.089381    2561 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7599-33545b" container="test-curl-7599-33545b"
Sep 18 18:11:01.091540 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:01.091520    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h56kn\" (UniqueName: \"kubernetes.io/projected/8d54b9b7-f526-4607-80a6-e8f8caafc260-kube-api-access-h56kn\") pod \"test-curl-7599-a22a68\" (UID: \"8d54b9b7-f526-4607-80a6-e8f8caafc260\") " pod="opendatahub/test-curl-7599-a22a68"
Sep 18 18:11:01.099564 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:01.099535    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h56kn\" (UniqueName: \"kubernetes.io/projected/8d54b9b7-f526-4607-80a6-e8f8caafc260-kube-api-access-h56kn\") pod \"test-curl-7599-a22a68\" (UID: \"8d54b9b7-f526-4607-80a6-e8f8caafc260\") " pod="opendatahub/test-curl-7599-a22a68"
Sep 18 18:11:01.259254 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:01.259174    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7599-a22a68"
Sep 18 18:11:01.440543 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:01.440502    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:11:01.532870 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:01.532847    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7599-a22a68"]
Sep 18 18:11:01.534715 ip-10-0-140-54 kubenswrapper[2561]: W0918 18:11:01.534691    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8d54b9b7_f526_4607_80a6_e8f8caafc260.slice/crio-045b4d4f58c05a6d026f113d503acd96a0522406893d0e1d18918a0faab12581 WatchSource:0}: Error finding container 045b4d4f58c05a6d026f113d503acd96a0522406893d0e1d18918a0faab12581: Status 404 returned error can't find the container with id 045b4d4f58c05a6d026f113d503acd96a0522406893d0e1d18918a0faab12581
Sep 18 18:11:02.096433 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:02.096389    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7599-a22a68" event={"ID":"8d54b9b7-f526-4607-80a6-e8f8caafc260","Type":"ContainerStarted","Data":"c58bbc5bc19987a3c7e1092ed7a87a99ec7a0f08d1032a53b232ff2d5a2a4908"}
Sep 18 18:11:02.096865 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:02.096441    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7599-a22a68" event={"ID":"8d54b9b7-f526-4607-80a6-e8f8caafc260","Type":"ContainerStarted","Data":"045b4d4f58c05a6d026f113d503acd96a0522406893d0e1d18918a0faab12581"}
Sep 18 18:11:02.096865 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:02.096447    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7599-33545b" podUID="a7d1f1d1-2621-4114-a7de-c427e5961b08" containerName="test-curl-7599-33545b" containerID="cri-o://5f21c096bb8fe3fdcc163a118edf0ab64da5a5f8bacc95ce3f08189340a00b59" gracePeriod=2
Sep 18 18:11:02.114420 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:02.114374    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7599-a22a68" podStartSLOduration=2.114352762 podStartE2EDuration="2.114352762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:11:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:11:02.113385686 +0000 UTC m=+1266.740517331" watchObservedRunningTime="2026-09-18 18:11:02.114352762 +0000 UTC m=+1266.741484382"
Sep 18 18:11:02.735285 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:02.735249    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7599-a22a68"]
Sep 18 18:11:02.740510 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:02.740485    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7599-a22a68"]
Sep 18 18:11:02.865722 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:02.865642    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7599-115647"]
Sep 18 18:11:02.866104 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:02.866091    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d54b9b7-f526-4607-80a6-e8f8caafc260" containerName="test-curl-7599-a22a68"
Sep 18 18:11:02.866167 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:02.866107    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d54b9b7-f526-4607-80a6-e8f8caafc260" containerName="test-curl-7599-a22a68"
Sep 18 18:11:02.866250 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:02.866199    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d54b9b7-f526-4607-80a6-e8f8caafc260" containerName="test-curl-7599-a22a68"
Sep 18 18:11:02.869314 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:02.869296    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7599-115647"
Sep 18 18:11:02.875793 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:02.875737    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7599-115647"]
Sep 18 18:11:02.910534 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:02.910496    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m98wz\" (UniqueName: \"kubernetes.io/projected/6608edf8-5600-4c8e-b63d-18062d25c236-kube-api-access-m98wz\") pod \"test-curl-7599-115647\" (UID: \"6608edf8-5600-4c8e-b63d-18062d25c236\") " pod="opendatahub/test-curl-7599-115647"
Sep 18 18:11:03.011912 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:03.011877    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m98wz\" (UniqueName: \"kubernetes.io/projected/6608edf8-5600-4c8e-b63d-18062d25c236-kube-api-access-m98wz\") pod \"test-curl-7599-115647\" (UID: \"6608edf8-5600-4c8e-b63d-18062d25c236\") " pod="opendatahub/test-curl-7599-115647"
Sep 18 18:11:03.020755 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:03.020731    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m98wz\" (UniqueName: \"kubernetes.io/projected/6608edf8-5600-4c8e-b63d-18062d25c236-kube-api-access-m98wz\") pod \"test-curl-7599-115647\" (UID: \"6608edf8-5600-4c8e-b63d-18062d25c236\") " pod="opendatahub/test-curl-7599-115647"
Sep 18 18:11:03.103359 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:11:03.103329    2561 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7599-a22a68" container="test-curl-7599-a22a68"
Sep 18 18:11:03.180403 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:03.180365    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7599-115647"
Sep 18 18:11:03.453062 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:03.453036    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7599-115647"]
Sep 18 18:11:03.455750 ip-10-0-140-54 kubenswrapper[2561]: W0918 18:11:03.455716    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6608edf8_5600_4c8e_b63d_18062d25c236.slice/crio-76fa97b95c338c0c5ced54237d608ec5be6e2c183697f25b39ef4935a5b1d5e4 WatchSource:0}: Error finding container 76fa97b95c338c0c5ced54237d608ec5be6e2c183697f25b39ef4935a5b1d5e4: Status 404 returned error can't find the container with id 76fa97b95c338c0c5ced54237d608ec5be6e2c183697f25b39ef4935a5b1d5e4
Sep 18 18:11:04.110031 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:04.110001    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7599-115647" event={"ID":"6608edf8-5600-4c8e-b63d-18062d25c236","Type":"ContainerStarted","Data":"cb98a06234738373f6055af86ecc301cd2802a089c8668f98de78c7b3bbc7d0a"}
Sep 18 18:11:04.110406 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:04.110042    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7599-115647" event={"ID":"6608edf8-5600-4c8e-b63d-18062d25c236","Type":"ContainerStarted","Data":"76fa97b95c338c0c5ced54237d608ec5be6e2c183697f25b39ef4935a5b1d5e4"}
Sep 18 18:11:04.110406 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:04.110140    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7599-a22a68" podUID="8d54b9b7-f526-4607-80a6-e8f8caafc260" containerName="test-curl-7599-a22a68" containerID="cri-o://c58bbc5bc19987a3c7e1092ed7a87a99ec7a0f08d1032a53b232ff2d5a2a4908" gracePeriod=2
Sep 18 18:11:04.125717 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:04.125674    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7599-115647" podStartSLOduration=2.125659481 podStartE2EDuration="2.125659481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:11:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:11:04.123443281 +0000 UTC m=+1268.750574900" watchObservedRunningTime="2026-09-18 18:11:04.125659481 +0000 UTC m=+1268.752791128"
Sep 18 18:11:04.288058 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:04.288036    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7599-33545b"
Sep 18 18:11:04.426614 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:04.426585    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7tzm8\" (UniqueName: \"kubernetes.io/projected/a7d1f1d1-2621-4114-a7de-c427e5961b08-kube-api-access-7tzm8\") pod \"a7d1f1d1-2621-4114-a7de-c427e5961b08\" (UID: \"a7d1f1d1-2621-4114-a7de-c427e5961b08\") "
Sep 18 18:11:04.428492 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:04.428466    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7d1f1d1-2621-4114-a7de-c427e5961b08-kube-api-access-7tzm8" (OuterVolumeSpecName: "kube-api-access-7tzm8") pod "a7d1f1d1-2621-4114-a7de-c427e5961b08" (UID: "a7d1f1d1-2621-4114-a7de-c427e5961b08"). InnerVolumeSpecName "kube-api-access-7tzm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:11:04.528176 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:04.528149    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tzm8\" (UniqueName: \"kubernetes.io/projected/a7d1f1d1-2621-4114-a7de-c427e5961b08-kube-api-access-7tzm8\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:11:04.712854 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:04.712761    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7599-115647"]
Sep 18 18:11:04.717867 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:04.717838    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7599-115647"]
Sep 18 18:11:04.882693 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:04.882657    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7599-5c55b0"]
Sep 18 18:11:04.883294 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:04.883276    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6608edf8-5600-4c8e-b63d-18062d25c236" containerName="test-curl-7599-115647"
Sep 18 18:11:04.883344 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:04.883298    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6608edf8-5600-4c8e-b63d-18062d25c236" containerName="test-curl-7599-115647"
Sep 18 18:11:04.883413 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:04.883402    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="6608edf8-5600-4c8e-b63d-18062d25c236" containerName="test-curl-7599-115647"
Sep 18 18:11:04.887059 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:04.887041    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7599-5c55b0"
Sep 18 18:11:04.912347 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:04.912316    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7599-5c55b0"]
Sep 18 18:11:05.031946 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:05.031859    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2r5l\" (UniqueName: \"kubernetes.io/projected/1a4ac16f-60fc-45fe-a087-de1ef53c8ee1-kube-api-access-r2r5l\") pod \"test-curl-7599-5c55b0\" (UID: \"1a4ac16f-60fc-45fe-a087-de1ef53c8ee1\") " pod="opendatahub/test-curl-7599-5c55b0"
Sep 18 18:11:05.117081 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:05.117045    2561 generic.go:356] "Generic (PLEG): container finished" podID="a7d1f1d1-2621-4114-a7de-c427e5961b08" containerID="5f21c096bb8fe3fdcc163a118edf0ab64da5a5f8bacc95ce3f08189340a00b59" exitCode=137
Sep 18 18:11:05.117556 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:05.117093    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7599-33545b"
Sep 18 18:11:05.117556 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:05.117150    2561 scope.go:117] "RemoveContainer" containerID="5f21c096bb8fe3fdcc163a118edf0ab64da5a5f8bacc95ce3f08189340a00b59"
Sep 18 18:11:05.117556 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:11:05.117431    2561 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7599-115647" container="test-curl-7599-115647"
Sep 18 18:11:05.126760 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:05.126559    2561 scope.go:117] "RemoveContainer" containerID="5f21c096bb8fe3fdcc163a118edf0ab64da5a5f8bacc95ce3f08189340a00b59"
Sep 18 18:11:05.126928 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:11:05.126896    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f21c096bb8fe3fdcc163a118edf0ab64da5a5f8bacc95ce3f08189340a00b59\": container with ID starting with 5f21c096bb8fe3fdcc163a118edf0ab64da5a5f8bacc95ce3f08189340a00b59 not found: ID does not exist" containerID="5f21c096bb8fe3fdcc163a118edf0ab64da5a5f8bacc95ce3f08189340a00b59"
Sep 18 18:11:05.127028 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:05.126938    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f21c096bb8fe3fdcc163a118edf0ab64da5a5f8bacc95ce3f08189340a00b59"} err="failed to get container status \"5f21c096bb8fe3fdcc163a118edf0ab64da5a5f8bacc95ce3f08189340a00b59\": rpc error: code = NotFound desc = could not find container \"5f21c096bb8fe3fdcc163a118edf0ab64da5a5f8bacc95ce3f08189340a00b59\": container with ID starting with 5f21c096bb8fe3fdcc163a118edf0ab64da5a5f8bacc95ce3f08189340a00b59 not found: ID does not exist"
Sep 18 18:11:05.129307 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:11:05.129281    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7599-33545b\" is forbidden: User \"system:node:ip-10-0-140-54.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-54.ec2.internal' and this object" podUID="a7d1f1d1-2621-4114-a7de-c427e5961b08" pod="opendatahub/test-curl-7599-33545b"
Sep 18 18:11:05.132373 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:05.132351    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2r5l\" (UniqueName: \"kubernetes.io/projected/1a4ac16f-60fc-45fe-a087-de1ef53c8ee1-kube-api-access-r2r5l\") pod \"test-curl-7599-5c55b0\" (UID: \"1a4ac16f-60fc-45fe-a087-de1ef53c8ee1\") " pod="opendatahub/test-curl-7599-5c55b0"
Sep 18 18:11:05.140716 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:05.140695    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2r5l\" (UniqueName: \"kubernetes.io/projected/1a4ac16f-60fc-45fe-a087-de1ef53c8ee1-kube-api-access-r2r5l\") pod \"test-curl-7599-5c55b0\" (UID: \"1a4ac16f-60fc-45fe-a087-de1ef53c8ee1\") " pod="opendatahub/test-curl-7599-5c55b0"
Sep 18 18:11:05.212793 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:05.212767    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7599-5c55b0"
Sep 18 18:11:05.487377 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:05.487353    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7599-5c55b0"]
Sep 18 18:11:05.489566 ip-10-0-140-54 kubenswrapper[2561]: W0918 18:11:05.489543    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1a4ac16f_60fc_45fe_a087_de1ef53c8ee1.slice/crio-ab428fd5ae134fe2217e2a721418f3a4b6e1cb3760196bd27b343d4cb5137805 WatchSource:0}: Error finding container ab428fd5ae134fe2217e2a721418f3a4b6e1cb3760196bd27b343d4cb5137805: Status 404 returned error can't find the container with id ab428fd5ae134fe2217e2a721418f3a4b6e1cb3760196bd27b343d4cb5137805
Sep 18 18:11:05.861033 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:11:05.860946    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7599-33545b\" is forbidden: User \"system:node:ip-10-0-140-54.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-54.ec2.internal' and this object" podUID="a7d1f1d1-2621-4114-a7de-c427e5961b08" pod="opendatahub/test-curl-7599-33545b"
Sep 18 18:11:05.862379 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:05.862359    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7d1f1d1-2621-4114-a7de-c427e5961b08" path="/var/lib/kubelet/pods/a7d1f1d1-2621-4114-a7de-c427e5961b08/volumes"
Sep 18 18:11:06.124356 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:06.124324    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7599-5c55b0" event={"ID":"1a4ac16f-60fc-45fe-a087-de1ef53c8ee1","Type":"ContainerStarted","Data":"15b0e9b996770011f398117d8bd32a5291c7e3c05a630250be570f3a2c75e085"}
Sep 18 18:11:06.124732 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:06.124368    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7599-5c55b0" event={"ID":"1a4ac16f-60fc-45fe-a087-de1ef53c8ee1","Type":"ContainerStarted","Data":"ab428fd5ae134fe2217e2a721418f3a4b6e1cb3760196bd27b343d4cb5137805"}
Sep 18 18:11:06.125284 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:06.125259    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7599-115647" podUID="6608edf8-5600-4c8e-b63d-18062d25c236" containerName="test-curl-7599-115647" containerID="cri-o://cb98a06234738373f6055af86ecc301cd2802a089c8668f98de78c7b3bbc7d0a" gracePeriod=2
Sep 18 18:11:06.141665 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:06.141588    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7599-5c55b0" podStartSLOduration=2.141568781 podStartE2EDuration="2.141568781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:11:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:11:06.140757184 +0000 UTC m=+1270.767888806" watchObservedRunningTime="2026-09-18 18:11:06.141568781 +0000 UTC m=+1270.768700402"
Sep 18 18:11:06.286863 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:06.286841    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7599-a22a68"
Sep 18 18:11:06.344017 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:06.343990    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h56kn\" (UniqueName: \"kubernetes.io/projected/8d54b9b7-f526-4607-80a6-e8f8caafc260-kube-api-access-h56kn\") pod \"8d54b9b7-f526-4607-80a6-e8f8caafc260\" (UID: \"8d54b9b7-f526-4607-80a6-e8f8caafc260\") "
Sep 18 18:11:06.345894 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:06.345864    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d54b9b7-f526-4607-80a6-e8f8caafc260-kube-api-access-h56kn" (OuterVolumeSpecName: "kube-api-access-h56kn") pod "8d54b9b7-f526-4607-80a6-e8f8caafc260" (UID: "8d54b9b7-f526-4607-80a6-e8f8caafc260"). InnerVolumeSpecName "kube-api-access-h56kn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:11:06.444893 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:06.444861    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h56kn\" (UniqueName: \"kubernetes.io/projected/8d54b9b7-f526-4607-80a6-e8f8caafc260-kube-api-access-h56kn\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:11:06.725544 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:06.725469    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7599-5c55b0"]
Sep 18 18:11:06.732795 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:06.732769    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7599-5c55b0"]
Sep 18 18:11:07.131944 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:07.131858    2561 generic.go:356] "Generic (PLEG): container finished" podID="8d54b9b7-f526-4607-80a6-e8f8caafc260" containerID="c58bbc5bc19987a3c7e1092ed7a87a99ec7a0f08d1032a53b232ff2d5a2a4908" exitCode=137
Sep 18 18:11:07.131944 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:07.131899    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7599-a22a68"
Sep 18 18:11:07.132402 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:07.131954    2561 scope.go:117] "RemoveContainer" containerID="c58bbc5bc19987a3c7e1092ed7a87a99ec7a0f08d1032a53b232ff2d5a2a4908"
Sep 18 18:11:07.132402 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:11:07.132205    2561 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7599-5c55b0" container="test-curl-7599-5c55b0"
Sep 18 18:11:07.145253 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:07.143064    2561 scope.go:117] "RemoveContainer" containerID="c58bbc5bc19987a3c7e1092ed7a87a99ec7a0f08d1032a53b232ff2d5a2a4908"
Sep 18 18:11:07.145673 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:11:07.145646    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c58bbc5bc19987a3c7e1092ed7a87a99ec7a0f08d1032a53b232ff2d5a2a4908\": container with ID starting with c58bbc5bc19987a3c7e1092ed7a87a99ec7a0f08d1032a53b232ff2d5a2a4908 not found: ID does not exist" containerID="c58bbc5bc19987a3c7e1092ed7a87a99ec7a0f08d1032a53b232ff2d5a2a4908"
Sep 18 18:11:07.145782 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:07.145679    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c58bbc5bc19987a3c7e1092ed7a87a99ec7a0f08d1032a53b232ff2d5a2a4908"} err="failed to get container status \"c58bbc5bc19987a3c7e1092ed7a87a99ec7a0f08d1032a53b232ff2d5a2a4908\": rpc error: code = NotFound desc = could not find container \"c58bbc5bc19987a3c7e1092ed7a87a99ec7a0f08d1032a53b232ff2d5a2a4908\": container with ID starting with c58bbc5bc19987a3c7e1092ed7a87a99ec7a0f08d1032a53b232ff2d5a2a4908 not found: ID does not exist"
Sep 18 18:11:07.149692 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:11:07.149665    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7599-a22a68\" is forbidden: User \"system:node:ip-10-0-140-54.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-54.ec2.internal' and this object" podUID="8d54b9b7-f526-4607-80a6-e8f8caafc260" pod="opendatahub/test-curl-7599-a22a68"
Sep 18 18:11:07.337071 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:07.337040    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:11:07.684402 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:07.684365    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:11:07.862394 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:07.862361    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d54b9b7-f526-4607-80a6-e8f8caafc260" path="/var/lib/kubelet/pods/8d54b9b7-f526-4607-80a6-e8f8caafc260/volumes"
Sep 18 18:11:07.939678 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:07.939589    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:11:08.138757 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:08.138630    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7599-5c55b0" podUID="1a4ac16f-60fc-45fe-a087-de1ef53c8ee1" containerName="test-curl-7599-5c55b0" containerID="cri-o://15b0e9b996770011f398117d8bd32a5291c7e3c05a630250be570f3a2c75e085" gracePeriod=2
Sep 18 18:11:08.315905 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:08.315883    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7599-115647"
Sep 18 18:11:08.466852 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:08.466819    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m98wz\" (UniqueName: \"kubernetes.io/projected/6608edf8-5600-4c8e-b63d-18062d25c236-kube-api-access-m98wz\") pod \"6608edf8-5600-4c8e-b63d-18062d25c236\" (UID: \"6608edf8-5600-4c8e-b63d-18062d25c236\") "
Sep 18 18:11:08.468883 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:08.468860    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6608edf8-5600-4c8e-b63d-18062d25c236-kube-api-access-m98wz" (OuterVolumeSpecName: "kube-api-access-m98wz") pod "6608edf8-5600-4c8e-b63d-18062d25c236" (UID: "6608edf8-5600-4c8e-b63d-18062d25c236"). InnerVolumeSpecName "kube-api-access-m98wz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:11:08.568105 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:08.568029    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m98wz\" (UniqueName: \"kubernetes.io/projected/6608edf8-5600-4c8e-b63d-18062d25c236-kube-api-access-m98wz\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:11:09.145599 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:09.145567    2561 generic.go:356] "Generic (PLEG): container finished" podID="6608edf8-5600-4c8e-b63d-18062d25c236" containerID="cb98a06234738373f6055af86ecc301cd2802a089c8668f98de78c7b3bbc7d0a" exitCode=137
Sep 18 18:11:09.146029 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:09.145612    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7599-115647"
Sep 18 18:11:09.146029 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:09.145671    2561 scope.go:117] "RemoveContainer" containerID="cb98a06234738373f6055af86ecc301cd2802a089c8668f98de78c7b3bbc7d0a"
Sep 18 18:11:09.155050 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:09.155033    2561 scope.go:117] "RemoveContainer" containerID="cb98a06234738373f6055af86ecc301cd2802a089c8668f98de78c7b3bbc7d0a"
Sep 18 18:11:09.155321 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:11:09.155303    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb98a06234738373f6055af86ecc301cd2802a089c8668f98de78c7b3bbc7d0a\": container with ID starting with cb98a06234738373f6055af86ecc301cd2802a089c8668f98de78c7b3bbc7d0a not found: ID does not exist" containerID="cb98a06234738373f6055af86ecc301cd2802a089c8668f98de78c7b3bbc7d0a"
Sep 18 18:11:09.155377 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:09.155328    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb98a06234738373f6055af86ecc301cd2802a089c8668f98de78c7b3bbc7d0a"} err="failed to get container status \"cb98a06234738373f6055af86ecc301cd2802a089c8668f98de78c7b3bbc7d0a\": rpc error: code = NotFound desc = could not find container \"cb98a06234738373f6055af86ecc301cd2802a089c8668f98de78c7b3bbc7d0a\": container with ID starting with cb98a06234738373f6055af86ecc301cd2802a089c8668f98de78c7b3bbc7d0a not found: ID does not exist"
Sep 18 18:11:09.156906 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:11:09.156885    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7599-115647\" is forbidden: User \"system:node:ip-10-0-140-54.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-54.ec2.internal' and this object" podUID="6608edf8-5600-4c8e-b63d-18062d25c236" pod="opendatahub/test-curl-7599-115647"
Sep 18 18:11:09.862638 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:09.862594    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6608edf8-5600-4c8e-b63d-18062d25c236" path="/var/lib/kubelet/pods/6608edf8-5600-4c8e-b63d-18062d25c236/volumes"
Sep 18 18:11:10.298897 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:10.298875    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7599-5c55b0"
Sep 18 18:11:10.327064 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:10.327038    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:11:10.387250 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:10.387216    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r2r5l\" (UniqueName: \"kubernetes.io/projected/1a4ac16f-60fc-45fe-a087-de1ef53c8ee1-kube-api-access-r2r5l\") pod \"1a4ac16f-60fc-45fe-a087-de1ef53c8ee1\" (UID: \"1a4ac16f-60fc-45fe-a087-de1ef53c8ee1\") "
Sep 18 18:11:10.389418 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:10.389386    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a4ac16f-60fc-45fe-a087-de1ef53c8ee1-kube-api-access-r2r5l" (OuterVolumeSpecName: "kube-api-access-r2r5l") pod "1a4ac16f-60fc-45fe-a087-de1ef53c8ee1" (UID: "1a4ac16f-60fc-45fe-a087-de1ef53c8ee1"). InnerVolumeSpecName "kube-api-access-r2r5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:11:10.479648 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:10.479615    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:11:10.488306 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:10.488280    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2r5l\" (UniqueName: \"kubernetes.io/projected/1a4ac16f-60fc-45fe-a087-de1ef53c8ee1-kube-api-access-r2r5l\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:11:10.590534 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:10.590468    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:11:11.161596 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:11.161563    2561 generic.go:356] "Generic (PLEG): container finished" podID="1a4ac16f-60fc-45fe-a087-de1ef53c8ee1" containerID="15b0e9b996770011f398117d8bd32a5291c7e3c05a630250be570f3a2c75e085" exitCode=137
Sep 18 18:11:11.161765 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:11.161602    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7599-5c55b0"
Sep 18 18:11:11.161765 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:11.161640    2561 scope.go:117] "RemoveContainer" containerID="15b0e9b996770011f398117d8bd32a5291c7e3c05a630250be570f3a2c75e085"
Sep 18 18:11:11.170561 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:11.170545    2561 scope.go:117] "RemoveContainer" containerID="15b0e9b996770011f398117d8bd32a5291c7e3c05a630250be570f3a2c75e085"
Sep 18 18:11:11.170815 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:11:11.170800    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15b0e9b996770011f398117d8bd32a5291c7e3c05a630250be570f3a2c75e085\": container with ID starting with 15b0e9b996770011f398117d8bd32a5291c7e3c05a630250be570f3a2c75e085 not found: ID does not exist" containerID="15b0e9b996770011f398117d8bd32a5291c7e3c05a630250be570f3a2c75e085"
Sep 18 18:11:11.170854 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:11.170822    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15b0e9b996770011f398117d8bd32a5291c7e3c05a630250be570f3a2c75e085"} err="failed to get container status \"15b0e9b996770011f398117d8bd32a5291c7e3c05a630250be570f3a2c75e085\": rpc error: code = NotFound desc = could not find container \"15b0e9b996770011f398117d8bd32a5291c7e3c05a630250be570f3a2c75e085\": container with ID starting with 15b0e9b996770011f398117d8bd32a5291c7e3c05a630250be570f3a2c75e085 not found: ID does not exist"
Sep 18 18:11:11.172109 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:11:11.172087    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7599-5c55b0\" is forbidden: User \"system:node:ip-10-0-140-54.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-54.ec2.internal' and this object" podUID="1a4ac16f-60fc-45fe-a087-de1ef53c8ee1" pod="opendatahub/test-curl-7599-5c55b0"
Sep 18 18:11:11.247393 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:11.247358    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:11:11.861539 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:11.861506    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a4ac16f-60fc-45fe-a087-de1ef53c8ee1" path="/var/lib/kubelet/pods/1a4ac16f-60fc-45fe-a087-de1ef53c8ee1/volumes"
Sep 18 18:11:12.122531 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:12.122449    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:11:13.055431 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:13.055397    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:11:13.866902 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:13.866864    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:11:14.623297 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:14.623262    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:11:15.461919 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:15.461884    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f7e72d9f-54b7648c9b-z87kp"]
Sep 18 18:11:15.462434 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:15.462415    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a4ac16f-60fc-45fe-a087-de1ef53c8ee1" containerName="test-curl-7599-5c55b0"
Sep 18 18:11:15.462434 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:15.462435    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a4ac16f-60fc-45fe-a087-de1ef53c8ee1" containerName="test-curl-7599-5c55b0"
Sep 18 18:11:15.462582 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:15.462573    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a4ac16f-60fc-45fe-a087-de1ef53c8ee1" containerName="test-curl-7599-5c55b0"
Sep 18 18:11:15.466793 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:15.466766    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7e72d9f-54b7648c9b-z87kp"
Sep 18 18:11:15.470725 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:15.470703    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-f7e72d9f-serving-cert\""
Sep 18 18:11:15.491373 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:15.491336    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f7e72d9f-54b7648c9b-z87kp"]
Sep 18 18:11:15.516205 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:15.516160    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:11:15.617297 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:15.617268    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:11:15.642716 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:15.642686    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94100fcf-733a-4b2e-b17d-25a7d14be7a3-maas-api-tls\") pod \"maas-api-e2e-mt-f7e72d9f-54b7648c9b-z87kp\" (UID: \"94100fcf-733a-4b2e-b17d-25a7d14be7a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7e72d9f-54b7648c9b-z87kp"
Sep 18 18:11:15.643074 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:15.642735    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8krlt\" (UniqueName: \"kubernetes.io/projected/94100fcf-733a-4b2e-b17d-25a7d14be7a3-kube-api-access-8krlt\") pod \"maas-api-e2e-mt-f7e72d9f-54b7648c9b-z87kp\" (UID: \"94100fcf-733a-4b2e-b17d-25a7d14be7a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7e72d9f-54b7648c9b-z87kp"
Sep 18 18:11:15.643074 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:15.642800    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94100fcf-733a-4b2e-b17d-25a7d14be7a3-openshift-service-ca\") pod \"maas-api-e2e-mt-f7e72d9f-54b7648c9b-z87kp\" (UID: \"94100fcf-733a-4b2e-b17d-25a7d14be7a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7e72d9f-54b7648c9b-z87kp"
Sep 18 18:11:15.643074 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:15.642871    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94100fcf-733a-4b2e-b17d-25a7d14be7a3-metrics-certs\") pod \"maas-api-e2e-mt-f7e72d9f-54b7648c9b-z87kp\" (UID: \"94100fcf-733a-4b2e-b17d-25a7d14be7a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7e72d9f-54b7648c9b-z87kp"
Sep 18 18:11:15.744135 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:15.744039    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94100fcf-733a-4b2e-b17d-25a7d14be7a3-maas-api-tls\") pod \"maas-api-e2e-mt-f7e72d9f-54b7648c9b-z87kp\" (UID: \"94100fcf-733a-4b2e-b17d-25a7d14be7a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7e72d9f-54b7648c9b-z87kp"
Sep 18 18:11:15.744135 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:15.744077    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8krlt\" (UniqueName: \"kubernetes.io/projected/94100fcf-733a-4b2e-b17d-25a7d14be7a3-kube-api-access-8krlt\") pod \"maas-api-e2e-mt-f7e72d9f-54b7648c9b-z87kp\" (UID: \"94100fcf-733a-4b2e-b17d-25a7d14be7a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7e72d9f-54b7648c9b-z87kp"
Sep 18 18:11:15.744135 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:15.744101    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94100fcf-733a-4b2e-b17d-25a7d14be7a3-openshift-service-ca\") pod \"maas-api-e2e-mt-f7e72d9f-54b7648c9b-z87kp\" (UID: \"94100fcf-733a-4b2e-b17d-25a7d14be7a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7e72d9f-54b7648c9b-z87kp"
Sep 18 18:11:15.744397 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:15.744168    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94100fcf-733a-4b2e-b17d-25a7d14be7a3-metrics-certs\") pod \"maas-api-e2e-mt-f7e72d9f-54b7648c9b-z87kp\" (UID: \"94100fcf-733a-4b2e-b17d-25a7d14be7a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7e72d9f-54b7648c9b-z87kp"
Sep 18 18:11:15.744397 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:11:15.744227    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-f7e72d9f-serving-cert: secret "maas-api-e2e-mt-f7e72d9f-serving-cert" not found
Sep 18 18:11:15.744397 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:11:15.744290    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94100fcf-733a-4b2e-b17d-25a7d14be7a3-maas-api-tls podName:94100fcf-733a-4b2e-b17d-25a7d14be7a3 nodeName:}" failed. No retries permitted until 2026-09-18 18:11:16.244271523 +0000 UTC m=+1280.871403136 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/94100fcf-733a-4b2e-b17d-25a7d14be7a3-maas-api-tls") pod "maas-api-e2e-mt-f7e72d9f-54b7648c9b-z87kp" (UID: "94100fcf-733a-4b2e-b17d-25a7d14be7a3") : secret "maas-api-e2e-mt-f7e72d9f-serving-cert" not found
Sep 18 18:11:15.744790 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:15.744772    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94100fcf-733a-4b2e-b17d-25a7d14be7a3-openshift-service-ca\") pod \"maas-api-e2e-mt-f7e72d9f-54b7648c9b-z87kp\" (UID: \"94100fcf-733a-4b2e-b17d-25a7d14be7a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7e72d9f-54b7648c9b-z87kp"
Sep 18 18:11:15.746638 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:15.746621    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94100fcf-733a-4b2e-b17d-25a7d14be7a3-metrics-certs\") pod \"maas-api-e2e-mt-f7e72d9f-54b7648c9b-z87kp\" (UID: \"94100fcf-733a-4b2e-b17d-25a7d14be7a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7e72d9f-54b7648c9b-z87kp"
Sep 18 18:11:15.755212 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:15.755190    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8krlt\" (UniqueName: \"kubernetes.io/projected/94100fcf-733a-4b2e-b17d-25a7d14be7a3-kube-api-access-8krlt\") pod \"maas-api-e2e-mt-f7e72d9f-54b7648c9b-z87kp\" (UID: \"94100fcf-733a-4b2e-b17d-25a7d14be7a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7e72d9f-54b7648c9b-z87kp"
Sep 18 18:11:15.981166 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:15.981134    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:11:16.208373 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:16.208346    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:11:16.249021 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:16.248986    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94100fcf-733a-4b2e-b17d-25a7d14be7a3-maas-api-tls\") pod \"maas-api-e2e-mt-f7e72d9f-54b7648c9b-z87kp\" (UID: \"94100fcf-733a-4b2e-b17d-25a7d14be7a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7e72d9f-54b7648c9b-z87kp"
Sep 18 18:11:16.251483 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:16.251455    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94100fcf-733a-4b2e-b17d-25a7d14be7a3-maas-api-tls\") pod \"maas-api-e2e-mt-f7e72d9f-54b7648c9b-z87kp\" (UID: \"94100fcf-733a-4b2e-b17d-25a7d14be7a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7e72d9f-54b7648c9b-z87kp"
Sep 18 18:11:16.307787 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:16.307761    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:11:16.343246 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:16.343217    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f7e72d9f-86676c8c8f-ptnrl"]
Sep 18 18:11:16.347328 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:16.347311    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f7e72d9f-86676c8c8f-ptnrl"
Sep 18 18:11:16.350041 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:16.350013    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-f7e72d9f-dockercfg-qkpwf\""
Sep 18 18:11:16.350310 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:16.350294    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-f7e72d9f\""
Sep 18 18:11:16.364266 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:16.364247    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f7e72d9f-86676c8c8f-ptnrl"]
Sep 18 18:11:16.377448 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:16.377423    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7e72d9f-54b7648c9b-z87kp"
Sep 18 18:11:16.401060 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:16.401033    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f7e72d9f-54b9945bdb-zwjgm"]
Sep 18 18:11:16.404725 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:16.404708    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f7e72d9f-54b9945bdb-zwjgm"
Sep 18 18:11:16.416799 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:16.416772    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f7e72d9f-54b9945bdb-zwjgm"]
Sep 18 18:11:16.451497 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:16.451413    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19a9dc6f-6b51-4639-bfb5-4b4122a552ee-plugins-config-volume\") pod \"payload-processing-e2e-mt-f7e72d9f-86676c8c8f-ptnrl\" (UID: \"19a9dc6f-6b51-4639-bfb5-4b4122a552ee\") " pod="openshift-ingress/payload-processing-e2e-mt-f7e72d9f-86676c8c8f-ptnrl"
Sep 18 18:11:16.451629 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:16.451550    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84ltk\" (UniqueName: \"kubernetes.io/projected/19a9dc6f-6b51-4639-bfb5-4b4122a552ee-kube-api-access-84ltk\") pod \"payload-processing-e2e-mt-f7e72d9f-86676c8c8f-ptnrl\" (UID: \"19a9dc6f-6b51-4639-bfb5-4b4122a552ee\") " pod="openshift-ingress/payload-processing-e2e-mt-f7e72d9f-86676c8c8f-ptnrl"
Sep 18 18:11:16.552934 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:16.552834    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57c13b34-8239-4165-b2c4-8e512de7b5b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f7e72d9f-54b9945bdb-zwjgm\" (UID: \"57c13b34-8239-4165-b2c4-8e512de7b5b3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f7e72d9f-54b9945bdb-zwjgm"
Sep 18 18:11:16.552934 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:16.552900    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19a9dc6f-6b51-4639-bfb5-4b4122a552ee-plugins-config-volume\") pod \"payload-processing-e2e-mt-f7e72d9f-86676c8c8f-ptnrl\" (UID: \"19a9dc6f-6b51-4639-bfb5-4b4122a552ee\") " pod="openshift-ingress/payload-processing-e2e-mt-f7e72d9f-86676c8c8f-ptnrl"
Sep 18 18:11:16.553145 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:16.552972    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84ltk\" (UniqueName: \"kubernetes.io/projected/19a9dc6f-6b51-4639-bfb5-4b4122a552ee-kube-api-access-84ltk\") pod \"payload-processing-e2e-mt-f7e72d9f-86676c8c8f-ptnrl\" (UID: \"19a9dc6f-6b51-4639-bfb5-4b4122a552ee\") " pod="openshift-ingress/payload-processing-e2e-mt-f7e72d9f-86676c8c8f-ptnrl"
Sep 18 18:11:16.553145 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:16.553089    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58mdj\" (UniqueName: \"kubernetes.io/projected/57c13b34-8239-4165-b2c4-8e512de7b5b3-kube-api-access-58mdj\") pod \"payload-pre-processing-e2e-mt-f7e72d9f-54b9945bdb-zwjgm\" (UID: \"57c13b34-8239-4165-b2c4-8e512de7b5b3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f7e72d9f-54b9945bdb-zwjgm"
Sep 18 18:11:16.553561 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:16.553543    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19a9dc6f-6b51-4639-bfb5-4b4122a552ee-plugins-config-volume\") pod \"payload-processing-e2e-mt-f7e72d9f-86676c8c8f-ptnrl\" (UID: \"19a9dc6f-6b51-4639-bfb5-4b4122a552ee\") " pod="openshift-ingress/payload-processing-e2e-mt-f7e72d9f-86676c8c8f-ptnrl"
Sep 18 18:11:16.561703 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:16.561682    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84ltk\" (UniqueName: \"kubernetes.io/projected/19a9dc6f-6b51-4639-bfb5-4b4122a552ee-kube-api-access-84ltk\") pod \"payload-processing-e2e-mt-f7e72d9f-86676c8c8f-ptnrl\" (UID: \"19a9dc6f-6b51-4639-bfb5-4b4122a552ee\") " pod="openshift-ingress/payload-processing-e2e-mt-f7e72d9f-86676c8c8f-ptnrl"
Sep 18 18:11:16.654490 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:16.654460    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58mdj\" (UniqueName: \"kubernetes.io/projected/57c13b34-8239-4165-b2c4-8e512de7b5b3-kube-api-access-58mdj\") pod \"payload-pre-processing-e2e-mt-f7e72d9f-54b9945bdb-zwjgm\" (UID: \"57c13b34-8239-4165-b2c4-8e512de7b5b3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f7e72d9f-54b9945bdb-zwjgm"
Sep 18 18:11:16.654853 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:16.654521    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57c13b34-8239-4165-b2c4-8e512de7b5b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f7e72d9f-54b9945bdb-zwjgm\" (UID: \"57c13b34-8239-4165-b2c4-8e512de7b5b3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f7e72d9f-54b9945bdb-zwjgm"
Sep 18 18:11:16.655279 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:16.655258    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57c13b34-8239-4165-b2c4-8e512de7b5b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f7e72d9f-54b9945bdb-zwjgm\" (UID: \"57c13b34-8239-4165-b2c4-8e512de7b5b3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f7e72d9f-54b9945bdb-zwjgm"
Sep 18 18:11:16.658291 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:16.658274    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f7e72d9f-86676c8c8f-ptnrl"
Sep 18 18:11:16.662690 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:16.662667    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f7e72d9f-54b7648c9b-z87kp"]
Sep 18 18:11:16.664362 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:16.664338    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58mdj\" (UniqueName: \"kubernetes.io/projected/57c13b34-8239-4165-b2c4-8e512de7b5b3-kube-api-access-58mdj\") pod \"payload-pre-processing-e2e-mt-f7e72d9f-54b9945bdb-zwjgm\" (UID: \"57c13b34-8239-4165-b2c4-8e512de7b5b3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f7e72d9f-54b9945bdb-zwjgm"
Sep 18 18:11:16.665043 ip-10-0-140-54 kubenswrapper[2561]: W0918 18:11:16.665021    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94100fcf_733a_4b2e_b17d_25a7d14be7a3.slice/crio-ec4f90301e9bed5de16b7bb7bb6eb5510ed8102961b887957495963463697e41 WatchSource:0}: Error finding container ec4f90301e9bed5de16b7bb7bb6eb5510ed8102961b887957495963463697e41: Status 404 returned error can't find the container with id ec4f90301e9bed5de16b7bb7bb6eb5510ed8102961b887957495963463697e41
Sep 18 18:11:16.722785 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:16.722554    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f7e72d9f-54b9945bdb-zwjgm"
Sep 18 18:11:16.936521 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:16.936494    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f7e72d9f-86676c8c8f-ptnrl"]
Sep 18 18:11:16.938155 ip-10-0-140-54 kubenswrapper[2561]: W0918 18:11:16.938113    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19a9dc6f_6b51_4639_bfb5_4b4122a552ee.slice/crio-2d67a7e4e051d28689ebaa828a4204accf6dc15c711366d782a4f7621416653c WatchSource:0}: Error finding container 2d67a7e4e051d28689ebaa828a4204accf6dc15c711366d782a4f7621416653c: Status 404 returned error can't find the container with id 2d67a7e4e051d28689ebaa828a4204accf6dc15c711366d782a4f7621416653c
Sep 18 18:11:17.005381 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:17.005356    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f7e72d9f-54b9945bdb-zwjgm"]
Sep 18 18:11:17.006008 ip-10-0-140-54 kubenswrapper[2561]: W0918 18:11:17.005985    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57c13b34_8239_4165_b2c4_8e512de7b5b3.slice/crio-86dc308301765e953d75e6a7ff822359c9e9ce9c318b3e2a3c17cbff5b8bb7b2 WatchSource:0}: Error finding container 86dc308301765e953d75e6a7ff822359c9e9ce9c318b3e2a3c17cbff5b8bb7b2: Status 404 returned error can't find the container with id 86dc308301765e953d75e6a7ff822359c9e9ce9c318b3e2a3c17cbff5b8bb7b2
Sep 18 18:11:17.196212 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:17.196139    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7e72d9f-54b7648c9b-z87kp" event={"ID":"94100fcf-733a-4b2e-b17d-25a7d14be7a3","Type":"ContainerStarted","Data":"76b3102bb53f95c2426e906b8740951d246e02a593b89cbf2b754d9a0131230b"}
Sep 18 18:11:17.196212 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:17.196196    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7e72d9f-54b7648c9b-z87kp" event={"ID":"94100fcf-733a-4b2e-b17d-25a7d14be7a3","Type":"ContainerStarted","Data":"ec4f90301e9bed5de16b7bb7bb6eb5510ed8102961b887957495963463697e41"}
Sep 18 18:11:17.196408 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:17.196306    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7e72d9f-54b7648c9b-z87kp"
Sep 18 18:11:17.198100 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:17.198066    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f7e72d9f-86676c8c8f-ptnrl" event={"ID":"19a9dc6f-6b51-4639-bfb5-4b4122a552ee","Type":"ContainerStarted","Data":"39ef5b47d9e86b9e7e648d8805c6a75b1b2b9d3a88609d1c7e1cf22aabe692e9"}
Sep 18 18:11:17.198248 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:17.198131    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-f7e72d9f-86676c8c8f-ptnrl"
Sep 18 18:11:17.198248 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:17.198150    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f7e72d9f-86676c8c8f-ptnrl" event={"ID":"19a9dc6f-6b51-4639-bfb5-4b4122a552ee","Type":"ContainerStarted","Data":"2d67a7e4e051d28689ebaa828a4204accf6dc15c711366d782a4f7621416653c"}
Sep 18 18:11:17.199758 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:17.199739    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f7e72d9f-54b9945bdb-zwjgm" event={"ID":"57c13b34-8239-4165-b2c4-8e512de7b5b3","Type":"ContainerStarted","Data":"5c08b7c0eb390c5c7361917f0779c76b3683c77efa0d93d9b692829a5c57be93"}
Sep 18 18:11:17.199870 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:17.199770    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f7e72d9f-54b9945bdb-zwjgm" event={"ID":"57c13b34-8239-4165-b2c4-8e512de7b5b3","Type":"ContainerStarted","Data":"86dc308301765e953d75e6a7ff822359c9e9ce9c318b3e2a3c17cbff5b8bb7b2"}
Sep 18 18:11:17.199870 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:17.199825    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-f7e72d9f-54b9945bdb-zwjgm"
Sep 18 18:11:17.221305 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:17.221254    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7e72d9f-54b7648c9b-z87kp" podStartSLOduration=1.784156015 podStartE2EDuration="2.221236663s" totalImagesPullingTime="437.080648ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:11:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:11:17.218771388 +0000 UTC m=+1281.845903042" watchObservedRunningTime="2026-09-18 18:11:17.221236663 +0000 UTC m=+1281.848368284"
Sep 18 18:11:17.238983 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:17.238941    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-f7e72d9f-86676c8c8f-ptnrl" podStartSLOduration=1.238928327 podStartE2EDuration="1.238928327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:11:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:11:17.237904315 +0000 UTC m=+1281.865035966" watchObservedRunningTime="2026-09-18 18:11:17.238928327 +0000 UTC m=+1281.866059995"
Sep 18 18:11:17.256907 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:17.256846    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-f7e72d9f-54b9945bdb-zwjgm" podStartSLOduration=1.256831056 podStartE2EDuration="1.256831056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:11:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:11:17.255169963 +0000 UTC m=+1281.882301584" watchObservedRunningTime="2026-09-18 18:11:17.256831056 +0000 UTC m=+1281.883962680"
Sep 18 18:11:23.210696 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:23.210665    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7e72d9f-54b7648c9b-z87kp"
Sep 18 18:11:28.207048 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:28.207016    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-f7e72d9f-86676c8c8f-ptnrl"
Sep 18 18:11:28.207447 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:28.207089    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-f7e72d9f-54b9945bdb-zwjgm"
Sep 18 18:11:39.044738 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:39.044698    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:11:39.849685 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:39.849649    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:11:40.367998 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:40.367961    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:11:40.581013 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:40.580978    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:11:41.652424 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:41.652391    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:11:42.488296 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:42.488260    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:11:43.452069 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:43.452037    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:11:44.392357 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:44.392327    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:11:44.763317 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:44.763287    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:11:45.148330 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:45.148300    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:11:46.013725 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:46.013699    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:11:46.844077 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:46.844040    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:11:47.850016 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:47.849978    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:11:51.672053 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:51.672013    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-502939-54d94b8c78-k54n7"]
Sep 18 18:11:51.672559 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:51.672283    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-502939-54d94b8c78-k54n7" podUID="9d4a5000-57d4-4443-b5fc-cafd2ef0ede3" containerName="maas-api" containerID="cri-o://fa86a863390ee0e790994a567cf76e491082f54bf5c9dd0e389dca2dbfa37420" gracePeriod=30
Sep 18 18:11:51.702300 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:51.702267    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:11:51.735171 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:51.735140    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-502939-6f789b6949-kfsvn"]
Sep 18 18:11:51.735408 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:51.735381    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-502939-6f789b6949-kfsvn" podUID="1685ba89-9c41-45a5-8793-a9e7d8d9d1da" containerName="payload-processing" containerID="cri-o://1e2a43de68b72462cb9b224089a5bab9b7c41930252ec56972d364c3cffa4317" gracePeriod=30
Sep 18 18:11:51.743746 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:51.743719    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-502939-56d954d58-zg4mx"]
Sep 18 18:11:51.744091 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:51.744059    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-502939-56d954d58-zg4mx" podUID="875d5495-8429-41d8-8625-e58b296b1886" containerName="payload-pre-processing" containerID="cri-o://6dc6510aafd2bb29b18dd668c1455e200d9f6b205b16a3e4e6efbdad157bd4c9" gracePeriod=30
Sep 18 18:11:51.785903 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:51.781935    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:11:51.862417 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:51.862381    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:11:51.931655 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:51.931519    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:11:52.013986 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:52.013958    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-502939-54d94b8c78-k54n7"
Sep 18 18:11:52.043751 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:52.043728    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-502939-6f789b6949-kfsvn"
Sep 18 18:11:52.057053 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:52.057024    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-502939-56d954d58-zg4mx"
Sep 18 18:11:52.107767 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:52.107722    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k4t8l\" (UniqueName: \"kubernetes.io/projected/9d4a5000-57d4-4443-b5fc-cafd2ef0ede3-kube-api-access-k4t8l\") pod \"9d4a5000-57d4-4443-b5fc-cafd2ef0ede3\" (UID: \"9d4a5000-57d4-4443-b5fc-cafd2ef0ede3\") "
Sep 18 18:11:52.107961 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:52.107814    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d4a5000-57d4-4443-b5fc-cafd2ef0ede3-maas-api-tls\") pod \"9d4a5000-57d4-4443-b5fc-cafd2ef0ede3\" (UID: \"9d4a5000-57d4-4443-b5fc-cafd2ef0ede3\") "
Sep 18 18:11:52.107961 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:52.107874    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d4a5000-57d4-4443-b5fc-cafd2ef0ede3-metrics-certs\") pod \"9d4a5000-57d4-4443-b5fc-cafd2ef0ede3\" (UID: \"9d4a5000-57d4-4443-b5fc-cafd2ef0ede3\") "
Sep 18 18:11:52.107961 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:52.107900    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d4a5000-57d4-4443-b5fc-cafd2ef0ede3-openshift-service-ca\") pod \"9d4a5000-57d4-4443-b5fc-cafd2ef0ede3\" (UID: \"9d4a5000-57d4-4443-b5fc-cafd2ef0ede3\") "
Sep 18 18:11:52.108421 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:52.108309    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d4a5000-57d4-4443-b5fc-cafd2ef0ede3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9d4a5000-57d4-4443-b5fc-cafd2ef0ede3" (UID: "9d4a5000-57d4-4443-b5fc-cafd2ef0ede3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:11:52.111187 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:52.111150    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d4a5000-57d4-4443-b5fc-cafd2ef0ede3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9d4a5000-57d4-4443-b5fc-cafd2ef0ede3" (UID: "9d4a5000-57d4-4443-b5fc-cafd2ef0ede3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:11:52.111187 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:52.111162    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d4a5000-57d4-4443-b5fc-cafd2ef0ede3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9d4a5000-57d4-4443-b5fc-cafd2ef0ede3" (UID: "9d4a5000-57d4-4443-b5fc-cafd2ef0ede3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:11:52.111187 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:52.111176    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d4a5000-57d4-4443-b5fc-cafd2ef0ede3-kube-api-access-k4t8l" (OuterVolumeSpecName: "kube-api-access-k4t8l") pod "9d4a5000-57d4-4443-b5fc-cafd2ef0ede3" (UID: "9d4a5000-57d4-4443-b5fc-cafd2ef0ede3"). InnerVolumeSpecName "kube-api-access-k4t8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:11:52.209219 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:52.209138    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/875d5495-8429-41d8-8625-e58b296b1886-plugins-config-volume\") pod \"875d5495-8429-41d8-8625-e58b296b1886\" (UID: \"875d5495-8429-41d8-8625-e58b296b1886\") "
Sep 18 18:11:52.209361 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:52.209295    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l9zl8\" (UniqueName: \"kubernetes.io/projected/1685ba89-9c41-45a5-8793-a9e7d8d9d1da-kube-api-access-l9zl8\") pod \"1685ba89-9c41-45a5-8793-a9e7d8d9d1da\" (UID: \"1685ba89-9c41-45a5-8793-a9e7d8d9d1da\") "
Sep 18 18:11:52.209361 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:52.209349    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1685ba89-9c41-45a5-8793-a9e7d8d9d1da-plugins-config-volume\") pod \"1685ba89-9c41-45a5-8793-a9e7d8d9d1da\" (UID: \"1685ba89-9c41-45a5-8793-a9e7d8d9d1da\") "
Sep 18 18:11:52.209439 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:52.209400    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h58xf\" (UniqueName: \"kubernetes.io/projected/875d5495-8429-41d8-8625-e58b296b1886-kube-api-access-h58xf\") pod \"875d5495-8429-41d8-8625-e58b296b1886\" (UID: \"875d5495-8429-41d8-8625-e58b296b1886\") "
Sep 18 18:11:52.209474 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:52.209448    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/875d5495-8429-41d8-8625-e58b296b1886-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "875d5495-8429-41d8-8625-e58b296b1886" (UID: "875d5495-8429-41d8-8625-e58b296b1886"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:11:52.209709 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:52.209683    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1685ba89-9c41-45a5-8793-a9e7d8d9d1da-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1685ba89-9c41-45a5-8793-a9e7d8d9d1da" (UID: "1685ba89-9c41-45a5-8793-a9e7d8d9d1da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:11:52.209837 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:52.209814    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d4a5000-57d4-4443-b5fc-cafd2ef0ede3-maas-api-tls\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:11:52.209886 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:52.209838    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d4a5000-57d4-4443-b5fc-cafd2ef0ede3-metrics-certs\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:11:52.209886 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:52.209849    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d4a5000-57d4-4443-b5fc-cafd2ef0ede3-openshift-service-ca\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:11:52.209886 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:52.209858    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/875d5495-8429-41d8-8625-e58b296b1886-plugins-config-volume\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:11:52.209886 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:52.209867    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4t8l\" (UniqueName: \"kubernetes.io/projected/9d4a5000-57d4-4443-b5fc-cafd2ef0ede3-kube-api-access-k4t8l\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:11:52.209886 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:52.209876    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1685ba89-9c41-45a5-8793-a9e7d8d9d1da-plugins-config-volume\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:11:52.211497 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:52.211477    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/875d5495-8429-41d8-8625-e58b296b1886-kube-api-access-h58xf" (OuterVolumeSpecName: "kube-api-access-h58xf") pod "875d5495-8429-41d8-8625-e58b296b1886" (UID: "875d5495-8429-41d8-8625-e58b296b1886"). InnerVolumeSpecName "kube-api-access-h58xf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:11:52.211554 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:52.211509    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1685ba89-9c41-45a5-8793-a9e7d8d9d1da-kube-api-access-l9zl8" (OuterVolumeSpecName: "kube-api-access-l9zl8") pod "1685ba89-9c41-45a5-8793-a9e7d8d9d1da" (UID: "1685ba89-9c41-45a5-8793-a9e7d8d9d1da"). InnerVolumeSpecName "kube-api-access-l9zl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:11:52.311226 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:52.311198    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9zl8\" (UniqueName: \"kubernetes.io/projected/1685ba89-9c41-45a5-8793-a9e7d8d9d1da-kube-api-access-l9zl8\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:11:52.311226 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:52.311222    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h58xf\" (UniqueName: \"kubernetes.io/projected/875d5495-8429-41d8-8625-e58b296b1886-kube-api-access-h58xf\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:11:52.324110 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:52.324083    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:11:52.410099 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:52.410064    2561 generic.go:356] "Generic (PLEG): container finished" podID="875d5495-8429-41d8-8625-e58b296b1886" containerID="6dc6510aafd2bb29b18dd668c1455e200d9f6b205b16a3e4e6efbdad157bd4c9" exitCode=0
Sep 18 18:11:52.410280 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:52.410144    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-502939-56d954d58-zg4mx"
Sep 18 18:11:52.410280 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:52.410153    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-502939-56d954d58-zg4mx" event={"ID":"875d5495-8429-41d8-8625-e58b296b1886","Type":"ContainerDied","Data":"6dc6510aafd2bb29b18dd668c1455e200d9f6b205b16a3e4e6efbdad157bd4c9"}
Sep 18 18:11:52.410280 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:52.410218    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-502939-56d954d58-zg4mx" event={"ID":"875d5495-8429-41d8-8625-e58b296b1886","Type":"ContainerDied","Data":"22e02a6d64e2307a63c0cedd77fe1b80c5f8eb5466a2dca72f1770496f166d96"}
Sep 18 18:11:52.410280 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:52.410248    2561 scope.go:117] "RemoveContainer" containerID="6dc6510aafd2bb29b18dd668c1455e200d9f6b205b16a3e4e6efbdad157bd4c9"
Sep 18 18:11:52.411544 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:52.411515    2561 generic.go:356] "Generic (PLEG): container finished" podID="1685ba89-9c41-45a5-8793-a9e7d8d9d1da" containerID="1e2a43de68b72462cb9b224089a5bab9b7c41930252ec56972d364c3cffa4317" exitCode=0
Sep 18 18:11:52.411686 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:52.411560    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-502939-6f789b6949-kfsvn"
Sep 18 18:11:52.411686 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:52.411597    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-502939-6f789b6949-kfsvn" event={"ID":"1685ba89-9c41-45a5-8793-a9e7d8d9d1da","Type":"ContainerDied","Data":"1e2a43de68b72462cb9b224089a5bab9b7c41930252ec56972d364c3cffa4317"}
Sep 18 18:11:52.411686 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:52.411667    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-502939-6f789b6949-kfsvn" event={"ID":"1685ba89-9c41-45a5-8793-a9e7d8d9d1da","Type":"ContainerDied","Data":"3c46ac5a34c115eb52015ad53a251c3abd31dcb7b5cf5d0e34aa2d8ea6b6b1ae"}
Sep 18 18:11:52.413315 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:52.413287    2561 generic.go:356] "Generic (PLEG): container finished" podID="9d4a5000-57d4-4443-b5fc-cafd2ef0ede3" containerID="fa86a863390ee0e790994a567cf76e491082f54bf5c9dd0e389dca2dbfa37420" exitCode=0
Sep 18 18:11:52.413447 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:52.413324    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-502939-54d94b8c78-k54n7"
Sep 18 18:11:52.413447 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:52.413361    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-502939-54d94b8c78-k54n7" event={"ID":"9d4a5000-57d4-4443-b5fc-cafd2ef0ede3","Type":"ContainerDied","Data":"fa86a863390ee0e790994a567cf76e491082f54bf5c9dd0e389dca2dbfa37420"}
Sep 18 18:11:52.413447 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:52.413399    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-502939-54d94b8c78-k54n7" event={"ID":"9d4a5000-57d4-4443-b5fc-cafd2ef0ede3","Type":"ContainerDied","Data":"77b883374f0eafc8b38de4a90eb1d5d2ef84f49b419534fa03ea3bcba56df370"}
Sep 18 18:11:52.421064 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:52.421044    2561 scope.go:117] "RemoveContainer" containerID="6dc6510aafd2bb29b18dd668c1455e200d9f6b205b16a3e4e6efbdad157bd4c9"
Sep 18 18:11:52.421381 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:11:52.421362    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dc6510aafd2bb29b18dd668c1455e200d9f6b205b16a3e4e6efbdad157bd4c9\": container with ID starting with 6dc6510aafd2bb29b18dd668c1455e200d9f6b205b16a3e4e6efbdad157bd4c9 not found: ID does not exist" containerID="6dc6510aafd2bb29b18dd668c1455e200d9f6b205b16a3e4e6efbdad157bd4c9"
Sep 18 18:11:52.421465 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:52.421388    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dc6510aafd2bb29b18dd668c1455e200d9f6b205b16a3e4e6efbdad157bd4c9"} err="failed to get container status \"6dc6510aafd2bb29b18dd668c1455e200d9f6b205b16a3e4e6efbdad157bd4c9\": rpc error: code = NotFound desc = could not find container \"6dc6510aafd2bb29b18dd668c1455e200d9f6b205b16a3e4e6efbdad157bd4c9\": container with ID starting with 6dc6510aafd2bb29b18dd668c1455e200d9f6b205b16a3e4e6efbdad157bd4c9 not found: ID does not exist"
Sep 18 18:11:52.421465 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:52.421403    2561 scope.go:117] "RemoveContainer" containerID="1e2a43de68b72462cb9b224089a5bab9b7c41930252ec56972d364c3cffa4317"
Sep 18 18:11:52.430387 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:52.430355    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:11:52.430851 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:52.430837    2561 scope.go:117] "RemoveContainer" containerID="1e2a43de68b72462cb9b224089a5bab9b7c41930252ec56972d364c3cffa4317"
Sep 18 18:11:52.431098 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:11:52.431083    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e2a43de68b72462cb9b224089a5bab9b7c41930252ec56972d364c3cffa4317\": container with ID starting with 1e2a43de68b72462cb9b224089a5bab9b7c41930252ec56972d364c3cffa4317 not found: ID does not exist" containerID="1e2a43de68b72462cb9b224089a5bab9b7c41930252ec56972d364c3cffa4317"
Sep 18 18:11:52.431163 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:52.431105    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e2a43de68b72462cb9b224089a5bab9b7c41930252ec56972d364c3cffa4317"} err="failed to get container status \"1e2a43de68b72462cb9b224089a5bab9b7c41930252ec56972d364c3cffa4317\": rpc error: code = NotFound desc = could not find container \"1e2a43de68b72462cb9b224089a5bab9b7c41930252ec56972d364c3cffa4317\": container with ID starting with 1e2a43de68b72462cb9b224089a5bab9b7c41930252ec56972d364c3cffa4317 not found: ID does not exist"
Sep 18 18:11:52.431163 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:52.431136    2561 scope.go:117] "RemoveContainer" containerID="fa86a863390ee0e790994a567cf76e491082f54bf5c9dd0e389dca2dbfa37420"
Sep 18 18:11:52.435525 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:52.435505    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-502939-56d954d58-zg4mx"]
Sep 18 18:11:52.440365 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:52.440345    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-502939-56d954d58-zg4mx"]
Sep 18 18:11:52.440575 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:52.440551    2561 scope.go:117] "RemoveContainer" containerID="fa86a863390ee0e790994a567cf76e491082f54bf5c9dd0e389dca2dbfa37420"
Sep 18 18:11:52.440854 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:11:52.440827    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa86a863390ee0e790994a567cf76e491082f54bf5c9dd0e389dca2dbfa37420\": container with ID starting with fa86a863390ee0e790994a567cf76e491082f54bf5c9dd0e389dca2dbfa37420 not found: ID does not exist" containerID="fa86a863390ee0e790994a567cf76e491082f54bf5c9dd0e389dca2dbfa37420"
Sep 18 18:11:52.440939 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:52.440864    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa86a863390ee0e790994a567cf76e491082f54bf5c9dd0e389dca2dbfa37420"} err="failed to get container status \"fa86a863390ee0e790994a567cf76e491082f54bf5c9dd0e389dca2dbfa37420\": rpc error: code = NotFound desc = could not find container \"fa86a863390ee0e790994a567cf76e491082f54bf5c9dd0e389dca2dbfa37420\": container with ID starting with fa86a863390ee0e790994a567cf76e491082f54bf5c9dd0e389dca2dbfa37420 not found: ID does not exist"
Sep 18 18:11:52.453625 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:52.453599    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-502939-6f789b6949-kfsvn"]
Sep 18 18:11:52.455770 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:52.455748    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-502939-6f789b6949-kfsvn"]
Sep 18 18:11:52.465435 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:52.465375    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-502939-54d94b8c78-k54n7"]
Sep 18 18:11:52.473340 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:52.473318    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-502939-54d94b8c78-k54n7"]
Sep 18 18:11:53.172155 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:53.172094    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:11:53.861911 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:53.861878    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1685ba89-9c41-45a5-8793-a9e7d8d9d1da" path="/var/lib/kubelet/pods/1685ba89-9c41-45a5-8793-a9e7d8d9d1da/volumes"
Sep 18 18:11:53.862291 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:53.862277    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="875d5495-8429-41d8-8625-e58b296b1886" path="/var/lib/kubelet/pods/875d5495-8429-41d8-8625-e58b296b1886/volumes"
Sep 18 18:11:53.862631 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:53.862619    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d4a5000-57d4-4443-b5fc-cafd2ef0ede3" path="/var/lib/kubelet/pods/9d4a5000-57d4-4443-b5fc-cafd2ef0ede3/volumes"
Sep 18 18:11:54.257020 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:54.256983    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:11:54.956535 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:54.956505    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:11:55.799976 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:55.799944    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:11:56.656408 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:56.656379    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:11:57.439991 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:57.439941    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:11:57.655211 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:11:57.655181    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:12:21.885268 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:21.885192    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:12:29.635583 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:29.635551    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-2c937c/noten-model-2c937c-kserve-54f7c6f799-b8dfl"]
Sep 18 18:12:29.636087 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:29.635976    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d4a5000-57d4-4443-b5fc-cafd2ef0ede3" containerName="maas-api"
Sep 18 18:12:29.636087 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:29.635987    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d4a5000-57d4-4443-b5fc-cafd2ef0ede3" containerName="maas-api"
Sep 18 18:12:29.636087 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:29.635996    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1685ba89-9c41-45a5-8793-a9e7d8d9d1da" containerName="payload-processing"
Sep 18 18:12:29.636087 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:29.636002    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1685ba89-9c41-45a5-8793-a9e7d8d9d1da" containerName="payload-processing"
Sep 18 18:12:29.636087 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:29.636030    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="875d5495-8429-41d8-8625-e58b296b1886" containerName="payload-pre-processing"
Sep 18 18:12:29.636087 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:29.636035    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="875d5495-8429-41d8-8625-e58b296b1886" containerName="payload-pre-processing"
Sep 18 18:12:29.636087 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:29.636094    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="875d5495-8429-41d8-8625-e58b296b1886" containerName="payload-pre-processing"
Sep 18 18:12:29.636474 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:29.636102    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1685ba89-9c41-45a5-8793-a9e7d8d9d1da" containerName="payload-processing"
Sep 18 18:12:29.636474 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:29.636135    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d4a5000-57d4-4443-b5fc-cafd2ef0ede3" containerName="maas-api"
Sep 18 18:12:29.639355 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:29.639335    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-2c937c/noten-model-2c937c-kserve-54f7c6f799-b8dfl"
Sep 18 18:12:29.642766 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:29.642740    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-2c937c\"/\"noten-model-2c937c-kserve-self-signed-certs\""
Sep 18 18:12:29.642899 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:29.642740    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-2c937c\"/\"default-dockercfg-qr4jc\""
Sep 18 18:12:29.642899 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:29.642751    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-2c937c\"/\"kube-root-ca.crt\""
Sep 18 18:12:29.642899 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:29.642854    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-2c937c\"/\"openshift-service-ca.crt\""
Sep 18 18:12:29.650585 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:29.650565    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-2c937c/noten-model-2c937c-kserve-54f7c6f799-b8dfl"]
Sep 18 18:12:29.743413 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:29.743380    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:12:29.780660 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:29.780632    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426-home\") pod \"noten-model-2c937c-kserve-54f7c6f799-b8dfl\" (UID: \"8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426\") " pod="ai-tenant-e2e-noten-2c937c/noten-model-2c937c-kserve-54f7c6f799-b8dfl"
Sep 18 18:12:29.780802 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:29.780695    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426-tls-certs\") pod \"noten-model-2c937c-kserve-54f7c6f799-b8dfl\" (UID: \"8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426\") " pod="ai-tenant-e2e-noten-2c937c/noten-model-2c937c-kserve-54f7c6f799-b8dfl"
Sep 18 18:12:29.780802 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:29.780729    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426-tmp-dir\") pod \"noten-model-2c937c-kserve-54f7c6f799-b8dfl\" (UID: \"8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426\") " pod="ai-tenant-e2e-noten-2c937c/noten-model-2c937c-kserve-54f7c6f799-b8dfl"
Sep 18 18:12:29.780802 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:29.780756    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dmcn\" (UniqueName: \"kubernetes.io/projected/8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426-kube-api-access-2dmcn\") pod \"noten-model-2c937c-kserve-54f7c6f799-b8dfl\" (UID: \"8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426\") " pod="ai-tenant-e2e-noten-2c937c/noten-model-2c937c-kserve-54f7c6f799-b8dfl"
Sep 18 18:12:29.780802 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:29.780771    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426-dshm\") pod \"noten-model-2c937c-kserve-54f7c6f799-b8dfl\" (UID: \"8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426\") " pod="ai-tenant-e2e-noten-2c937c/noten-model-2c937c-kserve-54f7c6f799-b8dfl"
Sep 18 18:12:29.780948 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:29.780821    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426-model-cache\") pod \"noten-model-2c937c-kserve-54f7c6f799-b8dfl\" (UID: \"8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426\") " pod="ai-tenant-e2e-noten-2c937c/noten-model-2c937c-kserve-54f7c6f799-b8dfl"
Sep 18 18:12:29.881859 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:29.881817    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426-home\") pod \"noten-model-2c937c-kserve-54f7c6f799-b8dfl\" (UID: \"8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426\") " pod="ai-tenant-e2e-noten-2c937c/noten-model-2c937c-kserve-54f7c6f799-b8dfl"
Sep 18 18:12:29.882057 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:29.881913    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426-tls-certs\") pod \"noten-model-2c937c-kserve-54f7c6f799-b8dfl\" (UID: \"8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426\") " pod="ai-tenant-e2e-noten-2c937c/noten-model-2c937c-kserve-54f7c6f799-b8dfl"
Sep 18 18:12:29.882057 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:29.881962    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426-tmp-dir\") pod \"noten-model-2c937c-kserve-54f7c6f799-b8dfl\" (UID: \"8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426\") " pod="ai-tenant-e2e-noten-2c937c/noten-model-2c937c-kserve-54f7c6f799-b8dfl"
Sep 18 18:12:29.882057 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:29.882004    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dmcn\" (UniqueName: \"kubernetes.io/projected/8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426-kube-api-access-2dmcn\") pod \"noten-model-2c937c-kserve-54f7c6f799-b8dfl\" (UID: \"8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426\") " pod="ai-tenant-e2e-noten-2c937c/noten-model-2c937c-kserve-54f7c6f799-b8dfl"
Sep 18 18:12:29.882057 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:29.882028    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426-dshm\") pod \"noten-model-2c937c-kserve-54f7c6f799-b8dfl\" (UID: \"8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426\") " pod="ai-tenant-e2e-noten-2c937c/noten-model-2c937c-kserve-54f7c6f799-b8dfl"
Sep 18 18:12:29.882309 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:29.882073    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426-model-cache\") pod \"noten-model-2c937c-kserve-54f7c6f799-b8dfl\" (UID: \"8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426\") " pod="ai-tenant-e2e-noten-2c937c/noten-model-2c937c-kserve-54f7c6f799-b8dfl"
Sep 18 18:12:29.882376 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:29.882316    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426-home\") pod \"noten-model-2c937c-kserve-54f7c6f799-b8dfl\" (UID: \"8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426\") " pod="ai-tenant-e2e-noten-2c937c/noten-model-2c937c-kserve-54f7c6f799-b8dfl"
Sep 18 18:12:29.882438 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:29.882388    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426-tmp-dir\") pod \"noten-model-2c937c-kserve-54f7c6f799-b8dfl\" (UID: \"8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426\") " pod="ai-tenant-e2e-noten-2c937c/noten-model-2c937c-kserve-54f7c6f799-b8dfl"
Sep 18 18:12:29.882543 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:29.882522    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426-model-cache\") pod \"noten-model-2c937c-kserve-54f7c6f799-b8dfl\" (UID: \"8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426\") " pod="ai-tenant-e2e-noten-2c937c/noten-model-2c937c-kserve-54f7c6f799-b8dfl"
Sep 18 18:12:29.884198 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:29.884180    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426-dshm\") pod \"noten-model-2c937c-kserve-54f7c6f799-b8dfl\" (UID: \"8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426\") " pod="ai-tenant-e2e-noten-2c937c/noten-model-2c937c-kserve-54f7c6f799-b8dfl"
Sep 18 18:12:29.884498 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:29.884483    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426-tls-certs\") pod \"noten-model-2c937c-kserve-54f7c6f799-b8dfl\" (UID: \"8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426\") " pod="ai-tenant-e2e-noten-2c937c/noten-model-2c937c-kserve-54f7c6f799-b8dfl"
Sep 18 18:12:29.892921 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:29.892863    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dmcn\" (UniqueName: \"kubernetes.io/projected/8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426-kube-api-access-2dmcn\") pod \"noten-model-2c937c-kserve-54f7c6f799-b8dfl\" (UID: \"8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426\") " pod="ai-tenant-e2e-noten-2c937c/noten-model-2c937c-kserve-54f7c6f799-b8dfl"
Sep 18 18:12:29.949907 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:29.949871    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-2c937c/noten-model-2c937c-kserve-54f7c6f799-b8dfl"
Sep 18 18:12:30.226641 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:30.226617    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-2c937c/noten-model-2c937c-kserve-54f7c6f799-b8dfl"]
Sep 18 18:12:30.228709 ip-10-0-140-54 kubenswrapper[2561]: W0918 18:12:30.228683    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8fa06fc6_6eaa_4543_a3a9_c2a14cdb0426.slice/crio-7f9a9a78dd54db694e7c00612d530d6cbf3bbcb88b044f20bdb41da1d3c908e9 WatchSource:0}: Error finding container 7f9a9a78dd54db694e7c00612d530d6cbf3bbcb88b044f20bdb41da1d3c908e9: Status 404 returned error can't find the container with id 7f9a9a78dd54db694e7c00612d530d6cbf3bbcb88b044f20bdb41da1d3c908e9
Sep 18 18:12:30.616758 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:30.616671    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-2c937c/noten-model-2c937c-kserve-54f7c6f799-b8dfl" event={"ID":"8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426","Type":"ContainerStarted","Data":"b0425e9b645bc5dd70ccbfa0f7b053854ec76fa5cbe57d9ad63007230030aafd"}
Sep 18 18:12:30.616758 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:30.616709    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-2c937c/noten-model-2c937c-kserve-54f7c6f799-b8dfl" event={"ID":"8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426","Type":"ContainerStarted","Data":"7f9a9a78dd54db694e7c00612d530d6cbf3bbcb88b044f20bdb41da1d3c908e9"}
Sep 18 18:12:30.635893 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:30.635842    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-2c937c/noten-model-2c937c-kserve-54f7c6f799-b8dfl" podStartSLOduration=1.6358233580000001 podStartE2EDuration="1.635823358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:12:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:12:30.635298319 +0000 UTC m=+1355.262429940" watchObservedRunningTime="2026-09-18 18:12:30.635823358 +0000 UTC m=+1355.262954977"
Sep 18 18:12:33.133754 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:33.133712    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:12:34.489890 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:34.489858    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:12:34.759911 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:34.759829    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:12:35.264007 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:35.263975    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:12:36.221672 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:36.221643    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:12:37.061445 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:37.061413    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:12:37.957995 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:37.957960    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:12:38.892459 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:38.892423    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:12:39.628947 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:39.628914    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:12:39.949974 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:39.949944    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-2c937c/noten-model-2c937c-kserve-54f7c6f799-b8dfl"
Sep 18 18:12:39.950187 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:39.950016    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-2c937c/noten-model-2c937c-kserve-54f7c6f799-b8dfl"
Sep 18 18:12:39.962862 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:39.962840    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-2c937c/noten-model-2c937c-kserve-54f7c6f799-b8dfl"
Sep 18 18:12:40.653571 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:40.653531    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:12:40.681730 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:40.681700    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-2c937c/noten-model-2c937c-kserve-54f7c6f799-b8dfl"
Sep 18 18:12:41.481470 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:41.481442    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:12:45.593205 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:45.593169    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ee3049-7cc8b98769-s9jj5"]
Sep 18 18:12:45.593687 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:45.593469    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ee3049-7cc8b98769-s9jj5" podUID="843c8a1a-4277-4e30-9fe3-1d726d6f0efb" containerName="maas-api" containerID="cri-o://fd61df0b726e0d3c517d80ea3ec7c9e6734091033a74683315828603e28bc237" gracePeriod=30
Sep 18 18:12:45.626280 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:45.626254    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:12:45.729646 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:45.729618    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:12:45.830162 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:45.830106    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:12:45.845398 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:45.845342    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ee3049-7cc8b98769-s9jj5"
Sep 18 18:12:45.928534 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:45.928506    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:12:45.936932 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:45.936910    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-njtkb\" (UniqueName: \"kubernetes.io/projected/843c8a1a-4277-4e30-9fe3-1d726d6f0efb-kube-api-access-njtkb\") pod \"843c8a1a-4277-4e30-9fe3-1d726d6f0efb\" (UID: \"843c8a1a-4277-4e30-9fe3-1d726d6f0efb\") "
Sep 18 18:12:45.937092 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:45.937011    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/843c8a1a-4277-4e30-9fe3-1d726d6f0efb-openshift-service-ca\") pod \"843c8a1a-4277-4e30-9fe3-1d726d6f0efb\" (UID: \"843c8a1a-4277-4e30-9fe3-1d726d6f0efb\") "
Sep 18 18:12:45.937092 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:45.937047    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/843c8a1a-4277-4e30-9fe3-1d726d6f0efb-maas-api-tls\") pod \"843c8a1a-4277-4e30-9fe3-1d726d6f0efb\" (UID: \"843c8a1a-4277-4e30-9fe3-1d726d6f0efb\") "
Sep 18 18:12:45.937231 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:45.937182    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/843c8a1a-4277-4e30-9fe3-1d726d6f0efb-metrics-certs\") pod \"843c8a1a-4277-4e30-9fe3-1d726d6f0efb\" (UID: \"843c8a1a-4277-4e30-9fe3-1d726d6f0efb\") "
Sep 18 18:12:45.937405 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:45.937365    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/843c8a1a-4277-4e30-9fe3-1d726d6f0efb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "843c8a1a-4277-4e30-9fe3-1d726d6f0efb" (UID: "843c8a1a-4277-4e30-9fe3-1d726d6f0efb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:12:45.937668 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:45.937647    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/843c8a1a-4277-4e30-9fe3-1d726d6f0efb-openshift-service-ca\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:12:45.939228 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:45.939205    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/843c8a1a-4277-4e30-9fe3-1d726d6f0efb-kube-api-access-njtkb" (OuterVolumeSpecName: "kube-api-access-njtkb") pod "843c8a1a-4277-4e30-9fe3-1d726d6f0efb" (UID: "843c8a1a-4277-4e30-9fe3-1d726d6f0efb"). InnerVolumeSpecName "kube-api-access-njtkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:12:45.939311 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:45.939222    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/843c8a1a-4277-4e30-9fe3-1d726d6f0efb-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "843c8a1a-4277-4e30-9fe3-1d726d6f0efb" (UID: "843c8a1a-4277-4e30-9fe3-1d726d6f0efb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:12:45.939311 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:45.939238    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/843c8a1a-4277-4e30-9fe3-1d726d6f0efb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "843c8a1a-4277-4e30-9fe3-1d726d6f0efb" (UID: "843c8a1a-4277-4e30-9fe3-1d726d6f0efb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:12:46.038780 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:46.038748    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njtkb\" (UniqueName: \"kubernetes.io/projected/843c8a1a-4277-4e30-9fe3-1d726d6f0efb-kube-api-access-njtkb\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:12:46.038780 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:46.038775    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/843c8a1a-4277-4e30-9fe3-1d726d6f0efb-maas-api-tls\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:12:46.038780 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:46.038784    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/843c8a1a-4277-4e30-9fe3-1d726d6f0efb-metrics-certs\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:12:46.214369 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:46.214338    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:12:46.320957 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:46.320928    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:12:46.706884 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:46.706851    2561 generic.go:356] "Generic (PLEG): container finished" podID="843c8a1a-4277-4e30-9fe3-1d726d6f0efb" containerID="fd61df0b726e0d3c517d80ea3ec7c9e6734091033a74683315828603e28bc237" exitCode=0
Sep 18 18:12:46.706884 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:46.706891    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ee3049-7cc8b98769-s9jj5"
Sep 18 18:12:46.707411 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:46.706931    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ee3049-7cc8b98769-s9jj5" event={"ID":"843c8a1a-4277-4e30-9fe3-1d726d6f0efb","Type":"ContainerDied","Data":"fd61df0b726e0d3c517d80ea3ec7c9e6734091033a74683315828603e28bc237"}
Sep 18 18:12:46.707411 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:46.706987    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ee3049-7cc8b98769-s9jj5" event={"ID":"843c8a1a-4277-4e30-9fe3-1d726d6f0efb","Type":"ContainerDied","Data":"4ddb541ef86e290c912983130e29cde48b7e337827bdc42b050c239e201db098"}
Sep 18 18:12:46.707411 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:46.707017    2561 scope.go:117] "RemoveContainer" containerID="fd61df0b726e0d3c517d80ea3ec7c9e6734091033a74683315828603e28bc237"
Sep 18 18:12:46.718474 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:46.718455    2561 scope.go:117] "RemoveContainer" containerID="fd61df0b726e0d3c517d80ea3ec7c9e6734091033a74683315828603e28bc237"
Sep 18 18:12:46.718753 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:12:46.718737    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd61df0b726e0d3c517d80ea3ec7c9e6734091033a74683315828603e28bc237\": container with ID starting with fd61df0b726e0d3c517d80ea3ec7c9e6734091033a74683315828603e28bc237 not found: ID does not exist" containerID="fd61df0b726e0d3c517d80ea3ec7c9e6734091033a74683315828603e28bc237"
Sep 18 18:12:46.718798 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:46.718760    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd61df0b726e0d3c517d80ea3ec7c9e6734091033a74683315828603e28bc237"} err="failed to get container status \"fd61df0b726e0d3c517d80ea3ec7c9e6734091033a74683315828603e28bc237\": rpc error: code = NotFound desc = could not find container \"fd61df0b726e0d3c517d80ea3ec7c9e6734091033a74683315828603e28bc237\": container with ID starting with fd61df0b726e0d3c517d80ea3ec7c9e6734091033a74683315828603e28bc237 not found: ID does not exist"
Sep 18 18:12:46.742275 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:46.742244    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ee3049-7cc8b98769-s9jj5"]
Sep 18 18:12:46.753609 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:46.753577    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ee3049-7cc8b98769-s9jj5"]
Sep 18 18:12:47.147888 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:47.147859    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:12:47.862543 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:47.862508    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="843c8a1a-4277-4e30-9fe3-1d726d6f0efb" path="/var/lib/kubelet/pods/843c8a1a-4277-4e30-9fe3-1d726d6f0efb/volumes"
Sep 18 18:12:47.894904 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:47.894874    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:12:48.747954 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:48.747923    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:12:49.621401 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:49.621370    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:12:50.494010 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:50.493979    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:12:51.289395 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:51.289352    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:12:54.674556 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:54.674530    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:12:59.024491 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:12:59.024459    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:13:02.063282 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:02.063249    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:13:02.258046 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:02.258004    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:13:02.426006 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:02.425975    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:13:07.345406 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:07.345373    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-2c937c/noten-model-2c937c-kserve-54f7c6f799-b8dfl"]
Sep 18 18:13:07.463318 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:07.463290    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:13:19.912856 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:19.912817    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:13:20.350949 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:20.350866    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:13:21.471718 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:21.471685    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:13:21.569863 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:21.569813    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:13:22.279782 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:22.279753    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:13:22.364657 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:22.364595    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-2c937c/noten-model-2c937c-kserve-54f7c6f799-b8dfl" podUID="8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426" containerName="main" containerID="cri-o://b0425e9b645bc5dd70ccbfa0f7b053854ec76fa5cbe57d9ad63007230030aafd" gracePeriod=45
Sep 18 18:13:22.624638 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:22.624614    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-2c937c/noten-model-2c937c-kserve-54f7c6f799-b8dfl"
Sep 18 18:13:22.692395 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:22.692355    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426-home\") pod \"8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426\" (UID: \"8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426\") "
Sep 18 18:13:22.692395 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:22.692401    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426-dshm\") pod \"8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426\" (UID: \"8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426\") "
Sep 18 18:13:22.692649 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:22.692425    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426-tls-certs\") pod \"8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426\" (UID: \"8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426\") "
Sep 18 18:13:22.692649 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:22.692456    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426-tmp-dir\") pod \"8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426\" (UID: \"8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426\") "
Sep 18 18:13:22.692649 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:22.692510    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2dmcn\" (UniqueName: \"kubernetes.io/projected/8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426-kube-api-access-2dmcn\") pod \"8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426\" (UID: \"8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426\") "
Sep 18 18:13:22.692649 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:22.692535    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426-model-cache\") pod \"8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426\" (UID: \"8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426\") "
Sep 18 18:13:22.692649 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:22.692593    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426-home" (OuterVolumeSpecName: "home") pod "8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426" (UID: "8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:13:22.692928 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:22.692762    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426" (UID: "8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:13:22.692928 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:22.692856    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426-model-cache" (OuterVolumeSpecName: "model-cache") pod "8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426" (UID: "8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:13:22.692928 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:22.692889    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426-home\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:13:22.692928 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:22.692901    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426-tmp-dir\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:13:22.694741 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:22.694712    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426-dshm" (OuterVolumeSpecName: "dshm") pod "8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426" (UID: "8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:13:22.694854 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:22.694757    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426" (UID: "8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:13:22.694895 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:22.694875    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426-kube-api-access-2dmcn" (OuterVolumeSpecName: "kube-api-access-2dmcn") pod "8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426" (UID: "8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426"). InnerVolumeSpecName "kube-api-access-2dmcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:13:22.794249 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:22.794176    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426-tls-certs\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:13:22.794249 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:22.794201    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dmcn\" (UniqueName: \"kubernetes.io/projected/8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426-kube-api-access-2dmcn\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:13:22.794249 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:22.794211    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426-model-cache\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:13:22.794249 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:22.794219    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426-dshm\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:13:22.916738 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:22.916707    2561 generic.go:356] "Generic (PLEG): container finished" podID="8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426" containerID="b0425e9b645bc5dd70ccbfa0f7b053854ec76fa5cbe57d9ad63007230030aafd" exitCode=0
Sep 18 18:13:22.916908 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:22.916745    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-2c937c/noten-model-2c937c-kserve-54f7c6f799-b8dfl"
Sep 18 18:13:22.916908 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:22.916761    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-2c937c/noten-model-2c937c-kserve-54f7c6f799-b8dfl" event={"ID":"8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426","Type":"ContainerDied","Data":"b0425e9b645bc5dd70ccbfa0f7b053854ec76fa5cbe57d9ad63007230030aafd"}
Sep 18 18:13:22.916908 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:22.916799    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-2c937c/noten-model-2c937c-kserve-54f7c6f799-b8dfl" event={"ID":"8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426","Type":"ContainerDied","Data":"7f9a9a78dd54db694e7c00612d530d6cbf3bbcb88b044f20bdb41da1d3c908e9"}
Sep 18 18:13:22.916908 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:22.916824    2561 scope.go:117] "RemoveContainer" containerID="b0425e9b645bc5dd70ccbfa0f7b053854ec76fa5cbe57d9ad63007230030aafd"
Sep 18 18:13:22.932210 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:22.932191    2561 scope.go:117] "RemoveContainer" containerID="b0425e9b645bc5dd70ccbfa0f7b053854ec76fa5cbe57d9ad63007230030aafd"
Sep 18 18:13:22.932546 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:13:22.932516    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0425e9b645bc5dd70ccbfa0f7b053854ec76fa5cbe57d9ad63007230030aafd\": container with ID starting with b0425e9b645bc5dd70ccbfa0f7b053854ec76fa5cbe57d9ad63007230030aafd not found: ID does not exist" containerID="b0425e9b645bc5dd70ccbfa0f7b053854ec76fa5cbe57d9ad63007230030aafd"
Sep 18 18:13:22.932625 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:22.932551    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0425e9b645bc5dd70ccbfa0f7b053854ec76fa5cbe57d9ad63007230030aafd"} err="failed to get container status \"b0425e9b645bc5dd70ccbfa0f7b053854ec76fa5cbe57d9ad63007230030aafd\": rpc error: code = NotFound desc = could not find container \"b0425e9b645bc5dd70ccbfa0f7b053854ec76fa5cbe57d9ad63007230030aafd\": container with ID starting with b0425e9b645bc5dd70ccbfa0f7b053854ec76fa5cbe57d9ad63007230030aafd not found: ID does not exist"
Sep 18 18:13:22.948154 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:22.948131    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-2c937c/noten-model-2c937c-kserve-54f7c6f799-b8dfl"]
Sep 18 18:13:22.952634 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:22.952616    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-2c937c/noten-model-2c937c-kserve-54f7c6f799-b8dfl"]
Sep 18 18:13:23.134076 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:23.133997    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:13:23.861764 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:23.861734    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426" path="/var/lib/kubelet/pods/8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426/volumes"
Sep 18 18:13:23.898927 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:23.898896    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:13:24.718528 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:24.718501    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:13:25.028073 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:25.027992    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-609aba/premium-w0-609aba-kserve-897b66447-b9bzg"]
Sep 18 18:13:25.130779 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:25.130748    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:13:25.786877 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:25.786845    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:13:28.708064 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:28.708035    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:13:28.993014 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:28.992903    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:13:29.553304 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:29.553275    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:13:29.806533 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:29.806464    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:13:29.930056 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:29.930028    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:13:31.672384 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:31.672342    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:13:31.718092 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:31.718057    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-609aba-58c9bb7b5c-w9mll"]
Sep 18 18:13:31.718366 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:31.718339    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-609aba-58c9bb7b5c-w9mll" podUID="222a170e-28a4-4f09-857e-20d2b245a957" containerName="payload-processing" containerID="cri-o://81d681646c2b39c89774fac7d899c644ddac7a24402d31ed66ade3ab4f3d75a4" gracePeriod=30
Sep 18 18:13:31.819655 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:31.819616    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:13:31.913090 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:31.912962    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:13:31.970163 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:31.970107    2561 generic.go:356] "Generic (PLEG): container finished" podID="222a170e-28a4-4f09-857e-20d2b245a957" containerID="81d681646c2b39c89774fac7d899c644ddac7a24402d31ed66ade3ab4f3d75a4" exitCode=0
Sep 18 18:13:31.970339 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:31.970163    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-609aba-58c9bb7b5c-w9mll" event={"ID":"222a170e-28a4-4f09-857e-20d2b245a957","Type":"ContainerDied","Data":"81d681646c2b39c89774fac7d899c644ddac7a24402d31ed66ade3ab4f3d75a4"}
Sep 18 18:13:31.989021 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:31.989001    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-609aba-58c9bb7b5c-w9mll"
Sep 18 18:13:32.013971 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:32.013930    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:13:32.083882 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:32.083853    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s25gc\" (UniqueName: \"kubernetes.io/projected/222a170e-28a4-4f09-857e-20d2b245a957-kube-api-access-s25gc\") pod \"222a170e-28a4-4f09-857e-20d2b245a957\" (UID: \"222a170e-28a4-4f09-857e-20d2b245a957\") "
Sep 18 18:13:32.084039 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:32.083997    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/222a170e-28a4-4f09-857e-20d2b245a957-plugins-config-volume\") pod \"222a170e-28a4-4f09-857e-20d2b245a957\" (UID: \"222a170e-28a4-4f09-857e-20d2b245a957\") "
Sep 18 18:13:32.084390 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:32.084363    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/222a170e-28a4-4f09-857e-20d2b245a957-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "222a170e-28a4-4f09-857e-20d2b245a957" (UID: "222a170e-28a4-4f09-857e-20d2b245a957"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:13:32.085981 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:32.085958    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/222a170e-28a4-4f09-857e-20d2b245a957-kube-api-access-s25gc" (OuterVolumeSpecName: "kube-api-access-s25gc") pod "222a170e-28a4-4f09-857e-20d2b245a957" (UID: "222a170e-28a4-4f09-857e-20d2b245a957"). InnerVolumeSpecName "kube-api-access-s25gc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:13:32.092577 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:32.092559    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:13:32.161555 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:32.161525    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:13:32.184956 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:32.184929    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/222a170e-28a4-4f09-857e-20d2b245a957-plugins-config-volume\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:13:32.184956 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:32.184952    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s25gc\" (UniqueName: \"kubernetes.io/projected/222a170e-28a4-4f09-857e-20d2b245a957-kube-api-access-s25gc\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:13:32.351791 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:32.351708    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:13:32.546803 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:32.546771    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:13:32.638606 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:32.638527    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:13:32.976536 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:32.976506    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-609aba-58c9bb7b5c-w9mll"
Sep 18 18:13:32.976942 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:32.976543    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-609aba-58c9bb7b5c-w9mll" event={"ID":"222a170e-28a4-4f09-857e-20d2b245a957","Type":"ContainerDied","Data":"9615c7a1475aabfcc3996630c14955897818fd83b4974a82bd30ffdc3551e896"}
Sep 18 18:13:32.976942 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:32.976613    2561 scope.go:117] "RemoveContainer" containerID="81d681646c2b39c89774fac7d899c644ddac7a24402d31ed66ade3ab4f3d75a4"
Sep 18 18:13:33.002044 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:33.002012    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-609aba-58c9bb7b5c-w9mll"]
Sep 18 18:13:33.007743 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:33.007721    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-609aba-58c9bb7b5c-w9mll"]
Sep 18 18:13:33.155444 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:33.155412    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:13:33.863920 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:33.863884    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="222a170e-28a4-4f09-857e-20d2b245a957" path="/var/lib/kubelet/pods/222a170e-28a4-4f09-857e-20d2b245a957/volumes"
Sep 18 18:13:33.941555 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:33.941522    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:13:34.679703 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:34.679671    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:13:35.537208 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:35.537176    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:13:36.212364 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:36.212330    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-53cde3-6b864cc5fd-tvlhr"]
Sep 18 18:13:36.213019 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:36.212998    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="222a170e-28a4-4f09-857e-20d2b245a957" containerName="payload-processing"
Sep 18 18:13:36.213076 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:36.213023    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="222a170e-28a4-4f09-857e-20d2b245a957" containerName="payload-processing"
Sep 18 18:13:36.213076 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:36.213042    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426" containerName="main"
Sep 18 18:13:36.213076 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:36.213051    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426" containerName="main"
Sep 18 18:13:36.213076 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:36.213065    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="843c8a1a-4277-4e30-9fe3-1d726d6f0efb" containerName="maas-api"
Sep 18 18:13:36.213076 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:36.213073    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="843c8a1a-4277-4e30-9fe3-1d726d6f0efb" containerName="maas-api"
Sep 18 18:13:36.213261 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:36.213224    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="222a170e-28a4-4f09-857e-20d2b245a957" containerName="payload-processing"
Sep 18 18:13:36.213261 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:36.213244    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fa06fc6-6eaa-4543-a3a9-c2a14cdb0426" containerName="main"
Sep 18 18:13:36.213261 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:36.213255    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="843c8a1a-4277-4e30-9fe3-1d726d6f0efb" containerName="maas-api"
Sep 18 18:13:36.218037 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:36.218015    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-53cde3-6b864cc5fd-tvlhr"
Sep 18 18:13:36.220287 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:36.220268    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-53cde3-serving-cert\""
Sep 18 18:13:36.228982 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:36.228959    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-53cde3-6b864cc5fd-tvlhr"]
Sep 18 18:13:36.263766 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:36.263736    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:13:36.327963 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:36.327935    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f46e6d38-157e-4272-84f9-bacc3b231773-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-53cde3-6b864cc5fd-tvlhr\" (UID: \"f46e6d38-157e-4272-84f9-bacc3b231773\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-53cde3-6b864cc5fd-tvlhr"
Sep 18 18:13:36.328166 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:36.327998    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f46e6d38-157e-4272-84f9-bacc3b231773-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-53cde3-6b864cc5fd-tvlhr\" (UID: \"f46e6d38-157e-4272-84f9-bacc3b231773\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-53cde3-6b864cc5fd-tvlhr"
Sep 18 18:13:36.328166 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:36.328064    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f46e6d38-157e-4272-84f9-bacc3b231773-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-53cde3-6b864cc5fd-tvlhr\" (UID: \"f46e6d38-157e-4272-84f9-bacc3b231773\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-53cde3-6b864cc5fd-tvlhr"
Sep 18 18:13:36.328166 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:36.328141    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dhkh\" (UniqueName: \"kubernetes.io/projected/f46e6d38-157e-4272-84f9-bacc3b231773-kube-api-access-2dhkh\") pod \"maas-api-e2e-shared-a-w3-53cde3-6b864cc5fd-tvlhr\" (UID: \"f46e6d38-157e-4272-84f9-bacc3b231773\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-53cde3-6b864cc5fd-tvlhr"
Sep 18 18:13:36.363310 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:36.363283    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:13:36.429227 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:36.429185    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f46e6d38-157e-4272-84f9-bacc3b231773-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-53cde3-6b864cc5fd-tvlhr\" (UID: \"f46e6d38-157e-4272-84f9-bacc3b231773\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-53cde3-6b864cc5fd-tvlhr"
Sep 18 18:13:36.429422 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:36.429270    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f46e6d38-157e-4272-84f9-bacc3b231773-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-53cde3-6b864cc5fd-tvlhr\" (UID: \"f46e6d38-157e-4272-84f9-bacc3b231773\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-53cde3-6b864cc5fd-tvlhr"
Sep 18 18:13:36.429422 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:36.429307    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f46e6d38-157e-4272-84f9-bacc3b231773-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-53cde3-6b864cc5fd-tvlhr\" (UID: \"f46e6d38-157e-4272-84f9-bacc3b231773\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-53cde3-6b864cc5fd-tvlhr"
Sep 18 18:13:36.429422 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:36.429340    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dhkh\" (UniqueName: \"kubernetes.io/projected/f46e6d38-157e-4272-84f9-bacc3b231773-kube-api-access-2dhkh\") pod \"maas-api-e2e-shared-a-w3-53cde3-6b864cc5fd-tvlhr\" (UID: \"f46e6d38-157e-4272-84f9-bacc3b231773\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-53cde3-6b864cc5fd-tvlhr"
Sep 18 18:13:36.429422 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:13:36.429406    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-53cde3-serving-cert: secret "maas-api-e2e-shared-a-w3-53cde3-serving-cert" not found
Sep 18 18:13:36.429647 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:13:36.429480    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f46e6d38-157e-4272-84f9-bacc3b231773-maas-api-tls podName:f46e6d38-157e-4272-84f9-bacc3b231773 nodeName:}" failed. No retries permitted until 2026-09-18 18:13:36.929461045 +0000 UTC m=+1421.556592643 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f46e6d38-157e-4272-84f9-bacc3b231773-maas-api-tls") pod "maas-api-e2e-shared-a-w3-53cde3-6b864cc5fd-tvlhr" (UID: "f46e6d38-157e-4272-84f9-bacc3b231773") : secret "maas-api-e2e-shared-a-w3-53cde3-serving-cert" not found
Sep 18 18:13:36.430035 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:36.430013    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f46e6d38-157e-4272-84f9-bacc3b231773-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-53cde3-6b864cc5fd-tvlhr\" (UID: \"f46e6d38-157e-4272-84f9-bacc3b231773\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-53cde3-6b864cc5fd-tvlhr"
Sep 18 18:13:36.431561 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:36.431543    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f46e6d38-157e-4272-84f9-bacc3b231773-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-53cde3-6b864cc5fd-tvlhr\" (UID: \"f46e6d38-157e-4272-84f9-bacc3b231773\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-53cde3-6b864cc5fd-tvlhr"
Sep 18 18:13:36.438536 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:36.438515    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dhkh\" (UniqueName: \"kubernetes.io/projected/f46e6d38-157e-4272-84f9-bacc3b231773-kube-api-access-2dhkh\") pod \"maas-api-e2e-shared-a-w3-53cde3-6b864cc5fd-tvlhr\" (UID: \"f46e6d38-157e-4272-84f9-bacc3b231773\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-53cde3-6b864cc5fd-tvlhr"
Sep 18 18:13:36.596439 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:36.596357    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:13:36.693176 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:36.693112    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:13:36.794723 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:36.794689    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:13:36.879947 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:36.879863    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:13:36.917685 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:36.917651    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-53cde3-7d857f96b9-p9gft"]
Sep 18 18:13:36.920868 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:36.920846    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-53cde3-7d857f96b9-p9gft"
Sep 18 18:13:36.923329 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:36.923305    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-53cde3\""
Sep 18 18:13:36.923472 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:36.923434    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-53cde3-dockercfg-fwnx8\""
Sep 18 18:13:36.933925 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:36.933905    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-53cde3-7d857f96b9-p9gft"]
Sep 18 18:13:36.934934 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:36.934911    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f46e6d38-157e-4272-84f9-bacc3b231773-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-53cde3-6b864cc5fd-tvlhr\" (UID: \"f46e6d38-157e-4272-84f9-bacc3b231773\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-53cde3-6b864cc5fd-tvlhr"
Sep 18 18:13:36.937517 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:36.937497    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f46e6d38-157e-4272-84f9-bacc3b231773-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-53cde3-6b864cc5fd-tvlhr\" (UID: \"f46e6d38-157e-4272-84f9-bacc3b231773\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-53cde3-6b864cc5fd-tvlhr"
Sep 18 18:13:37.035784 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:37.035753    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4r5q\" (UniqueName: \"kubernetes.io/projected/0b5fbe79-9f46-4567-90f9-52f7c4ba11a8-kube-api-access-x4r5q\") pod \"payload-processing-e2e-shared-a-w3-53cde3-7d857f96b9-p9gft\" (UID: \"0b5fbe79-9f46-4567-90f9-52f7c4ba11a8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-53cde3-7d857f96b9-p9gft"
Sep 18 18:13:37.035925 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:37.035888    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b5fbe79-9f46-4567-90f9-52f7c4ba11a8-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-53cde3-7d857f96b9-p9gft\" (UID: \"0b5fbe79-9f46-4567-90f9-52f7c4ba11a8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-53cde3-7d857f96b9-p9gft"
Sep 18 18:13:37.132346 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:37.132264    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-53cde3-6b864cc5fd-tvlhr"
Sep 18 18:13:37.136618 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:37.136589    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4r5q\" (UniqueName: \"kubernetes.io/projected/0b5fbe79-9f46-4567-90f9-52f7c4ba11a8-kube-api-access-x4r5q\") pod \"payload-processing-e2e-shared-a-w3-53cde3-7d857f96b9-p9gft\" (UID: \"0b5fbe79-9f46-4567-90f9-52f7c4ba11a8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-53cde3-7d857f96b9-p9gft"
Sep 18 18:13:37.136811 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:37.136775    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b5fbe79-9f46-4567-90f9-52f7c4ba11a8-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-53cde3-7d857f96b9-p9gft\" (UID: \"0b5fbe79-9f46-4567-90f9-52f7c4ba11a8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-53cde3-7d857f96b9-p9gft"
Sep 18 18:13:37.137405 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:37.137388    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b5fbe79-9f46-4567-90f9-52f7c4ba11a8-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-53cde3-7d857f96b9-p9gft\" (UID: \"0b5fbe79-9f46-4567-90f9-52f7c4ba11a8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-53cde3-7d857f96b9-p9gft"
Sep 18 18:13:37.145987 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:37.145957    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4r5q\" (UniqueName: \"kubernetes.io/projected/0b5fbe79-9f46-4567-90f9-52f7c4ba11a8-kube-api-access-x4r5q\") pod \"payload-processing-e2e-shared-a-w3-53cde3-7d857f96b9-p9gft\" (UID: \"0b5fbe79-9f46-4567-90f9-52f7c4ba11a8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-53cde3-7d857f96b9-p9gft"
Sep 18 18:13:37.233196 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:37.233166    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-53cde3-7d857f96b9-p9gft"
Sep 18 18:13:37.410629 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:37.410600    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-53cde3-6b864cc5fd-tvlhr"]
Sep 18 18:13:37.412368 ip-10-0-140-54 kubenswrapper[2561]: W0918 18:13:37.412341    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf46e6d38_157e_4272_84f9_bacc3b231773.slice/crio-44202567d04cd2575cf4b8665261746720515f98f6011850511ae81d66c29f0d WatchSource:0}: Error finding container 44202567d04cd2575cf4b8665261746720515f98f6011850511ae81d66c29f0d: Status 404 returned error can't find the container with id 44202567d04cd2575cf4b8665261746720515f98f6011850511ae81d66c29f0d
Sep 18 18:13:37.413306 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:37.413282    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:13:37.513277 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:37.513255    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-53cde3-7d857f96b9-p9gft"]
Sep 18 18:13:37.514854 ip-10-0-140-54 kubenswrapper[2561]: W0918 18:13:37.514818    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b5fbe79_9f46_4567_90f9_52f7c4ba11a8.slice/crio-443bdd7446369c43dde6688cf984fb9e8c60544c1a907f80cd2cdbd5765519de WatchSource:0}: Error finding container 443bdd7446369c43dde6688cf984fb9e8c60544c1a907f80cd2cdbd5765519de: Status 404 returned error can't find the container with id 443bdd7446369c43dde6688cf984fb9e8c60544c1a907f80cd2cdbd5765519de
Sep 18 18:13:38.008185 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:38.008152    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-53cde3-6b864cc5fd-tvlhr" event={"ID":"f46e6d38-157e-4272-84f9-bacc3b231773","Type":"ContainerStarted","Data":"44202567d04cd2575cf4b8665261746720515f98f6011850511ae81d66c29f0d"}
Sep 18 18:13:38.010136 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:38.009909    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-53cde3-7d857f96b9-p9gft" event={"ID":"0b5fbe79-9f46-4567-90f9-52f7c4ba11a8","Type":"ContainerStarted","Data":"e0043796f38845e0c0f65502cafe918b69f240fb54b76c02cf4999a2637c544c"}
Sep 18 18:13:38.010136 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:38.009963    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-53cde3-7d857f96b9-p9gft"
Sep 18 18:13:38.010136 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:38.009996    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-53cde3-7d857f96b9-p9gft" event={"ID":"0b5fbe79-9f46-4567-90f9-52f7c4ba11a8","Type":"ContainerStarted","Data":"443bdd7446369c43dde6688cf984fb9e8c60544c1a907f80cd2cdbd5765519de"}
Sep 18 18:13:38.033818 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:38.033764    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-53cde3-7d857f96b9-p9gft" podStartSLOduration=2.033742004 podStartE2EDuration="2.033742004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:13:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:13:38.03121399 +0000 UTC m=+1422.658345609" watchObservedRunningTime="2026-09-18 18:13:38.033742004 +0000 UTC m=+1422.660873626"
Sep 18 18:13:38.233517 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:38.233438    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:13:39.017657 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:39.017620    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-53cde3-6b864cc5fd-tvlhr" event={"ID":"f46e6d38-157e-4272-84f9-bacc3b231773","Type":"ContainerStarted","Data":"f51d1b73cbb85c4b1a8762156286dd841d28fea13d4383c902e5277f93093484"}
Sep 18 18:13:39.017844 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:39.017827    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-53cde3-6b864cc5fd-tvlhr"
Sep 18 18:13:39.038389 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:39.038338    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-53cde3-6b864cc5fd-tvlhr" podStartSLOduration=2.506973727 podStartE2EDuration="3.038323192s" totalImagesPullingTime="531.349465ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:13:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:13:39.036522589 +0000 UTC m=+1423.663654209" watchObservedRunningTime="2026-09-18 18:13:39.038323192 +0000 UTC m=+1423.665454812"
Sep 18 18:13:39.355466 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:39.355379    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:13:39.458970 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:39.458941    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:13:39.646672 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:39.646640    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:13:40.047436 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:40.047345    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-609aba/premium-w0-609aba-kserve-897b66447-b9bzg" podUID="3a870971-8984-4e51-822a-1726f7f1a85d" containerName="main" containerID="cri-o://88907f86a40babfeaff18093b90f5a1cf0f1ca179967d39bf2e58c5e94f58752" gracePeriod=45
Sep 18 18:13:40.296990 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:40.296963    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-609aba/premium-w0-609aba-kserve-897b66447-b9bzg"
Sep 18 18:13:40.371508 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:40.371428    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a870971-8984-4e51-822a-1726f7f1a85d-dshm\") pod \"3a870971-8984-4e51-822a-1726f7f1a85d\" (UID: \"3a870971-8984-4e51-822a-1726f7f1a85d\") "
Sep 18 18:13:40.371508 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:40.371459    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a870971-8984-4e51-822a-1726f7f1a85d-tls-certs\") pod \"3a870971-8984-4e51-822a-1726f7f1a85d\" (UID: \"3a870971-8984-4e51-822a-1726f7f1a85d\") "
Sep 18 18:13:40.371998 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:40.371569    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a870971-8984-4e51-822a-1726f7f1a85d-model-cache\") pod \"3a870971-8984-4e51-822a-1726f7f1a85d\" (UID: \"3a870971-8984-4e51-822a-1726f7f1a85d\") "
Sep 18 18:13:40.371998 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:40.371598    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-chn2x\" (UniqueName: \"kubernetes.io/projected/3a870971-8984-4e51-822a-1726f7f1a85d-kube-api-access-chn2x\") pod \"3a870971-8984-4e51-822a-1726f7f1a85d\" (UID: \"3a870971-8984-4e51-822a-1726f7f1a85d\") "
Sep 18 18:13:40.371998 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:40.371621    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a870971-8984-4e51-822a-1726f7f1a85d-home\") pod \"3a870971-8984-4e51-822a-1726f7f1a85d\" (UID: \"3a870971-8984-4e51-822a-1726f7f1a85d\") "
Sep 18 18:13:40.371998 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:40.371677    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a870971-8984-4e51-822a-1726f7f1a85d-tmp-dir\") pod \"3a870971-8984-4e51-822a-1726f7f1a85d\" (UID: \"3a870971-8984-4e51-822a-1726f7f1a85d\") "
Sep 18 18:13:40.371998 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:40.371841    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a870971-8984-4e51-822a-1726f7f1a85d-model-cache" (OuterVolumeSpecName: "model-cache") pod "3a870971-8984-4e51-822a-1726f7f1a85d" (UID: "3a870971-8984-4e51-822a-1726f7f1a85d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:13:40.371998 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:40.371969    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a870971-8984-4e51-822a-1726f7f1a85d-model-cache\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:13:40.371998 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:40.371965    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a870971-8984-4e51-822a-1726f7f1a85d-home" (OuterVolumeSpecName: "home") pod "3a870971-8984-4e51-822a-1726f7f1a85d" (UID: "3a870971-8984-4e51-822a-1726f7f1a85d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:13:40.372400 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:40.372056    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a870971-8984-4e51-822a-1726f7f1a85d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3a870971-8984-4e51-822a-1726f7f1a85d" (UID: "3a870971-8984-4e51-822a-1726f7f1a85d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:13:40.373804 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:40.373777    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a870971-8984-4e51-822a-1726f7f1a85d-dshm" (OuterVolumeSpecName: "dshm") pod "3a870971-8984-4e51-822a-1726f7f1a85d" (UID: "3a870971-8984-4e51-822a-1726f7f1a85d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:13:40.374008 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:40.373985    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a870971-8984-4e51-822a-1726f7f1a85d-kube-api-access-chn2x" (OuterVolumeSpecName: "kube-api-access-chn2x") pod "3a870971-8984-4e51-822a-1726f7f1a85d" (UID: "3a870971-8984-4e51-822a-1726f7f1a85d"). InnerVolumeSpecName "kube-api-access-chn2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:13:40.374110 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:40.374007    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a870971-8984-4e51-822a-1726f7f1a85d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3a870971-8984-4e51-822a-1726f7f1a85d" (UID: "3a870971-8984-4e51-822a-1726f7f1a85d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:13:40.472853 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:40.472820    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a870971-8984-4e51-822a-1726f7f1a85d-tmp-dir\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:13:40.472853 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:40.472848    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a870971-8984-4e51-822a-1726f7f1a85d-dshm\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:13:40.472853 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:40.472857    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a870971-8984-4e51-822a-1726f7f1a85d-tls-certs\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:13:40.473061 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:40.472866    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chn2x\" (UniqueName: \"kubernetes.io/projected/3a870971-8984-4e51-822a-1726f7f1a85d-kube-api-access-chn2x\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:13:40.473061 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:40.472877    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a870971-8984-4e51-822a-1726f7f1a85d-home\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:13:41.028906 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:41.028870    2561 generic.go:356] "Generic (PLEG): container finished" podID="3a870971-8984-4e51-822a-1726f7f1a85d" containerID="88907f86a40babfeaff18093b90f5a1cf0f1ca179967d39bf2e58c5e94f58752" exitCode=0
Sep 18 18:13:41.029054 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:41.028916    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-609aba/premium-w0-609aba-kserve-897b66447-b9bzg"
Sep 18 18:13:41.029054 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:41.028949    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-609aba/premium-w0-609aba-kserve-897b66447-b9bzg" event={"ID":"3a870971-8984-4e51-822a-1726f7f1a85d","Type":"ContainerDied","Data":"88907f86a40babfeaff18093b90f5a1cf0f1ca179967d39bf2e58c5e94f58752"}
Sep 18 18:13:41.029054 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:41.028993    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-609aba/premium-w0-609aba-kserve-897b66447-b9bzg" event={"ID":"3a870971-8984-4e51-822a-1726f7f1a85d","Type":"ContainerDied","Data":"e2d0792925252c176f4b143bba28b6c0c64339eebdcf8b5e3e01547a70392413"}
Sep 18 18:13:41.029054 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:41.029012    2561 scope.go:117] "RemoveContainer" containerID="88907f86a40babfeaff18093b90f5a1cf0f1ca179967d39bf2e58c5e94f58752"
Sep 18 18:13:41.038176 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:41.038159    2561 scope.go:117] "RemoveContainer" containerID="88907f86a40babfeaff18093b90f5a1cf0f1ca179967d39bf2e58c5e94f58752"
Sep 18 18:13:41.038422 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:13:41.038405    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88907f86a40babfeaff18093b90f5a1cf0f1ca179967d39bf2e58c5e94f58752\": container with ID starting with 88907f86a40babfeaff18093b90f5a1cf0f1ca179967d39bf2e58c5e94f58752 not found: ID does not exist" containerID="88907f86a40babfeaff18093b90f5a1cf0f1ca179967d39bf2e58c5e94f58752"
Sep 18 18:13:41.038494 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:41.038427    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88907f86a40babfeaff18093b90f5a1cf0f1ca179967d39bf2e58c5e94f58752"} err="failed to get container status \"88907f86a40babfeaff18093b90f5a1cf0f1ca179967d39bf2e58c5e94f58752\": rpc error: code = NotFound desc = could not find container \"88907f86a40babfeaff18093b90f5a1cf0f1ca179967d39bf2e58c5e94f58752\": container with ID starting with 88907f86a40babfeaff18093b90f5a1cf0f1ca179967d39bf2e58c5e94f58752 not found: ID does not exist"
Sep 18 18:13:41.052472 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:41.052448    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-609aba/premium-w0-609aba-kserve-897b66447-b9bzg"]
Sep 18 18:13:41.059347 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:41.059327    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-609aba/premium-w0-609aba-kserve-897b66447-b9bzg"]
Sep 18 18:13:41.861856 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:41.861815    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a870971-8984-4e51-822a-1726f7f1a85d" path="/var/lib/kubelet/pods/3a870971-8984-4e51-822a-1726f7f1a85d/volumes"
Sep 18 18:13:44.270067 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:44.270034    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:13:45.029048 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:45.029007    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-53cde3-6b864cc5fd-tvlhr"
Sep 18 18:13:49.019908 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:49.019836    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-53cde3-7d857f96b9-p9gft"
Sep 18 18:13:52.126728 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:52.126695    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:13:52.226069 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:52.226039    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:13:58.424427 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:58.424389    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:13:59.044355 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:59.044326    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:13:59.837376 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:13:59.837334    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:00.486391 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:00.486352    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:01.306825 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:01.306786    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:01.687339 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:01.687312    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:01.787297 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:01.787268    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:02.145957 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:02.145932    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:02.581550 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:02.581469    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:02.715631 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:02.715598    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:02.923628 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:02.923593    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:03.244367 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:03.244294    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:03.395286 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:03.395252    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:06.056684 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:06.056649    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:06.683154 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:06.683101    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:06.951084 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:06.951005    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:07.102315 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:07.102276    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:07.584467 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:07.584430    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:08.443524 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:08.443484    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:09.186546 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:09.186505    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:09.936062 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:09.936034    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:10.749005 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:10.748974    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:10.850662 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:10.850631    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:10.954573 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:10.954539    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:11.198218 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:11.198193    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:11.300834 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:11.300806    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:32.621580 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:32.621546    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:33.252271 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:33.252232    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:33.920384 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:33.920348    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:34.609810 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:34.609781    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:35.238181 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:35.238147    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:36.031356 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:36.031322    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:36.662681 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:36.662651    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:36.834104 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:36.834071    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:43.404848 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:43.404814    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:43.461516 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:43.461492    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:43.604688 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:43.604661    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:45.633866 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:45.633831    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:46.431397 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:46.431365    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:47.078378 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:47.078343    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:47.692389 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:47.692352    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:48.321556 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:48.321522    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:48.934712 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:48.934682    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:49.721744 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:49.721714    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:50.366641 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:50.366613    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:51.017903 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:51.017871    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-53cde3/auth-test-model-53cde3-kserve-75dcb68b95-d4j87"]
Sep 18 18:14:51.018349 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:51.018335    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a870971-8984-4e51-822a-1726f7f1a85d" containerName="main"
Sep 18 18:14:51.018399 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:51.018350    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a870971-8984-4e51-822a-1726f7f1a85d" containerName="main"
Sep 18 18:14:51.018451 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:51.018441    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a870971-8984-4e51-822a-1726f7f1a85d" containerName="main"
Sep 18 18:14:51.021826 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:51.021807    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-53cde3/auth-test-model-53cde3-kserve-75dcb68b95-d4j87"
Sep 18 18:14:51.024875 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:51.024853    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-53cde3\"/\"auth-test-model-53cde3-kserve-self-signed-certs\""
Sep 18 18:14:51.024991 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:51.024948    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-53cde3\"/\"openshift-service-ca.crt\""
Sep 18 18:14:51.024991 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:51.024965    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-53cde3\"/\"default-dockercfg-rks92\""
Sep 18 18:14:51.024991 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:51.024987    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-53cde3\"/\"kube-root-ca.crt\""
Sep 18 18:14:51.031325 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:51.031306    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-53cde3/auth-test-model-53cde3-kserve-75dcb68b95-d4j87"]
Sep 18 18:14:51.042235 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:51.042212    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:51.149195 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:51.149163    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/536e0d16-7967-4753-9123-f335ac1a763f-home\") pod \"auth-test-model-53cde3-kserve-75dcb68b95-d4j87\" (UID: \"536e0d16-7967-4753-9123-f335ac1a763f\") " pod="ai-tenant-e2e-shared-a-w3-53cde3/auth-test-model-53cde3-kserve-75dcb68b95-d4j87"
Sep 18 18:14:51.149379 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:51.149199    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/536e0d16-7967-4753-9123-f335ac1a763f-tmp-dir\") pod \"auth-test-model-53cde3-kserve-75dcb68b95-d4j87\" (UID: \"536e0d16-7967-4753-9123-f335ac1a763f\") " pod="ai-tenant-e2e-shared-a-w3-53cde3/auth-test-model-53cde3-kserve-75dcb68b95-d4j87"
Sep 18 18:14:51.149379 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:51.149233    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/536e0d16-7967-4753-9123-f335ac1a763f-tls-certs\") pod \"auth-test-model-53cde3-kserve-75dcb68b95-d4j87\" (UID: \"536e0d16-7967-4753-9123-f335ac1a763f\") " pod="ai-tenant-e2e-shared-a-w3-53cde3/auth-test-model-53cde3-kserve-75dcb68b95-d4j87"
Sep 18 18:14:51.149379 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:51.149254    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/536e0d16-7967-4753-9123-f335ac1a763f-dshm\") pod \"auth-test-model-53cde3-kserve-75dcb68b95-d4j87\" (UID: \"536e0d16-7967-4753-9123-f335ac1a763f\") " pod="ai-tenant-e2e-shared-a-w3-53cde3/auth-test-model-53cde3-kserve-75dcb68b95-d4j87"
Sep 18 18:14:51.149379 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:51.149341    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6x4k\" (UniqueName: \"kubernetes.io/projected/536e0d16-7967-4753-9123-f335ac1a763f-kube-api-access-b6x4k\") pod \"auth-test-model-53cde3-kserve-75dcb68b95-d4j87\" (UID: \"536e0d16-7967-4753-9123-f335ac1a763f\") " pod="ai-tenant-e2e-shared-a-w3-53cde3/auth-test-model-53cde3-kserve-75dcb68b95-d4j87"
Sep 18 18:14:51.149604 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:51.149390    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/536e0d16-7967-4753-9123-f335ac1a763f-model-cache\") pod \"auth-test-model-53cde3-kserve-75dcb68b95-d4j87\" (UID: \"536e0d16-7967-4753-9123-f335ac1a763f\") " pod="ai-tenant-e2e-shared-a-w3-53cde3/auth-test-model-53cde3-kserve-75dcb68b95-d4j87"
Sep 18 18:14:51.250273 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:51.250239    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/536e0d16-7967-4753-9123-f335ac1a763f-tmp-dir\") pod \"auth-test-model-53cde3-kserve-75dcb68b95-d4j87\" (UID: \"536e0d16-7967-4753-9123-f335ac1a763f\") " pod="ai-tenant-e2e-shared-a-w3-53cde3/auth-test-model-53cde3-kserve-75dcb68b95-d4j87"
Sep 18 18:14:51.250455 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:51.250290    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/536e0d16-7967-4753-9123-f335ac1a763f-tls-certs\") pod \"auth-test-model-53cde3-kserve-75dcb68b95-d4j87\" (UID: \"536e0d16-7967-4753-9123-f335ac1a763f\") " pod="ai-tenant-e2e-shared-a-w3-53cde3/auth-test-model-53cde3-kserve-75dcb68b95-d4j87"
Sep 18 18:14:51.250455 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:51.250314    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/536e0d16-7967-4753-9123-f335ac1a763f-dshm\") pod \"auth-test-model-53cde3-kserve-75dcb68b95-d4j87\" (UID: \"536e0d16-7967-4753-9123-f335ac1a763f\") " pod="ai-tenant-e2e-shared-a-w3-53cde3/auth-test-model-53cde3-kserve-75dcb68b95-d4j87"
Sep 18 18:14:51.250455 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:51.250388    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6x4k\" (UniqueName: \"kubernetes.io/projected/536e0d16-7967-4753-9123-f335ac1a763f-kube-api-access-b6x4k\") pod \"auth-test-model-53cde3-kserve-75dcb68b95-d4j87\" (UID: \"536e0d16-7967-4753-9123-f335ac1a763f\") " pod="ai-tenant-e2e-shared-a-w3-53cde3/auth-test-model-53cde3-kserve-75dcb68b95-d4j87"
Sep 18 18:14:51.250455 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:51.250423    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/536e0d16-7967-4753-9123-f335ac1a763f-model-cache\") pod \"auth-test-model-53cde3-kserve-75dcb68b95-d4j87\" (UID: \"536e0d16-7967-4753-9123-f335ac1a763f\") " pod="ai-tenant-e2e-shared-a-w3-53cde3/auth-test-model-53cde3-kserve-75dcb68b95-d4j87"
Sep 18 18:14:51.250672 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:51.250503    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/536e0d16-7967-4753-9123-f335ac1a763f-home\") pod \"auth-test-model-53cde3-kserve-75dcb68b95-d4j87\" (UID: \"536e0d16-7967-4753-9123-f335ac1a763f\") " pod="ai-tenant-e2e-shared-a-w3-53cde3/auth-test-model-53cde3-kserve-75dcb68b95-d4j87"
Sep 18 18:14:51.250733 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:51.250693    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/536e0d16-7967-4753-9123-f335ac1a763f-tmp-dir\") pod \"auth-test-model-53cde3-kserve-75dcb68b95-d4j87\" (UID: \"536e0d16-7967-4753-9123-f335ac1a763f\") " pod="ai-tenant-e2e-shared-a-w3-53cde3/auth-test-model-53cde3-kserve-75dcb68b95-d4j87"
Sep 18 18:14:51.250828 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:51.250809    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/536e0d16-7967-4753-9123-f335ac1a763f-model-cache\") pod \"auth-test-model-53cde3-kserve-75dcb68b95-d4j87\" (UID: \"536e0d16-7967-4753-9123-f335ac1a763f\") " pod="ai-tenant-e2e-shared-a-w3-53cde3/auth-test-model-53cde3-kserve-75dcb68b95-d4j87"
Sep 18 18:14:51.250889 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:51.250854    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/536e0d16-7967-4753-9123-f335ac1a763f-home\") pod \"auth-test-model-53cde3-kserve-75dcb68b95-d4j87\" (UID: \"536e0d16-7967-4753-9123-f335ac1a763f\") " pod="ai-tenant-e2e-shared-a-w3-53cde3/auth-test-model-53cde3-kserve-75dcb68b95-d4j87"
Sep 18 18:14:51.252641 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:51.252621    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/536e0d16-7967-4753-9123-f335ac1a763f-dshm\") pod \"auth-test-model-53cde3-kserve-75dcb68b95-d4j87\" (UID: \"536e0d16-7967-4753-9123-f335ac1a763f\") " pod="ai-tenant-e2e-shared-a-w3-53cde3/auth-test-model-53cde3-kserve-75dcb68b95-d4j87"
Sep 18 18:14:51.252888 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:51.252873    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/536e0d16-7967-4753-9123-f335ac1a763f-tls-certs\") pod \"auth-test-model-53cde3-kserve-75dcb68b95-d4j87\" (UID: \"536e0d16-7967-4753-9123-f335ac1a763f\") " pod="ai-tenant-e2e-shared-a-w3-53cde3/auth-test-model-53cde3-kserve-75dcb68b95-d4j87"
Sep 18 18:14:51.258809 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:51.258792    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6x4k\" (UniqueName: \"kubernetes.io/projected/536e0d16-7967-4753-9123-f335ac1a763f-kube-api-access-b6x4k\") pod \"auth-test-model-53cde3-kserve-75dcb68b95-d4j87\" (UID: \"536e0d16-7967-4753-9123-f335ac1a763f\") " pod="ai-tenant-e2e-shared-a-w3-53cde3/auth-test-model-53cde3-kserve-75dcb68b95-d4j87"
Sep 18 18:14:51.332941 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:51.332854    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-53cde3/auth-test-model-53cde3-kserve-75dcb68b95-d4j87"
Sep 18 18:14:51.605193 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:51.605167    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-53cde3/auth-test-model-53cde3-kserve-75dcb68b95-d4j87"]
Sep 18 18:14:51.606891 ip-10-0-140-54 kubenswrapper[2561]: W0918 18:14:51.606853    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod536e0d16_7967_4753_9123_f335ac1a763f.slice/crio-12ed042b70573d9110e54fd80311859b1e3738907eb2bf2a8303de43c7804c89 WatchSource:0}: Error finding container 12ed042b70573d9110e54fd80311859b1e3738907eb2bf2a8303de43c7804c89: Status 404 returned error can't find the container with id 12ed042b70573d9110e54fd80311859b1e3738907eb2bf2a8303de43c7804c89
Sep 18 18:14:52.410256 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:52.410222    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-53cde3/auth-test-model-53cde3-kserve-75dcb68b95-d4j87" event={"ID":"536e0d16-7967-4753-9123-f335ac1a763f","Type":"ContainerStarted","Data":"b2f1ceebf586d1ed6a1f20f1d98bb4d930cb95dc5900c516e77993bd1f0832da"}
Sep 18 18:14:52.410704 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:52.410267    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-53cde3/auth-test-model-53cde3-kserve-75dcb68b95-d4j87" event={"ID":"536e0d16-7967-4753-9123-f335ac1a763f","Type":"ContainerStarted","Data":"12ed042b70573d9110e54fd80311859b1e3738907eb2bf2a8303de43c7804c89"}
Sep 18 18:14:52.427645 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:52.427582    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-53cde3/auth-test-model-53cde3-kserve-75dcb68b95-d4j87" podStartSLOduration=1.427567307 podStartE2EDuration="1.427567307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:14:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:14:52.426077706 +0000 UTC m=+1497.053209327" watchObservedRunningTime="2026-09-18 18:14:52.427567307 +0000 UTC m=+1497.054698927"
Sep 18 18:14:55.681558 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:55.681521    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f7e72d9f-54b7648c9b-z87kp"]
Sep 18 18:14:55.682025 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:55.681758    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7e72d9f-54b7648c9b-z87kp" podUID="94100fcf-733a-4b2e-b17d-25a7d14be7a3" containerName="maas-api" containerID="cri-o://76b3102bb53f95c2426e906b8740951d246e02a593b89cbf2b754d9a0131230b" gracePeriod=30
Sep 18 18:14:55.711099 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:55.711069    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:55.737470 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:55.737439    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f7e72d9f-86676c8c8f-ptnrl"]
Sep 18 18:14:55.737922 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:55.737660    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-f7e72d9f-86676c8c8f-ptnrl" podUID="19a9dc6f-6b51-4639-bfb5-4b4122a552ee" containerName="payload-processing" containerID="cri-o://39ef5b47d9e86b9e7e648d8805c6a75b1b2b9d3a88609d1c7e1cf22aabe692e9" gracePeriod=30
Sep 18 18:14:55.746921 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:55.746894    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f7e72d9f-54b9945bdb-zwjgm"]
Sep 18 18:14:55.748361 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:55.747657    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-f7e72d9f-54b9945bdb-zwjgm" podUID="57c13b34-8239-4165-b2c4-8e512de7b5b3" containerName="payload-pre-processing" containerID="cri-o://5c08b7c0eb390c5c7361917f0779c76b3683c77efa0d93d9b692829a5c57be93" gracePeriod=30
Sep 18 18:14:55.779301 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:55.779255    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:55.835676 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:55.835649    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:55.888778 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:55.887360    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-5kjlz_75f2a1f7-95a7-462e-9ea7-23c51dd29599/console-operator/1.log"
Sep 18 18:14:55.897784 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:55.897234    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-5kjlz_75f2a1f7-95a7-462e-9ea7-23c51dd29599/console-operator/1.log"
Sep 18 18:14:55.900313 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:55.899955    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:55.908811 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:55.908777    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkcgk_c5d026b7-af6b-4899-85f4-6554c0f7cd52/ovn-acl-logging/0.log"
Sep 18 18:14:55.912015 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:55.911890    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkcgk_c5d026b7-af6b-4899-85f4-6554c0f7cd52/ovn-acl-logging/0.log"
Sep 18 18:14:56.020828 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:56.020805    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7e72d9f-54b7648c9b-z87kp"
Sep 18 18:14:56.031945 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:56.031910    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:56.052818 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:56.052795    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f7e72d9f-86676c8c8f-ptnrl"
Sep 18 18:14:56.067096 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:56.067077    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f7e72d9f-54b9945bdb-zwjgm"
Sep 18 18:14:56.096450 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:56.095974    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94100fcf-733a-4b2e-b17d-25a7d14be7a3-metrics-certs\") pod \"94100fcf-733a-4b2e-b17d-25a7d14be7a3\" (UID: \"94100fcf-733a-4b2e-b17d-25a7d14be7a3\") "
Sep 18 18:14:56.096450 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:56.096026    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94100fcf-733a-4b2e-b17d-25a7d14be7a3-maas-api-tls\") pod \"94100fcf-733a-4b2e-b17d-25a7d14be7a3\" (UID: \"94100fcf-733a-4b2e-b17d-25a7d14be7a3\") "
Sep 18 18:14:56.096450 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:56.096093    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8krlt\" (UniqueName: \"kubernetes.io/projected/94100fcf-733a-4b2e-b17d-25a7d14be7a3-kube-api-access-8krlt\") pod \"94100fcf-733a-4b2e-b17d-25a7d14be7a3\" (UID: \"94100fcf-733a-4b2e-b17d-25a7d14be7a3\") "
Sep 18 18:14:56.096450 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:56.096270    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94100fcf-733a-4b2e-b17d-25a7d14be7a3-openshift-service-ca\") pod \"94100fcf-733a-4b2e-b17d-25a7d14be7a3\" (UID: \"94100fcf-733a-4b2e-b17d-25a7d14be7a3\") "
Sep 18 18:14:56.097636 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:56.097601    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94100fcf-733a-4b2e-b17d-25a7d14be7a3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "94100fcf-733a-4b2e-b17d-25a7d14be7a3" (UID: "94100fcf-733a-4b2e-b17d-25a7d14be7a3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:14:56.099886 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:56.099859    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94100fcf-733a-4b2e-b17d-25a7d14be7a3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "94100fcf-733a-4b2e-b17d-25a7d14be7a3" (UID: "94100fcf-733a-4b2e-b17d-25a7d14be7a3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:14:56.100155 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:56.100128    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94100fcf-733a-4b2e-b17d-25a7d14be7a3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "94100fcf-733a-4b2e-b17d-25a7d14be7a3" (UID: "94100fcf-733a-4b2e-b17d-25a7d14be7a3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:14:56.100248 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:56.100234    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94100fcf-733a-4b2e-b17d-25a7d14be7a3-kube-api-access-8krlt" (OuterVolumeSpecName: "kube-api-access-8krlt") pod "94100fcf-733a-4b2e-b17d-25a7d14be7a3" (UID: "94100fcf-733a-4b2e-b17d-25a7d14be7a3"). InnerVolumeSpecName "kube-api-access-8krlt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:14:56.197111 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:56.197077    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-84ltk\" (UniqueName: \"kubernetes.io/projected/19a9dc6f-6b51-4639-bfb5-4b4122a552ee-kube-api-access-84ltk\") pod \"19a9dc6f-6b51-4639-bfb5-4b4122a552ee\" (UID: \"19a9dc6f-6b51-4639-bfb5-4b4122a552ee\") "
Sep 18 18:14:56.197298 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:56.197172    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19a9dc6f-6b51-4639-bfb5-4b4122a552ee-plugins-config-volume\") pod \"19a9dc6f-6b51-4639-bfb5-4b4122a552ee\" (UID: \"19a9dc6f-6b51-4639-bfb5-4b4122a552ee\") "
Sep 18 18:14:56.197298 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:56.197204    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57c13b34-8239-4165-b2c4-8e512de7b5b3-plugins-config-volume\") pod \"57c13b34-8239-4165-b2c4-8e512de7b5b3\" (UID: \"57c13b34-8239-4165-b2c4-8e512de7b5b3\") "
Sep 18 18:14:56.197298 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:56.197250    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-58mdj\" (UniqueName: \"kubernetes.io/projected/57c13b34-8239-4165-b2c4-8e512de7b5b3-kube-api-access-58mdj\") pod \"57c13b34-8239-4165-b2c4-8e512de7b5b3\" (UID: \"57c13b34-8239-4165-b2c4-8e512de7b5b3\") "
Sep 18 18:14:56.197560 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:56.197531    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19a9dc6f-6b51-4639-bfb5-4b4122a552ee-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "19a9dc6f-6b51-4639-bfb5-4b4122a552ee" (UID: "19a9dc6f-6b51-4639-bfb5-4b4122a552ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:14:56.197670 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:56.197544    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94100fcf-733a-4b2e-b17d-25a7d14be7a3-metrics-certs\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:14:56.197670 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:56.197596    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94100fcf-733a-4b2e-b17d-25a7d14be7a3-maas-api-tls\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:14:56.197670 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:56.197606    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8krlt\" (UniqueName: \"kubernetes.io/projected/94100fcf-733a-4b2e-b17d-25a7d14be7a3-kube-api-access-8krlt\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:14:56.197670 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:56.197619    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94100fcf-733a-4b2e-b17d-25a7d14be7a3-openshift-service-ca\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:14:56.197670 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:56.197621    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57c13b34-8239-4165-b2c4-8e512de7b5b3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "57c13b34-8239-4165-b2c4-8e512de7b5b3" (UID: "57c13b34-8239-4165-b2c4-8e512de7b5b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:14:56.199517 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:56.199493    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19a9dc6f-6b51-4639-bfb5-4b4122a552ee-kube-api-access-84ltk" (OuterVolumeSpecName: "kube-api-access-84ltk") pod "19a9dc6f-6b51-4639-bfb5-4b4122a552ee" (UID: "19a9dc6f-6b51-4639-bfb5-4b4122a552ee"). InnerVolumeSpecName "kube-api-access-84ltk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:14:56.199813 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:56.199789    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57c13b34-8239-4165-b2c4-8e512de7b5b3-kube-api-access-58mdj" (OuterVolumeSpecName: "kube-api-access-58mdj") pod "57c13b34-8239-4165-b2c4-8e512de7b5b3" (UID: "57c13b34-8239-4165-b2c4-8e512de7b5b3"). InnerVolumeSpecName "kube-api-access-58mdj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:14:56.266373 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:56.266290    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:56.299061 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:56.299034    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19a9dc6f-6b51-4639-bfb5-4b4122a552ee-plugins-config-volume\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:14:56.299061 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:56.299057    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57c13b34-8239-4165-b2c4-8e512de7b5b3-plugins-config-volume\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:14:56.299061 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:56.299067    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58mdj\" (UniqueName: \"kubernetes.io/projected/57c13b34-8239-4165-b2c4-8e512de7b5b3-kube-api-access-58mdj\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:14:56.299299 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:56.299075    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84ltk\" (UniqueName: \"kubernetes.io/projected/19a9dc6f-6b51-4639-bfb5-4b4122a552ee-kube-api-access-84ltk\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:14:56.366664 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:56.366633    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:56.431082 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:56.431052    2561 generic.go:356] "Generic (PLEG): container finished" podID="19a9dc6f-6b51-4639-bfb5-4b4122a552ee" containerID="39ef5b47d9e86b9e7e648d8805c6a75b1b2b9d3a88609d1c7e1cf22aabe692e9" exitCode=0
Sep 18 18:14:56.431288 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:56.431097    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f7e72d9f-86676c8c8f-ptnrl"
Sep 18 18:14:56.431288 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:56.431144    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f7e72d9f-86676c8c8f-ptnrl" event={"ID":"19a9dc6f-6b51-4639-bfb5-4b4122a552ee","Type":"ContainerDied","Data":"39ef5b47d9e86b9e7e648d8805c6a75b1b2b9d3a88609d1c7e1cf22aabe692e9"}
Sep 18 18:14:56.431288 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:56.431203    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f7e72d9f-86676c8c8f-ptnrl" event={"ID":"19a9dc6f-6b51-4639-bfb5-4b4122a552ee","Type":"ContainerDied","Data":"2d67a7e4e051d28689ebaa828a4204accf6dc15c711366d782a4f7621416653c"}
Sep 18 18:14:56.431288 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:56.431234    2561 scope.go:117] "RemoveContainer" containerID="39ef5b47d9e86b9e7e648d8805c6a75b1b2b9d3a88609d1c7e1cf22aabe692e9"
Sep 18 18:14:56.432720 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:56.432697    2561 generic.go:356] "Generic (PLEG): container finished" podID="57c13b34-8239-4165-b2c4-8e512de7b5b3" containerID="5c08b7c0eb390c5c7361917f0779c76b3683c77efa0d93d9b692829a5c57be93" exitCode=0
Sep 18 18:14:56.432854 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:56.432731    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f7e72d9f-54b9945bdb-zwjgm"
Sep 18 18:14:56.432854 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:56.432783    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f7e72d9f-54b9945bdb-zwjgm" event={"ID":"57c13b34-8239-4165-b2c4-8e512de7b5b3","Type":"ContainerDied","Data":"5c08b7c0eb390c5c7361917f0779c76b3683c77efa0d93d9b692829a5c57be93"}
Sep 18 18:14:56.432854 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:56.432833    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f7e72d9f-54b9945bdb-zwjgm" event={"ID":"57c13b34-8239-4165-b2c4-8e512de7b5b3","Type":"ContainerDied","Data":"86dc308301765e953d75e6a7ff822359c9e9ce9c318b3e2a3c17cbff5b8bb7b2"}
Sep 18 18:14:56.434365 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:56.434338    2561 generic.go:356] "Generic (PLEG): container finished" podID="94100fcf-733a-4b2e-b17d-25a7d14be7a3" containerID="76b3102bb53f95c2426e906b8740951d246e02a593b89cbf2b754d9a0131230b" exitCode=0
Sep 18 18:14:56.434507 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:56.434377    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7e72d9f-54b7648c9b-z87kp"
Sep 18 18:14:56.434507 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:56.434436    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7e72d9f-54b7648c9b-z87kp" event={"ID":"94100fcf-733a-4b2e-b17d-25a7d14be7a3","Type":"ContainerDied","Data":"76b3102bb53f95c2426e906b8740951d246e02a593b89cbf2b754d9a0131230b"}
Sep 18 18:14:56.434507 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:56.434488    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7e72d9f-54b7648c9b-z87kp" event={"ID":"94100fcf-733a-4b2e-b17d-25a7d14be7a3","Type":"ContainerDied","Data":"ec4f90301e9bed5de16b7bb7bb6eb5510ed8102961b887957495963463697e41"}
Sep 18 18:14:56.442806 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:56.442784    2561 scope.go:117] "RemoveContainer" containerID="39ef5b47d9e86b9e7e648d8805c6a75b1b2b9d3a88609d1c7e1cf22aabe692e9"
Sep 18 18:14:56.443101 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:14:56.443081    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39ef5b47d9e86b9e7e648d8805c6a75b1b2b9d3a88609d1c7e1cf22aabe692e9\": container with ID starting with 39ef5b47d9e86b9e7e648d8805c6a75b1b2b9d3a88609d1c7e1cf22aabe692e9 not found: ID does not exist" containerID="39ef5b47d9e86b9e7e648d8805c6a75b1b2b9d3a88609d1c7e1cf22aabe692e9"
Sep 18 18:14:56.443200 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:56.443128    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39ef5b47d9e86b9e7e648d8805c6a75b1b2b9d3a88609d1c7e1cf22aabe692e9"} err="failed to get container status \"39ef5b47d9e86b9e7e648d8805c6a75b1b2b9d3a88609d1c7e1cf22aabe692e9\": rpc error: code = NotFound desc = could not find container \"39ef5b47d9e86b9e7e648d8805c6a75b1b2b9d3a88609d1c7e1cf22aabe692e9\": container with ID starting with 39ef5b47d9e86b9e7e648d8805c6a75b1b2b9d3a88609d1c7e1cf22aabe692e9 not found: ID does not exist"
Sep 18 18:14:56.443200 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:56.443151    2561 scope.go:117] "RemoveContainer" containerID="5c08b7c0eb390c5c7361917f0779c76b3683c77efa0d93d9b692829a5c57be93"
Sep 18 18:14:56.451920 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:56.451901    2561 scope.go:117] "RemoveContainer" containerID="5c08b7c0eb390c5c7361917f0779c76b3683c77efa0d93d9b692829a5c57be93"
Sep 18 18:14:56.452200 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:14:56.452182    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c08b7c0eb390c5c7361917f0779c76b3683c77efa0d93d9b692829a5c57be93\": container with ID starting with 5c08b7c0eb390c5c7361917f0779c76b3683c77efa0d93d9b692829a5c57be93 not found: ID does not exist" containerID="5c08b7c0eb390c5c7361917f0779c76b3683c77efa0d93d9b692829a5c57be93"
Sep 18 18:14:56.452284 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:56.452209    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c08b7c0eb390c5c7361917f0779c76b3683c77efa0d93d9b692829a5c57be93"} err="failed to get container status \"5c08b7c0eb390c5c7361917f0779c76b3683c77efa0d93d9b692829a5c57be93\": rpc error: code = NotFound desc = could not find container \"5c08b7c0eb390c5c7361917f0779c76b3683c77efa0d93d9b692829a5c57be93\": container with ID starting with 5c08b7c0eb390c5c7361917f0779c76b3683c77efa0d93d9b692829a5c57be93 not found: ID does not exist"
Sep 18 18:14:56.452284 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:56.452231    2561 scope.go:117] "RemoveContainer" containerID="76b3102bb53f95c2426e906b8740951d246e02a593b89cbf2b754d9a0131230b"
Sep 18 18:14:56.457330 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:56.457306    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f7e72d9f-86676c8c8f-ptnrl"]
Sep 18 18:14:56.461063 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:56.461040    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f7e72d9f-86676c8c8f-ptnrl"]
Sep 18 18:14:56.462022 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:56.462006    2561 scope.go:117] "RemoveContainer" containerID="76b3102bb53f95c2426e906b8740951d246e02a593b89cbf2b754d9a0131230b"
Sep 18 18:14:56.462290 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:14:56.462274    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76b3102bb53f95c2426e906b8740951d246e02a593b89cbf2b754d9a0131230b\": container with ID starting with 76b3102bb53f95c2426e906b8740951d246e02a593b89cbf2b754d9a0131230b not found: ID does not exist" containerID="76b3102bb53f95c2426e906b8740951d246e02a593b89cbf2b754d9a0131230b"
Sep 18 18:14:56.462343 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:56.462295    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76b3102bb53f95c2426e906b8740951d246e02a593b89cbf2b754d9a0131230b"} err="failed to get container status \"76b3102bb53f95c2426e906b8740951d246e02a593b89cbf2b754d9a0131230b\": rpc error: code = NotFound desc = could not find container \"76b3102bb53f95c2426e906b8740951d246e02a593b89cbf2b754d9a0131230b\": container with ID starting with 76b3102bb53f95c2426e906b8740951d246e02a593b89cbf2b754d9a0131230b not found: ID does not exist"
Sep 18 18:14:56.472630 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:56.472606    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f7e72d9f-54b7648c9b-z87kp"]
Sep 18 18:14:56.478431 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:56.478411    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f7e72d9f-54b7648c9b-z87kp"]
Sep 18 18:14:56.491351 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:56.491326    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f7e72d9f-54b9945bdb-zwjgm"]
Sep 18 18:14:56.495805 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:56.495784    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f7e72d9f-54b9945bdb-zwjgm"]
Sep 18 18:14:57.142484 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:57.142404    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:57.853489 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:57.853458    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:57.862731 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:57.862704    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19a9dc6f-6b51-4639-bfb5-4b4122a552ee" path="/var/lib/kubelet/pods/19a9dc6f-6b51-4639-bfb5-4b4122a552ee/volumes"
Sep 18 18:14:57.863065 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:57.863052    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57c13b34-8239-4165-b2c4-8e512de7b5b3" path="/var/lib/kubelet/pods/57c13b34-8239-4165-b2c4-8e512de7b5b3/volumes"
Sep 18 18:14:57.863425 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:57.863413    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94100fcf-733a-4b2e-b17d-25a7d14be7a3" path="/var/lib/kubelet/pods/94100fcf-733a-4b2e-b17d-25a7d14be7a3/volumes"
Sep 18 18:14:58.490034 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:58.489999    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:59.250145 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:59.250101    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:14:59.879617 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:14:59.879589    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:15:00.266364 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:15:00.266332    2561 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19a9dc6f_6b51_4639_bfb5_4b4122a552ee.slice/crio-39ef5b47d9e86b9e7e648d8805c6a75b1b2b9d3a88609d1c7e1cf22aabe692e9.scope\": RecentStats: unable to find data in memory cache]"
Sep 18 18:15:01.333549 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:01.333520    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-53cde3/auth-test-model-53cde3-kserve-75dcb68b95-d4j87"
Sep 18 18:15:01.333968 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:01.333561    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-53cde3/auth-test-model-53cde3-kserve-75dcb68b95-d4j87"
Sep 18 18:15:01.345703 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:01.345670    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-53cde3/auth-test-model-53cde3-kserve-75dcb68b95-d4j87"
Sep 18 18:15:01.346438 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:01.346419    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:15:01.482904 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:01.482871    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-53cde3/auth-test-model-53cde3-kserve-75dcb68b95-d4j87"
Sep 18 18:15:02.815279 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:02.815244    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-03e512/distinct-w4-03e512-kserve-b5d574f95-t8m5k"]
Sep 18 18:15:02.815664 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:02.815652    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19a9dc6f-6b51-4639-bfb5-4b4122a552ee" containerName="payload-processing"
Sep 18 18:15:02.815664 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:02.815663    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="19a9dc6f-6b51-4639-bfb5-4b4122a552ee" containerName="payload-processing"
Sep 18 18:15:02.815735 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:02.815689    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57c13b34-8239-4165-b2c4-8e512de7b5b3" containerName="payload-pre-processing"
Sep 18 18:15:02.815735 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:02.815695    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="57c13b34-8239-4165-b2c4-8e512de7b5b3" containerName="payload-pre-processing"
Sep 18 18:15:02.815735 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:02.815703    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94100fcf-733a-4b2e-b17d-25a7d14be7a3" containerName="maas-api"
Sep 18 18:15:02.815735 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:02.815710    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="94100fcf-733a-4b2e-b17d-25a7d14be7a3" containerName="maas-api"
Sep 18 18:15:02.815856 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:02.815776    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="94100fcf-733a-4b2e-b17d-25a7d14be7a3" containerName="maas-api"
Sep 18 18:15:02.815856 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:02.815785    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="19a9dc6f-6b51-4639-bfb5-4b4122a552ee" containerName="payload-processing"
Sep 18 18:15:02.815856 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:02.815794    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="57c13b34-8239-4165-b2c4-8e512de7b5b3" containerName="payload-pre-processing"
Sep 18 18:15:02.820732 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:02.820711    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-03e512/distinct-w4-03e512-kserve-b5d574f95-t8m5k"
Sep 18 18:15:02.822957 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:02.822936    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-03e512\"/\"distinct-w4-03e512-kserve-self-signed-certs\""
Sep 18 18:15:02.828164 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:02.828143    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-03e512/distinct-w4-03e512-kserve-b5d574f95-t8m5k"]
Sep 18 18:15:02.930808 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:02.930780    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:15:02.963690 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:02.963659    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6430cf8a-abc8-4ebb-8c5e-d963fc017ead-dshm\") pod \"distinct-w4-03e512-kserve-b5d574f95-t8m5k\" (UID: \"6430cf8a-abc8-4ebb-8c5e-d963fc017ead\") " pod="e2e-models-e2e-worker-w4-03e512/distinct-w4-03e512-kserve-b5d574f95-t8m5k"
Sep 18 18:15:02.963839 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:02.963697    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6430cf8a-abc8-4ebb-8c5e-d963fc017ead-tmp-dir\") pod \"distinct-w4-03e512-kserve-b5d574f95-t8m5k\" (UID: \"6430cf8a-abc8-4ebb-8c5e-d963fc017ead\") " pod="e2e-models-e2e-worker-w4-03e512/distinct-w4-03e512-kserve-b5d574f95-t8m5k"
Sep 18 18:15:02.963839 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:02.963722    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6430cf8a-abc8-4ebb-8c5e-d963fc017ead-model-cache\") pod \"distinct-w4-03e512-kserve-b5d574f95-t8m5k\" (UID: \"6430cf8a-abc8-4ebb-8c5e-d963fc017ead\") " pod="e2e-models-e2e-worker-w4-03e512/distinct-w4-03e512-kserve-b5d574f95-t8m5k"
Sep 18 18:15:02.963839 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:02.963799    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6430cf8a-abc8-4ebb-8c5e-d963fc017ead-home\") pod \"distinct-w4-03e512-kserve-b5d574f95-t8m5k\" (UID: \"6430cf8a-abc8-4ebb-8c5e-d963fc017ead\") " pod="e2e-models-e2e-worker-w4-03e512/distinct-w4-03e512-kserve-b5d574f95-t8m5k"
Sep 18 18:15:02.963839 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:02.963827    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qz8k\" (UniqueName: \"kubernetes.io/projected/6430cf8a-abc8-4ebb-8c5e-d963fc017ead-kube-api-access-9qz8k\") pod \"distinct-w4-03e512-kserve-b5d574f95-t8m5k\" (UID: \"6430cf8a-abc8-4ebb-8c5e-d963fc017ead\") " pod="e2e-models-e2e-worker-w4-03e512/distinct-w4-03e512-kserve-b5d574f95-t8m5k"
Sep 18 18:15:02.964086 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:02.963967    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6430cf8a-abc8-4ebb-8c5e-d963fc017ead-tls-certs\") pod \"distinct-w4-03e512-kserve-b5d574f95-t8m5k\" (UID: \"6430cf8a-abc8-4ebb-8c5e-d963fc017ead\") " pod="e2e-models-e2e-worker-w4-03e512/distinct-w4-03e512-kserve-b5d574f95-t8m5k"
Sep 18 18:15:03.065675 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:03.065590    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6430cf8a-abc8-4ebb-8c5e-d963fc017ead-tls-certs\") pod \"distinct-w4-03e512-kserve-b5d574f95-t8m5k\" (UID: \"6430cf8a-abc8-4ebb-8c5e-d963fc017ead\") " pod="e2e-models-e2e-worker-w4-03e512/distinct-w4-03e512-kserve-b5d574f95-t8m5k"
Sep 18 18:15:03.065675 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:03.065671    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6430cf8a-abc8-4ebb-8c5e-d963fc017ead-dshm\") pod \"distinct-w4-03e512-kserve-b5d574f95-t8m5k\" (UID: \"6430cf8a-abc8-4ebb-8c5e-d963fc017ead\") " pod="e2e-models-e2e-worker-w4-03e512/distinct-w4-03e512-kserve-b5d574f95-t8m5k"
Sep 18 18:15:03.065922 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:03.065712    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6430cf8a-abc8-4ebb-8c5e-d963fc017ead-tmp-dir\") pod \"distinct-w4-03e512-kserve-b5d574f95-t8m5k\" (UID: \"6430cf8a-abc8-4ebb-8c5e-d963fc017ead\") " pod="e2e-models-e2e-worker-w4-03e512/distinct-w4-03e512-kserve-b5d574f95-t8m5k"
Sep 18 18:15:03.065922 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:03.065749    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6430cf8a-abc8-4ebb-8c5e-d963fc017ead-model-cache\") pod \"distinct-w4-03e512-kserve-b5d574f95-t8m5k\" (UID: \"6430cf8a-abc8-4ebb-8c5e-d963fc017ead\") " pod="e2e-models-e2e-worker-w4-03e512/distinct-w4-03e512-kserve-b5d574f95-t8m5k"
Sep 18 18:15:03.065922 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:03.065792    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6430cf8a-abc8-4ebb-8c5e-d963fc017ead-home\") pod \"distinct-w4-03e512-kserve-b5d574f95-t8m5k\" (UID: \"6430cf8a-abc8-4ebb-8c5e-d963fc017ead\") " pod="e2e-models-e2e-worker-w4-03e512/distinct-w4-03e512-kserve-b5d574f95-t8m5k"
Sep 18 18:15:03.065922 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:03.065818    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qz8k\" (UniqueName: \"kubernetes.io/projected/6430cf8a-abc8-4ebb-8c5e-d963fc017ead-kube-api-access-9qz8k\") pod \"distinct-w4-03e512-kserve-b5d574f95-t8m5k\" (UID: \"6430cf8a-abc8-4ebb-8c5e-d963fc017ead\") " pod="e2e-models-e2e-worker-w4-03e512/distinct-w4-03e512-kserve-b5d574f95-t8m5k"
Sep 18 18:15:03.066269 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:03.066237    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6430cf8a-abc8-4ebb-8c5e-d963fc017ead-tmp-dir\") pod \"distinct-w4-03e512-kserve-b5d574f95-t8m5k\" (UID: \"6430cf8a-abc8-4ebb-8c5e-d963fc017ead\") " pod="e2e-models-e2e-worker-w4-03e512/distinct-w4-03e512-kserve-b5d574f95-t8m5k"
Sep 18 18:15:03.066411 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:03.066272    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6430cf8a-abc8-4ebb-8c5e-d963fc017ead-home\") pod \"distinct-w4-03e512-kserve-b5d574f95-t8m5k\" (UID: \"6430cf8a-abc8-4ebb-8c5e-d963fc017ead\") " pod="e2e-models-e2e-worker-w4-03e512/distinct-w4-03e512-kserve-b5d574f95-t8m5k"
Sep 18 18:15:03.066517 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:03.066496    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6430cf8a-abc8-4ebb-8c5e-d963fc017ead-model-cache\") pod \"distinct-w4-03e512-kserve-b5d574f95-t8m5k\" (UID: \"6430cf8a-abc8-4ebb-8c5e-d963fc017ead\") " pod="e2e-models-e2e-worker-w4-03e512/distinct-w4-03e512-kserve-b5d574f95-t8m5k"
Sep 18 18:15:03.067957 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:03.067928    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6430cf8a-abc8-4ebb-8c5e-d963fc017ead-dshm\") pod \"distinct-w4-03e512-kserve-b5d574f95-t8m5k\" (UID: \"6430cf8a-abc8-4ebb-8c5e-d963fc017ead\") " pod="e2e-models-e2e-worker-w4-03e512/distinct-w4-03e512-kserve-b5d574f95-t8m5k"
Sep 18 18:15:03.068223 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:03.068206    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6430cf8a-abc8-4ebb-8c5e-d963fc017ead-tls-certs\") pod \"distinct-w4-03e512-kserve-b5d574f95-t8m5k\" (UID: \"6430cf8a-abc8-4ebb-8c5e-d963fc017ead\") " pod="e2e-models-e2e-worker-w4-03e512/distinct-w4-03e512-kserve-b5d574f95-t8m5k"
Sep 18 18:15:03.074332 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:03.074302    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qz8k\" (UniqueName: \"kubernetes.io/projected/6430cf8a-abc8-4ebb-8c5e-d963fc017ead-kube-api-access-9qz8k\") pod \"distinct-w4-03e512-kserve-b5d574f95-t8m5k\" (UID: \"6430cf8a-abc8-4ebb-8c5e-d963fc017ead\") " pod="e2e-models-e2e-worker-w4-03e512/distinct-w4-03e512-kserve-b5d574f95-t8m5k"
Sep 18 18:15:03.130287 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:03.130241    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-03e512/distinct-w4-03e512-kserve-b5d574f95-t8m5k"
Sep 18 18:15:03.404643 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:03.404616    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-03e512/distinct-w4-03e512-kserve-b5d574f95-t8m5k"]
Sep 18 18:15:03.406262 ip-10-0-140-54 kubenswrapper[2561]: W0918 18:15:03.406235    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6430cf8a_abc8_4ebb_8c5e_d963fc017ead.slice/crio-3824e8e9b2e93f56bb6f49aa4d0ee4856925ed2bb8f59d5f4b617ead31b1cd16 WatchSource:0}: Error finding container 3824e8e9b2e93f56bb6f49aa4d0ee4856925ed2bb8f59d5f4b617ead31b1cd16: Status 404 returned error can't find the container with id 3824e8e9b2e93f56bb6f49aa4d0ee4856925ed2bb8f59d5f4b617ead31b1cd16
Sep 18 18:15:03.480727 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:03.480695    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-03e512/distinct-w4-03e512-kserve-b5d574f95-t8m5k" event={"ID":"6430cf8a-abc8-4ebb-8c5e-d963fc017ead","Type":"ContainerStarted","Data":"f593bb7a57534b554deecca79feb29e2578f576a8aaa2dbed3795f7c6c6a0d5a"}
Sep 18 18:15:03.480855 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:03.480744    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-03e512/distinct-w4-03e512-kserve-b5d574f95-t8m5k" event={"ID":"6430cf8a-abc8-4ebb-8c5e-d963fc017ead","Type":"ContainerStarted","Data":"3824e8e9b2e93f56bb6f49aa4d0ee4856925ed2bb8f59d5f4b617ead31b1cd16"}
Sep 18 18:15:03.499257 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:03.499213    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-03e512/distinct-w4-03e512-kserve-b5d574f95-t8m5k" podStartSLOduration=1.499197955 podStartE2EDuration="1.499197955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:15:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:15:03.497414222 +0000 UTC m=+1508.124545858" watchObservedRunningTime="2026-09-18 18:15:03.499197955 +0000 UTC m=+1508.126329613"
Sep 18 18:15:04.613094 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:15:04.613062    2561 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19a9dc6f_6b51_4639_bfb5_4b4122a552ee.slice/crio-39ef5b47d9e86b9e7e648d8805c6a75b1b2b9d3a88609d1c7e1cf22aabe692e9.scope\": RecentStats: unable to find data in memory cache]"
Sep 18 18:15:10.319844 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:15:10.319802    2561 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19a9dc6f_6b51_4639_bfb5_4b4122a552ee.slice/crio-39ef5b47d9e86b9e7e648d8805c6a75b1b2b9d3a88609d1c7e1cf22aabe692e9.scope\": RecentStats: unable to find data in memory cache]"
Sep 18 18:15:13.130540 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:13.130499    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-03e512/distinct-w4-03e512-kserve-b5d574f95-t8m5k"
Sep 18 18:15:13.132850 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:13.130566    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-03e512/distinct-w4-03e512-kserve-b5d574f95-t8m5k"
Sep 18 18:15:13.149334 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:13.143922    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-03e512/distinct-w4-03e512-kserve-b5d574f95-t8m5k"
Sep 18 18:15:13.544582 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:13.544552    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-03e512/distinct-w4-03e512-kserve-b5d574f95-t8m5k"
Sep 18 18:15:14.746393 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:15:14.746359    2561 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19a9dc6f_6b51_4639_bfb5_4b4122a552ee.slice/crio-39ef5b47d9e86b9e7e648d8805c6a75b1b2b9d3a88609d1c7e1cf22aabe692e9.scope\": RecentStats: unable to find data in memory cache]"
Sep 18 18:15:14.746849 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:15:14.746768    2561 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19a9dc6f_6b51_4639_bfb5_4b4122a552ee.slice/crio-39ef5b47d9e86b9e7e648d8805c6a75b1b2b9d3a88609d1c7e1cf22aabe692e9.scope\": RecentStats: unable to find data in memory cache]"
Sep 18 18:15:19.642349 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:15:19.642273    2561 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19a9dc6f_6b51_4639_bfb5_4b4122a552ee.slice/crio-39ef5b47d9e86b9e7e648d8805c6a75b1b2b9d3a88609d1c7e1cf22aabe692e9.scope\": RecentStats: unable to find data in memory cache]"
Sep 18 18:15:19.699711 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:19.699678    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:15:19.762992 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:19.762957    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:15:20.330448 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:15:20.330416    2561 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19a9dc6f_6b51_4639_bfb5_4b4122a552ee.slice/crio-39ef5b47d9e86b9e7e648d8805c6a75b1b2b9d3a88609d1c7e1cf22aabe692e9.scope\": RecentStats: unable to find data in memory cache]"
Sep 18 18:15:20.349875 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:20.349849    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:15:30.383440 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:15:30.383401    2561 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19a9dc6f_6b51_4639_bfb5_4b4122a552ee.slice/crio-39ef5b47d9e86b9e7e648d8805c6a75b1b2b9d3a88609d1c7e1cf22aabe692e9.scope\": RecentStats: unable to find data in memory cache]"
Sep 18 18:15:32.917322 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:32.917285    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:15:34.602445 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:15:34.602411    2561 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19a9dc6f_6b51_4639_bfb5_4b4122a552ee.slice/crio-39ef5b47d9e86b9e7e648d8805c6a75b1b2b9d3a88609d1c7e1cf22aabe692e9.scope\": RecentStats: unable to find data in memory cache]"
Sep 18 18:15:38.344164 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:38.344125    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:15:38.831326 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:38.831296    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:15:40.436230 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:15:40.436192    2561 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19a9dc6f_6b51_4639_bfb5_4b4122a552ee.slice/crio-39ef5b47d9e86b9e7e648d8805c6a75b1b2b9d3a88609d1c7e1cf22aabe692e9.scope\": RecentStats: unable to find data in memory cache]"
Sep 18 18:15:40.670798 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:40.670771    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:15:46.128836 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:46.128805    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-341bd4/auth-test-model-341bd4-kserve-5c4967bb4c-mqf6c"]
Sep 18 18:15:46.132596 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:46.132570    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-341bd4/auth-test-model-341bd4-kserve-5c4967bb4c-mqf6c"
Sep 18 18:15:46.135995 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:46.135967    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-341bd4\"/\"kube-root-ca.crt\""
Sep 18 18:15:46.136136 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:46.135967    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-341bd4\"/\"default-dockercfg-j284w\""
Sep 18 18:15:46.136136 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:46.135967    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-341bd4\"/\"auth-test-model-341bd4-kserve-self-signed-certs\""
Sep 18 18:15:46.136136 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:46.135978    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-341bd4\"/\"openshift-service-ca.crt\""
Sep 18 18:15:46.142863 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:46.142840    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-341bd4/auth-test-model-341bd4-kserve-5c4967bb4c-mqf6c"]
Sep 18 18:15:46.160618 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:46.160592    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:15:46.273071 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:46.273036    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0864e158-2933-4867-b3b9-8ef5d9749ad5-tls-certs\") pod \"auth-test-model-341bd4-kserve-5c4967bb4c-mqf6c\" (UID: \"0864e158-2933-4867-b3b9-8ef5d9749ad5\") " pod="ai-tenant-e2e-shared-b-w3-341bd4/auth-test-model-341bd4-kserve-5c4967bb4c-mqf6c"
Sep 18 18:15:46.273272 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:46.273080    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmrwf\" (UniqueName: \"kubernetes.io/projected/0864e158-2933-4867-b3b9-8ef5d9749ad5-kube-api-access-cmrwf\") pod \"auth-test-model-341bd4-kserve-5c4967bb4c-mqf6c\" (UID: \"0864e158-2933-4867-b3b9-8ef5d9749ad5\") " pod="ai-tenant-e2e-shared-b-w3-341bd4/auth-test-model-341bd4-kserve-5c4967bb4c-mqf6c"
Sep 18 18:15:46.273272 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:46.273141    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0864e158-2933-4867-b3b9-8ef5d9749ad5-tmp-dir\") pod \"auth-test-model-341bd4-kserve-5c4967bb4c-mqf6c\" (UID: \"0864e158-2933-4867-b3b9-8ef5d9749ad5\") " pod="ai-tenant-e2e-shared-b-w3-341bd4/auth-test-model-341bd4-kserve-5c4967bb4c-mqf6c"
Sep 18 18:15:46.273272 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:46.273223    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0864e158-2933-4867-b3b9-8ef5d9749ad5-dshm\") pod \"auth-test-model-341bd4-kserve-5c4967bb4c-mqf6c\" (UID: \"0864e158-2933-4867-b3b9-8ef5d9749ad5\") " pod="ai-tenant-e2e-shared-b-w3-341bd4/auth-test-model-341bd4-kserve-5c4967bb4c-mqf6c"
Sep 18 18:15:46.273455 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:46.273275    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0864e158-2933-4867-b3b9-8ef5d9749ad5-model-cache\") pod \"auth-test-model-341bd4-kserve-5c4967bb4c-mqf6c\" (UID: \"0864e158-2933-4867-b3b9-8ef5d9749ad5\") " pod="ai-tenant-e2e-shared-b-w3-341bd4/auth-test-model-341bd4-kserve-5c4967bb4c-mqf6c"
Sep 18 18:15:46.273455 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:46.273342    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0864e158-2933-4867-b3b9-8ef5d9749ad5-home\") pod \"auth-test-model-341bd4-kserve-5c4967bb4c-mqf6c\" (UID: \"0864e158-2933-4867-b3b9-8ef5d9749ad5\") " pod="ai-tenant-e2e-shared-b-w3-341bd4/auth-test-model-341bd4-kserve-5c4967bb4c-mqf6c"
Sep 18 18:15:46.374352 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:46.374314    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0864e158-2933-4867-b3b9-8ef5d9749ad5-tls-certs\") pod \"auth-test-model-341bd4-kserve-5c4967bb4c-mqf6c\" (UID: \"0864e158-2933-4867-b3b9-8ef5d9749ad5\") " pod="ai-tenant-e2e-shared-b-w3-341bd4/auth-test-model-341bd4-kserve-5c4967bb4c-mqf6c"
Sep 18 18:15:46.374352 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:46.374358    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmrwf\" (UniqueName: \"kubernetes.io/projected/0864e158-2933-4867-b3b9-8ef5d9749ad5-kube-api-access-cmrwf\") pod \"auth-test-model-341bd4-kserve-5c4967bb4c-mqf6c\" (UID: \"0864e158-2933-4867-b3b9-8ef5d9749ad5\") " pod="ai-tenant-e2e-shared-b-w3-341bd4/auth-test-model-341bd4-kserve-5c4967bb4c-mqf6c"
Sep 18 18:15:46.374561 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:46.374393    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0864e158-2933-4867-b3b9-8ef5d9749ad5-tmp-dir\") pod \"auth-test-model-341bd4-kserve-5c4967bb4c-mqf6c\" (UID: \"0864e158-2933-4867-b3b9-8ef5d9749ad5\") " pod="ai-tenant-e2e-shared-b-w3-341bd4/auth-test-model-341bd4-kserve-5c4967bb4c-mqf6c"
Sep 18 18:15:46.374561 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:46.374463    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0864e158-2933-4867-b3b9-8ef5d9749ad5-dshm\") pod \"auth-test-model-341bd4-kserve-5c4967bb4c-mqf6c\" (UID: \"0864e158-2933-4867-b3b9-8ef5d9749ad5\") " pod="ai-tenant-e2e-shared-b-w3-341bd4/auth-test-model-341bd4-kserve-5c4967bb4c-mqf6c"
Sep 18 18:15:46.374561 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:46.374521    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0864e158-2933-4867-b3b9-8ef5d9749ad5-model-cache\") pod \"auth-test-model-341bd4-kserve-5c4967bb4c-mqf6c\" (UID: \"0864e158-2933-4867-b3b9-8ef5d9749ad5\") " pod="ai-tenant-e2e-shared-b-w3-341bd4/auth-test-model-341bd4-kserve-5c4967bb4c-mqf6c"
Sep 18 18:15:46.374693 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:46.374568    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0864e158-2933-4867-b3b9-8ef5d9749ad5-home\") pod \"auth-test-model-341bd4-kserve-5c4967bb4c-mqf6c\" (UID: \"0864e158-2933-4867-b3b9-8ef5d9749ad5\") " pod="ai-tenant-e2e-shared-b-w3-341bd4/auth-test-model-341bd4-kserve-5c4967bb4c-mqf6c"
Sep 18 18:15:46.374905 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:46.374880    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0864e158-2933-4867-b3b9-8ef5d9749ad5-tmp-dir\") pod \"auth-test-model-341bd4-kserve-5c4967bb4c-mqf6c\" (UID: \"0864e158-2933-4867-b3b9-8ef5d9749ad5\") " pod="ai-tenant-e2e-shared-b-w3-341bd4/auth-test-model-341bd4-kserve-5c4967bb4c-mqf6c"
Sep 18 18:15:46.374993 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:46.374898    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0864e158-2933-4867-b3b9-8ef5d9749ad5-model-cache\") pod \"auth-test-model-341bd4-kserve-5c4967bb4c-mqf6c\" (UID: \"0864e158-2933-4867-b3b9-8ef5d9749ad5\") " pod="ai-tenant-e2e-shared-b-w3-341bd4/auth-test-model-341bd4-kserve-5c4967bb4c-mqf6c"
Sep 18 18:15:46.374993 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:46.374940    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0864e158-2933-4867-b3b9-8ef5d9749ad5-home\") pod \"auth-test-model-341bd4-kserve-5c4967bb4c-mqf6c\" (UID: \"0864e158-2933-4867-b3b9-8ef5d9749ad5\") " pod="ai-tenant-e2e-shared-b-w3-341bd4/auth-test-model-341bd4-kserve-5c4967bb4c-mqf6c"
Sep 18 18:15:46.376673 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:46.376651    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0864e158-2933-4867-b3b9-8ef5d9749ad5-dshm\") pod \"auth-test-model-341bd4-kserve-5c4967bb4c-mqf6c\" (UID: \"0864e158-2933-4867-b3b9-8ef5d9749ad5\") " pod="ai-tenant-e2e-shared-b-w3-341bd4/auth-test-model-341bd4-kserve-5c4967bb4c-mqf6c"
Sep 18 18:15:46.376907 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:46.376889    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0864e158-2933-4867-b3b9-8ef5d9749ad5-tls-certs\") pod \"auth-test-model-341bd4-kserve-5c4967bb4c-mqf6c\" (UID: \"0864e158-2933-4867-b3b9-8ef5d9749ad5\") " pod="ai-tenant-e2e-shared-b-w3-341bd4/auth-test-model-341bd4-kserve-5c4967bb4c-mqf6c"
Sep 18 18:15:46.383331 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:46.383277    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmrwf\" (UniqueName: \"kubernetes.io/projected/0864e158-2933-4867-b3b9-8ef5d9749ad5-kube-api-access-cmrwf\") pod \"auth-test-model-341bd4-kserve-5c4967bb4c-mqf6c\" (UID: \"0864e158-2933-4867-b3b9-8ef5d9749ad5\") " pod="ai-tenant-e2e-shared-b-w3-341bd4/auth-test-model-341bd4-kserve-5c4967bb4c-mqf6c"
Sep 18 18:15:46.443214 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:46.443174    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-341bd4/auth-test-model-341bd4-kserve-5c4967bb4c-mqf6c"
Sep 18 18:15:46.720990 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:46.720959    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-341bd4/auth-test-model-341bd4-kserve-5c4967bb4c-mqf6c"]
Sep 18 18:15:46.722466 ip-10-0-140-54 kubenswrapper[2561]: W0918 18:15:46.722436    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0864e158_2933_4867_b3b9_8ef5d9749ad5.slice/crio-d66a1c44bb198a763d61dd2b08b26645b387c76522cd6ede96c857204560082b WatchSource:0}: Error finding container d66a1c44bb198a763d61dd2b08b26645b387c76522cd6ede96c857204560082b: Status 404 returned error can't find the container with id d66a1c44bb198a763d61dd2b08b26645b387c76522cd6ede96c857204560082b
Sep 18 18:15:47.716202 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:47.716169    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-341bd4/auth-test-model-341bd4-kserve-5c4967bb4c-mqf6c" event={"ID":"0864e158-2933-4867-b3b9-8ef5d9749ad5","Type":"ContainerStarted","Data":"58675b48e308be18cbe39aa419a9f1837d5a66f965b4c61e316d21eb84b29766"}
Sep 18 18:15:47.716585 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:47.716209    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-341bd4/auth-test-model-341bd4-kserve-5c4967bb4c-mqf6c" event={"ID":"0864e158-2933-4867-b3b9-8ef5d9749ad5","Type":"ContainerStarted","Data":"d66a1c44bb198a763d61dd2b08b26645b387c76522cd6ede96c857204560082b"}
Sep 18 18:15:47.737066 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:47.737012    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-341bd4/auth-test-model-341bd4-kserve-5c4967bb4c-mqf6c" podStartSLOduration=1.736993655 podStartE2EDuration="1.736993655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:15:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:15:47.734374632 +0000 UTC m=+1552.361506264" watchObservedRunningTime="2026-09-18 18:15:47.736993655 +0000 UTC m=+1552.364125275"
Sep 18 18:15:49.648716 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:15:49.648686    2561 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19a9dc6f_6b51_4639_bfb5_4b4122a552ee.slice/crio-39ef5b47d9e86b9e7e648d8805c6a75b1b2b9d3a88609d1c7e1cf22aabe692e9.scope\": RecentStats: unable to find data in memory cache]"
Sep 18 18:15:50.451070 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:15:50.451032    2561 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19a9dc6f_6b51_4639_bfb5_4b4122a552ee.slice/crio-39ef5b47d9e86b9e7e648d8805c6a75b1b2b9d3a88609d1c7e1cf22aabe692e9.scope\": RecentStats: unable to find data in memory cache]"
Sep 18 18:15:55.839055 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:55.839021    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:15:56.444246 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:56.444186    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-341bd4/auth-test-model-341bd4-kserve-5c4967bb4c-mqf6c"
Sep 18 18:15:56.444466 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:56.444266    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-341bd4/auth-test-model-341bd4-kserve-5c4967bb4c-mqf6c"
Sep 18 18:15:56.456909 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:56.456877    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-341bd4/auth-test-model-341bd4-kserve-5c4967bb4c-mqf6c"
Sep 18 18:15:56.774472 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:15:56.774388    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-341bd4/auth-test-model-341bd4-kserve-5c4967bb4c-mqf6c"
Sep 18 18:16:03.350163 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:16:03.350109    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:10.351929 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:16:10.351890    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:10.452843 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:16:10.452813    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:10.625677 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:16:10.625567    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:12.035900 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:16:12.035866    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:12.134801 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:16:12.134770    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:12.301371 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:16:12.301281    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:12.899975 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:16:12.899935    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:12.989930 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:16:12.989897    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:13.087781 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:16:13.087752    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:14.219094 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:16:14.219064    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:14.351469 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:16:14.351438    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:20.075545 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:16:20.075512    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:20.139605 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:16:20.139580    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:20.772351 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:16:20.772318    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:26.755462 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:16:26.755397    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:26.819546 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:16:26.819512    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:26.905667 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:16:26.905633    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:27.004858 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:16:27.004830    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:37.465257 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:16:37.465227    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:37.711879 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:16:37.711848    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:47.930756 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:16:47.930724    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:52.179082 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:16:52.179048    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:16:58.606904 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:16:58.606870    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:17:03.145655 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:17:03.145624    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:17:04.419035 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:17:04.419005    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:17:04.515744 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:17:04.515712    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:17:04.753643 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:17:04.753566    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:17:05.050905 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:17:05.050827    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:17:05.643638 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:17:05.643606    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:17:07.748410 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:17:07.748379    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:17:08.551532 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:17:08.551494    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:17:08.657482 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:17:08.657447    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:17:09.405696 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:17:09.405667    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:17:10.132384 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:17:10.132353    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:17:10.813817 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:17:10.813786    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:17:11.557084 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:17:11.557054    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:17:12.233810 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:17:12.233771    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:17:12.290283 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:17:12.290254    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:17:12.676369 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:17:12.676335    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:17:12.779182 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:17:12.779153    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:17:12.821171 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:17:12.821138    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e9a4f9-66d59b466c-psswn"]
Sep 18 18:17:12.824917 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:17:12.824896    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e9a4f9-66d59b466c-psswn"
Sep 18 18:17:12.827356 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:17:12.827338    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-e9a4f9-dockercfg-nw4s6\""
Sep 18 18:17:12.827443 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:17:12.827336    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-e9a4f9\""
Sep 18 18:17:12.834449 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:17:12.834423    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50f1cb39-8933-498e-9789-9d4bfa0416f7-plugins-config-volume\") pod \"payload-processing-e2e-api-a-e9a4f9-66d59b466c-psswn\" (UID: \"50f1cb39-8933-498e-9789-9d4bfa0416f7\") " pod="openshift-ingress/payload-processing-e2e-api-a-e9a4f9-66d59b466c-psswn"
Sep 18 18:17:12.834558 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:17:12.834488    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvxg7\" (UniqueName: \"kubernetes.io/projected/50f1cb39-8933-498e-9789-9d4bfa0416f7-kube-api-access-wvxg7\") pod \"payload-processing-e2e-api-a-e9a4f9-66d59b466c-psswn\" (UID: \"50f1cb39-8933-498e-9789-9d4bfa0416f7\") " pod="openshift-ingress/payload-processing-e2e-api-a-e9a4f9-66d59b466c-psswn"
Sep 18 18:17:12.836151 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:17:12.836104    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e9a4f9-66d59b466c-psswn"]
Sep 18 18:17:12.935091 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:17:12.935007    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50f1cb39-8933-498e-9789-9d4bfa0416f7-plugins-config-volume\") pod \"payload-processing-e2e-api-a-e9a4f9-66d59b466c-psswn\" (UID: \"50f1cb39-8933-498e-9789-9d4bfa0416f7\") " pod="openshift-ingress/payload-processing-e2e-api-a-e9a4f9-66d59b466c-psswn"
Sep 18 18:17:12.935091 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:17:12.935069    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvxg7\" (UniqueName: \"kubernetes.io/projected/50f1cb39-8933-498e-9789-9d4bfa0416f7-kube-api-access-wvxg7\") pod \"payload-processing-e2e-api-a-e9a4f9-66d59b466c-psswn\" (UID: \"50f1cb39-8933-498e-9789-9d4bfa0416f7\") " pod="openshift-ingress/payload-processing-e2e-api-a-e9a4f9-66d59b466c-psswn"
Sep 18 18:17:12.935696 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:17:12.935678    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50f1cb39-8933-498e-9789-9d4bfa0416f7-plugins-config-volume\") pod \"payload-processing-e2e-api-a-e9a4f9-66d59b466c-psswn\" (UID: \"50f1cb39-8933-498e-9789-9d4bfa0416f7\") " pod="openshift-ingress/payload-processing-e2e-api-a-e9a4f9-66d59b466c-psswn"
Sep 18 18:17:12.943431 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:17:12.943410    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvxg7\" (UniqueName: \"kubernetes.io/projected/50f1cb39-8933-498e-9789-9d4bfa0416f7-kube-api-access-wvxg7\") pod \"payload-processing-e2e-api-a-e9a4f9-66d59b466c-psswn\" (UID: \"50f1cb39-8933-498e-9789-9d4bfa0416f7\") " pod="openshift-ingress/payload-processing-e2e-api-a-e9a4f9-66d59b466c-psswn"
Sep 18 18:17:13.136257 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:17:13.136220    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e9a4f9-66d59b466c-psswn"
Sep 18 18:17:13.411960 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:17:13.411930    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e9a4f9-66d59b466c-psswn"]
Sep 18 18:17:13.414471 ip-10-0-140-54 kubenswrapper[2561]: W0918 18:17:13.414441    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50f1cb39_8933_498e_9789_9d4bfa0416f7.slice/crio-3ee52f3da9ec4d45d8e8ebd5a852720bd2d299ad2f334409a02813d7fb0733f8 WatchSource:0}: Error finding container 3ee52f3da9ec4d45d8e8ebd5a852720bd2d299ad2f334409a02813d7fb0733f8: Status 404 returned error can't find the container with id 3ee52f3da9ec4d45d8e8ebd5a852720bd2d299ad2f334409a02813d7fb0733f8
Sep 18 18:17:14.160673 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:17:14.160637    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e9a4f9-66d59b466c-psswn" event={"ID":"50f1cb39-8933-498e-9789-9d4bfa0416f7","Type":"ContainerStarted","Data":"4b53c5230757b3c72eeec92370b21d900beb4416614ce9ab7c6142f18517c453"}
Sep 18 18:17:14.160907 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:17:14.160694    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e9a4f9-66d59b466c-psswn" event={"ID":"50f1cb39-8933-498e-9789-9d4bfa0416f7","Type":"ContainerStarted","Data":"3ee52f3da9ec4d45d8e8ebd5a852720bd2d299ad2f334409a02813d7fb0733f8"}
Sep 18 18:17:14.160907 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:17:14.160738    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-e9a4f9-66d59b466c-psswn"
Sep 18 18:17:14.178240 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:17:14.178189    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-e9a4f9-66d59b466c-psswn" podStartSLOduration=2.178174912 podStartE2EDuration="2.178174912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:17:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:17:14.177000643 +0000 UTC m=+1638.804132276" watchObservedRunningTime="2026-09-18 18:17:14.178174912 +0000 UTC m=+1638.805306532"
Sep 18 18:17:25.167886 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:17:25.167850    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-e9a4f9-66d59b466c-psswn"
Sep 18 18:17:34.057282 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:17:34.057249    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:17:34.825506 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:17:34.825430    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:17:35.457642 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:17:35.457605    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:17:36.185314 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:17:36.185272    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:17:36.934155 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:17:36.934097    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:17:37.703908 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:17:37.703878    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:17:38.508340 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:17:38.508308    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:17:38.608600 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:17:38.608572    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:17:38.950529 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:17:38.950484    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:17:39.263804 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:17:39.263719    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:17:39.359801 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:17:39.359771    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:17:41.401041 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:17:41.401001    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:17:42.081461 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:17:42.081426    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:17:42.175851 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:17:42.175818    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:17:42.275860 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:17:42.275807    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:17:42.376335 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:17:42.376253    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:17:43.131682 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:17:43.131642    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:17:43.839435 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:17:43.839406    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:17:44.631348 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:17:44.631321    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:17:45.271678 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:17:45.271642    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:17:45.917263 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:17:45.917232    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:17:46.824315 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:17:46.824287    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:17:46.902346 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:17:46.902309    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:17:47.223753 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:17:47.223716    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:17:47.324872 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:17:47.324840    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:17:47.400186 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:17:47.400155    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:08.714403 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:08.714366    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:09.417587 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:09.417556    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:10.096267 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:10.096237    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:10.761644 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:10.761603    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:11.525957 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:11.525929    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:12.148251 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:12.148219    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:12.249030 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:12.248997    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:13.150081 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:13.150050    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:13.747657 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:13.747621    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:15.285830 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:15.285789    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:16.097636 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:16.097606    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:16.947716 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:16.947677    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:17.630839 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:17.630810    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:18.493508 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:18.493437    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:19.242647 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:19.242613    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:19.967353 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:19.967322    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:20.730288 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:20.730258    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:21.377302 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:21.377269    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:30.481436 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:30.481402    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:30.511353 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:30.511324    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e9a4f9-66d59b466c-psswn"]
Sep 18 18:18:30.511864 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:30.511832    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-e9a4f9-66d59b466c-psswn" podUID="50f1cb39-8933-498e-9789-9d4bfa0416f7" containerName="payload-processing" containerID="cri-o://4b53c5230757b3c72eeec92370b21d900beb4416614ce9ab7c6142f18517c453" gracePeriod=30
Sep 18 18:18:30.585844 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:30.585804    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:30.686852 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:30.686763    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:30.756983 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:30.756955    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e9a4f9-66d59b466c-psswn"
Sep 18 18:18:30.802869 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:30.802838    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:30.804942 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:30.804919    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wvxg7\" (UniqueName: \"kubernetes.io/projected/50f1cb39-8933-498e-9789-9d4bfa0416f7-kube-api-access-wvxg7\") pod \"50f1cb39-8933-498e-9789-9d4bfa0416f7\" (UID: \"50f1cb39-8933-498e-9789-9d4bfa0416f7\") "
Sep 18 18:18:30.805100 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:30.804965    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50f1cb39-8933-498e-9789-9d4bfa0416f7-plugins-config-volume\") pod \"50f1cb39-8933-498e-9789-9d4bfa0416f7\" (UID: \"50f1cb39-8933-498e-9789-9d4bfa0416f7\") "
Sep 18 18:18:30.805420 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:30.805397    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50f1cb39-8933-498e-9789-9d4bfa0416f7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "50f1cb39-8933-498e-9789-9d4bfa0416f7" (UID: "50f1cb39-8933-498e-9789-9d4bfa0416f7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:18:30.807087 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:30.807054    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50f1cb39-8933-498e-9789-9d4bfa0416f7-kube-api-access-wvxg7" (OuterVolumeSpecName: "kube-api-access-wvxg7") pod "50f1cb39-8933-498e-9789-9d4bfa0416f7" (UID: "50f1cb39-8933-498e-9789-9d4bfa0416f7"). InnerVolumeSpecName "kube-api-access-wvxg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:18:30.906602 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:30.906567    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50f1cb39-8933-498e-9789-9d4bfa0416f7-plugins-config-volume\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:18:30.906602 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:30.906600    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvxg7\" (UniqueName: \"kubernetes.io/projected/50f1cb39-8933-498e-9789-9d4bfa0416f7-kube-api-access-wvxg7\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:18:31.133683 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:31.133604    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:31.217046 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:31.217014    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:31.569813 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:31.569779    2561 generic.go:356] "Generic (PLEG): container finished" podID="50f1cb39-8933-498e-9789-9d4bfa0416f7" containerID="4b53c5230757b3c72eeec92370b21d900beb4416614ce9ab7c6142f18517c453" exitCode=0
Sep 18 18:18:31.570274 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:31.569831    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e9a4f9-66d59b466c-psswn"
Sep 18 18:18:31.570274 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:31.569870    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e9a4f9-66d59b466c-psswn" event={"ID":"50f1cb39-8933-498e-9789-9d4bfa0416f7","Type":"ContainerDied","Data":"4b53c5230757b3c72eeec92370b21d900beb4416614ce9ab7c6142f18517c453"}
Sep 18 18:18:31.570274 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:31.569922    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e9a4f9-66d59b466c-psswn" event={"ID":"50f1cb39-8933-498e-9789-9d4bfa0416f7","Type":"ContainerDied","Data":"3ee52f3da9ec4d45d8e8ebd5a852720bd2d299ad2f334409a02813d7fb0733f8"}
Sep 18 18:18:31.570274 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:31.569945    2561 scope.go:117] "RemoveContainer" containerID="4b53c5230757b3c72eeec92370b21d900beb4416614ce9ab7c6142f18517c453"
Sep 18 18:18:31.579381 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:31.579368    2561 scope.go:117] "RemoveContainer" containerID="4b53c5230757b3c72eeec92370b21d900beb4416614ce9ab7c6142f18517c453"
Sep 18 18:18:31.579653 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:18:31.579631    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b53c5230757b3c72eeec92370b21d900beb4416614ce9ab7c6142f18517c453\": container with ID starting with 4b53c5230757b3c72eeec92370b21d900beb4416614ce9ab7c6142f18517c453 not found: ID does not exist" containerID="4b53c5230757b3c72eeec92370b21d900beb4416614ce9ab7c6142f18517c453"
Sep 18 18:18:31.579707 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:31.579661    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b53c5230757b3c72eeec92370b21d900beb4416614ce9ab7c6142f18517c453"} err="failed to get container status \"4b53c5230757b3c72eeec92370b21d900beb4416614ce9ab7c6142f18517c453\": rpc error: code = NotFound desc = could not find container \"4b53c5230757b3c72eeec92370b21d900beb4416614ce9ab7c6142f18517c453\": container with ID starting with 4b53c5230757b3c72eeec92370b21d900beb4416614ce9ab7c6142f18517c453 not found: ID does not exist"
Sep 18 18:18:31.592097 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:31.592070    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e9a4f9-66d59b466c-psswn"]
Sep 18 18:18:31.598908 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:31.598882    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e9a4f9-66d59b466c-psswn"]
Sep 18 18:18:31.862164 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:31.862071    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50f1cb39-8933-498e-9789-9d4bfa0416f7" path="/var/lib/kubelet/pods/50f1cb39-8933-498e-9789-9d4bfa0416f7/volumes"
Sep 18 18:18:32.457442 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:32.457408    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:33.147209 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:33.147172    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:33.383135 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:33.383093    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:34.054701 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:34.054667    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:34.844970 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:34.844885    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:35.500333 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:35.500279    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:36.199386 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:36.199354    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:36.952911 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:36.952887    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:37.051309 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:37.051250    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:37.211468 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:37.211376    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:37.796362 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:37.796325    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:38.379402 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:38.379361    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-03e512/distinct-w4-03e512-kserve-b5d574f95-t8m5k"]
Sep 18 18:18:38.402206 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:38.402175    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-03e512/simulator-w4-03e512-kserve-645699d677-9nddq"]
Sep 18 18:18:38.475528 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:38.475490    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:38.823926 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:38.823881    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:39.690366 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:39.690335    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:40.560528 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:40.560496    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:41.380211 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:41.380167    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:42.390100 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:42.390068    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:43.144139 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:43.144088    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:43.926731 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:43.926699    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:47.714414 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:47.714382    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:49.537710 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:49.537681    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-03e512-7878686c45-mw6vc"]
Sep 18 18:18:49.538211 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:49.537912    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-03e512-7878686c45-mw6vc" podUID="6a5f114b-5a1f-48bf-800a-f3d70cb2fd5b" containerName="maas-api" containerID="cri-o://bb894e682b2e13a646d831c151c350404c07c28c9f4e78f8951b591a88ddd10e" gracePeriod=30
Sep 18 18:18:49.568179 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:49.568148    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:49.598018 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:49.597985    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-03e512-85bdf7444c-vwgzw"]
Sep 18 18:18:49.598325 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:49.598266    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-03e512-85bdf7444c-vwgzw" podUID="9606a239-122e-4814-9048-ab09b406183f" containerName="payload-processing" containerID="cri-o://dccd6dd8ffe8845a0148f14408e5b413f3bd3399a7c7e028d9293ecc1eb9a2dc" gracePeriod=30
Sep 18 18:18:49.604623 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:49.604588    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-03e512-5bd997d694-gp4gc"]
Sep 18 18:18:49.605196 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:49.604804    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-03e512-5bd997d694-gp4gc" podUID="33b62f5a-0f4d-4687-b901-05874f328161" containerName="payload-pre-processing" containerID="cri-o://86a11957ff99e383ff1f57570a531080a057392a18f2772cbdc70c13e32917ee" gracePeriod=30
Sep 18 18:18:49.639502 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:49.639479    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:49.680053 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:18:49.680027    2561 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33b62f5a_0f4d_4687_b901_05874f328161.slice/crio-86a11957ff99e383ff1f57570a531080a057392a18f2772cbdc70c13e32917ee.scope\": RecentStats: unable to find data in memory cache]"
Sep 18 18:18:49.710696 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:49.710673    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:49.813564 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:49.813536    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:49.900235 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:49.900208    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-03e512-85bdf7444c-vwgzw"
Sep 18 18:18:49.903689 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:49.903671    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-03e512-5bd997d694-gp4gc"
Sep 18 18:18:49.983087 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:49.983060    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-03e512-7878686c45-mw6vc"
Sep 18 18:18:49.999788 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:49.999767    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9606a239-122e-4814-9048-ab09b406183f-plugins-config-volume\") pod \"9606a239-122e-4814-9048-ab09b406183f\" (UID: \"9606a239-122e-4814-9048-ab09b406183f\") "
Sep 18 18:18:49.999916 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:49.999899    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v4xcw\" (UniqueName: \"kubernetes.io/projected/9606a239-122e-4814-9048-ab09b406183f-kube-api-access-v4xcw\") pod \"9606a239-122e-4814-9048-ab09b406183f\" (UID: \"9606a239-122e-4814-9048-ab09b406183f\") "
Sep 18 18:18:49.999960 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:49.999947    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rgkxf\" (UniqueName: \"kubernetes.io/projected/33b62f5a-0f4d-4687-b901-05874f328161-kube-api-access-rgkxf\") pod \"33b62f5a-0f4d-4687-b901-05874f328161\" (UID: \"33b62f5a-0f4d-4687-b901-05874f328161\") "
Sep 18 18:18:50.000015 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:49.999994    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33b62f5a-0f4d-4687-b901-05874f328161-plugins-config-volume\") pod \"33b62f5a-0f4d-4687-b901-05874f328161\" (UID: \"33b62f5a-0f4d-4687-b901-05874f328161\") "
Sep 18 18:18:50.000203 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:50.000180    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9606a239-122e-4814-9048-ab09b406183f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9606a239-122e-4814-9048-ab09b406183f" (UID: "9606a239-122e-4814-9048-ab09b406183f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:18:50.000388 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:50.000364    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9606a239-122e-4814-9048-ab09b406183f-plugins-config-volume\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:18:50.000502 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:50.000466    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33b62f5a-0f4d-4687-b901-05874f328161-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "33b62f5a-0f4d-4687-b901-05874f328161" (UID: "33b62f5a-0f4d-4687-b901-05874f328161"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:18:50.002036 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:50.002012    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9606a239-122e-4814-9048-ab09b406183f-kube-api-access-v4xcw" (OuterVolumeSpecName: "kube-api-access-v4xcw") pod "9606a239-122e-4814-9048-ab09b406183f" (UID: "9606a239-122e-4814-9048-ab09b406183f"). InnerVolumeSpecName "kube-api-access-v4xcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:18:50.002180 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:50.002054    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33b62f5a-0f4d-4687-b901-05874f328161-kube-api-access-rgkxf" (OuterVolumeSpecName: "kube-api-access-rgkxf") pod "33b62f5a-0f4d-4687-b901-05874f328161" (UID: "33b62f5a-0f4d-4687-b901-05874f328161"). InnerVolumeSpecName "kube-api-access-rgkxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:18:50.101466 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:50.101388    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a5f114b-5a1f-48bf-800a-f3d70cb2fd5b-metrics-certs\") pod \"6a5f114b-5a1f-48bf-800a-f3d70cb2fd5b\" (UID: \"6a5f114b-5a1f-48bf-800a-f3d70cb2fd5b\") "
Sep 18 18:18:50.101635 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:50.101478    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c4rmr\" (UniqueName: \"kubernetes.io/projected/6a5f114b-5a1f-48bf-800a-f3d70cb2fd5b-kube-api-access-c4rmr\") pod \"6a5f114b-5a1f-48bf-800a-f3d70cb2fd5b\" (UID: \"6a5f114b-5a1f-48bf-800a-f3d70cb2fd5b\") "
Sep 18 18:18:50.101635 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:50.101540    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a5f114b-5a1f-48bf-800a-f3d70cb2fd5b-maas-api-tls\") pod \"6a5f114b-5a1f-48bf-800a-f3d70cb2fd5b\" (UID: \"6a5f114b-5a1f-48bf-800a-f3d70cb2fd5b\") "
Sep 18 18:18:50.101635 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:50.101562    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a5f114b-5a1f-48bf-800a-f3d70cb2fd5b-openshift-service-ca\") pod \"6a5f114b-5a1f-48bf-800a-f3d70cb2fd5b\" (UID: \"6a5f114b-5a1f-48bf-800a-f3d70cb2fd5b\") "
Sep 18 18:18:50.101812 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:50.101751    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33b62f5a-0f4d-4687-b901-05874f328161-plugins-config-volume\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:18:50.101812 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:50.101763    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4xcw\" (UniqueName: \"kubernetes.io/projected/9606a239-122e-4814-9048-ab09b406183f-kube-api-access-v4xcw\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:18:50.101812 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:50.101772    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgkxf\" (UniqueName: \"kubernetes.io/projected/33b62f5a-0f4d-4687-b901-05874f328161-kube-api-access-rgkxf\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:18:50.101996 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:50.101972    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a5f114b-5a1f-48bf-800a-f3d70cb2fd5b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6a5f114b-5a1f-48bf-800a-f3d70cb2fd5b" (UID: "6a5f114b-5a1f-48bf-800a-f3d70cb2fd5b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:18:50.103570 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:50.103537    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a5f114b-5a1f-48bf-800a-f3d70cb2fd5b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6a5f114b-5a1f-48bf-800a-f3d70cb2fd5b" (UID: "6a5f114b-5a1f-48bf-800a-f3d70cb2fd5b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:18:50.103692 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:50.103586    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a5f114b-5a1f-48bf-800a-f3d70cb2fd5b-kube-api-access-c4rmr" (OuterVolumeSpecName: "kube-api-access-c4rmr") pod "6a5f114b-5a1f-48bf-800a-f3d70cb2fd5b" (UID: "6a5f114b-5a1f-48bf-800a-f3d70cb2fd5b"). InnerVolumeSpecName "kube-api-access-c4rmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:18:50.103692 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:50.103632    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a5f114b-5a1f-48bf-800a-f3d70cb2fd5b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6a5f114b-5a1f-48bf-800a-f3d70cb2fd5b" (UID: "6a5f114b-5a1f-48bf-800a-f3d70cb2fd5b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:18:50.138947 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:50.138922    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:50.202767 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:50.202734    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a5f114b-5a1f-48bf-800a-f3d70cb2fd5b-maas-api-tls\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:18:50.202767 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:50.202760    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a5f114b-5a1f-48bf-800a-f3d70cb2fd5b-openshift-service-ca\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:18:50.202978 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:50.202775    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a5f114b-5a1f-48bf-800a-f3d70cb2fd5b-metrics-certs\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:18:50.202978 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:50.202787    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4rmr\" (UniqueName: \"kubernetes.io/projected/6a5f114b-5a1f-48bf-800a-f3d70cb2fd5b-kube-api-access-c4rmr\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:18:50.266191 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:50.266160    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:50.676520 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:50.676489    2561 generic.go:356] "Generic (PLEG): container finished" podID="6a5f114b-5a1f-48bf-800a-f3d70cb2fd5b" containerID="bb894e682b2e13a646d831c151c350404c07c28c9f4e78f8951b591a88ddd10e" exitCode=0
Sep 18 18:18:50.676965 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:50.676541    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-03e512-7878686c45-mw6vc"
Sep 18 18:18:50.676965 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:50.676571    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-03e512-7878686c45-mw6vc" event={"ID":"6a5f114b-5a1f-48bf-800a-f3d70cb2fd5b","Type":"ContainerDied","Data":"bb894e682b2e13a646d831c151c350404c07c28c9f4e78f8951b591a88ddd10e"}
Sep 18 18:18:50.676965 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:50.676619    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-03e512-7878686c45-mw6vc" event={"ID":"6a5f114b-5a1f-48bf-800a-f3d70cb2fd5b","Type":"ContainerDied","Data":"43635579b90fcfbd10db01c5b21243ab09fc99eea9dd3fe5d259e0c4b7b22196"}
Sep 18 18:18:50.676965 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:50.676639    2561 scope.go:117] "RemoveContainer" containerID="bb894e682b2e13a646d831c151c350404c07c28c9f4e78f8951b591a88ddd10e"
Sep 18 18:18:50.678322 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:50.678051    2561 generic.go:356] "Generic (PLEG): container finished" podID="9606a239-122e-4814-9048-ab09b406183f" containerID="dccd6dd8ffe8845a0148f14408e5b413f3bd3399a7c7e028d9293ecc1eb9a2dc" exitCode=0
Sep 18 18:18:50.678322 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:50.678083    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-03e512-85bdf7444c-vwgzw" event={"ID":"9606a239-122e-4814-9048-ab09b406183f","Type":"ContainerDied","Data":"dccd6dd8ffe8845a0148f14408e5b413f3bd3399a7c7e028d9293ecc1eb9a2dc"}
Sep 18 18:18:50.678322 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:50.678141    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-03e512-85bdf7444c-vwgzw" event={"ID":"9606a239-122e-4814-9048-ab09b406183f","Type":"ContainerDied","Data":"7189cc763e45ce09cdb1364c7d07af730b950fbb2107297cbcf435aef7987546"}
Sep 18 18:18:50.678322 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:50.678093    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-03e512-85bdf7444c-vwgzw"
Sep 18 18:18:50.679599 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:50.679575    2561 generic.go:356] "Generic (PLEG): container finished" podID="33b62f5a-0f4d-4687-b901-05874f328161" containerID="86a11957ff99e383ff1f57570a531080a057392a18f2772cbdc70c13e32917ee" exitCode=0
Sep 18 18:18:50.679724 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:50.679616    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-03e512-5bd997d694-gp4gc" event={"ID":"33b62f5a-0f4d-4687-b901-05874f328161","Type":"ContainerDied","Data":"86a11957ff99e383ff1f57570a531080a057392a18f2772cbdc70c13e32917ee"}
Sep 18 18:18:50.679724 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:50.679621    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-03e512-5bd997d694-gp4gc"
Sep 18 18:18:50.679724 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:50.679651    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-03e512-5bd997d694-gp4gc" event={"ID":"33b62f5a-0f4d-4687-b901-05874f328161","Type":"ContainerDied","Data":"6427647716ec7ead848c9a2c5095426826ec276f9ee15056501f60d4b3f1a667"}
Sep 18 18:18:50.688498 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:50.688457    2561 scope.go:117] "RemoveContainer" containerID="bb894e682b2e13a646d831c151c350404c07c28c9f4e78f8951b591a88ddd10e"
Sep 18 18:18:50.688853 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:18:50.688833    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb894e682b2e13a646d831c151c350404c07c28c9f4e78f8951b591a88ddd10e\": container with ID starting with bb894e682b2e13a646d831c151c350404c07c28c9f4e78f8951b591a88ddd10e not found: ID does not exist" containerID="bb894e682b2e13a646d831c151c350404c07c28c9f4e78f8951b591a88ddd10e"
Sep 18 18:18:50.688948 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:50.688859    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb894e682b2e13a646d831c151c350404c07c28c9f4e78f8951b591a88ddd10e"} err="failed to get container status \"bb894e682b2e13a646d831c151c350404c07c28c9f4e78f8951b591a88ddd10e\": rpc error: code = NotFound desc = could not find container \"bb894e682b2e13a646d831c151c350404c07c28c9f4e78f8951b591a88ddd10e\": container with ID starting with bb894e682b2e13a646d831c151c350404c07c28c9f4e78f8951b591a88ddd10e not found: ID does not exist"
Sep 18 18:18:50.688948 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:50.688876    2561 scope.go:117] "RemoveContainer" containerID="dccd6dd8ffe8845a0148f14408e5b413f3bd3399a7c7e028d9293ecc1eb9a2dc"
Sep 18 18:18:50.697541 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:50.697523    2561 scope.go:117] "RemoveContainer" containerID="dccd6dd8ffe8845a0148f14408e5b413f3bd3399a7c7e028d9293ecc1eb9a2dc"
Sep 18 18:18:50.697807 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:18:50.697785    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dccd6dd8ffe8845a0148f14408e5b413f3bd3399a7c7e028d9293ecc1eb9a2dc\": container with ID starting with dccd6dd8ffe8845a0148f14408e5b413f3bd3399a7c7e028d9293ecc1eb9a2dc not found: ID does not exist" containerID="dccd6dd8ffe8845a0148f14408e5b413f3bd3399a7c7e028d9293ecc1eb9a2dc"
Sep 18 18:18:50.697882 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:50.697817    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dccd6dd8ffe8845a0148f14408e5b413f3bd3399a7c7e028d9293ecc1eb9a2dc"} err="failed to get container status \"dccd6dd8ffe8845a0148f14408e5b413f3bd3399a7c7e028d9293ecc1eb9a2dc\": rpc error: code = NotFound desc = could not find container \"dccd6dd8ffe8845a0148f14408e5b413f3bd3399a7c7e028d9293ecc1eb9a2dc\": container with ID starting with dccd6dd8ffe8845a0148f14408e5b413f3bd3399a7c7e028d9293ecc1eb9a2dc not found: ID does not exist"
Sep 18 18:18:50.697882 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:50.697845    2561 scope.go:117] "RemoveContainer" containerID="86a11957ff99e383ff1f57570a531080a057392a18f2772cbdc70c13e32917ee"
Sep 18 18:18:50.703625 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:50.703605    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-03e512-7878686c45-mw6vc"]
Sep 18 18:18:50.705839 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:50.705815    2561 scope.go:117] "RemoveContainer" containerID="86a11957ff99e383ff1f57570a531080a057392a18f2772cbdc70c13e32917ee"
Sep 18 18:18:50.706082 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:18:50.706065    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86a11957ff99e383ff1f57570a531080a057392a18f2772cbdc70c13e32917ee\": container with ID starting with 86a11957ff99e383ff1f57570a531080a057392a18f2772cbdc70c13e32917ee not found: ID does not exist" containerID="86a11957ff99e383ff1f57570a531080a057392a18f2772cbdc70c13e32917ee"
Sep 18 18:18:50.706175 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:50.706087    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86a11957ff99e383ff1f57570a531080a057392a18f2772cbdc70c13e32917ee"} err="failed to get container status \"86a11957ff99e383ff1f57570a531080a057392a18f2772cbdc70c13e32917ee\": rpc error: code = NotFound desc = could not find container \"86a11957ff99e383ff1f57570a531080a057392a18f2772cbdc70c13e32917ee\": container with ID starting with 86a11957ff99e383ff1f57570a531080a057392a18f2772cbdc70c13e32917ee not found: ID does not exist"
Sep 18 18:18:50.710295 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:50.710275    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-03e512-7878686c45-mw6vc"]
Sep 18 18:18:50.721130 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:50.721100    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-03e512-5bd997d694-gp4gc"]
Sep 18 18:18:50.723132 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:50.723096    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-03e512-5bd997d694-gp4gc"]
Sep 18 18:18:50.732233 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:50.732212    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-03e512-85bdf7444c-vwgzw"]
Sep 18 18:18:50.736406 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:50.736387    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-03e512-85bdf7444c-vwgzw"]
Sep 18 18:18:50.953418 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:50.953387    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:51.030786 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:51.030756    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:51.103220 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:51.103182    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:51.199596 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:51.199568    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:51.284462 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:51.284388    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:51.414405 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:51.414373    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:51.665478 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:51.665445    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:51.861687 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:51.861652    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33b62f5a-0f4d-4687-b901-05874f328161" path="/var/lib/kubelet/pods/33b62f5a-0f4d-4687-b901-05874f328161/volumes"
Sep 18 18:18:51.862097 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:51.862082    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a5f114b-5a1f-48bf-800a-f3d70cb2fd5b" path="/var/lib/kubelet/pods/6a5f114b-5a1f-48bf-800a-f3d70cb2fd5b/volumes"
Sep 18 18:18:51.862449 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:51.862436    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9606a239-122e-4814-9048-ab09b406183f" path="/var/lib/kubelet/pods/9606a239-122e-4814-9048-ab09b406183f/volumes"
Sep 18 18:18:52.362790 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:52.362744    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:53.400384 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:53.400345    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-03e512/distinct-w4-03e512-kserve-b5d574f95-t8m5k" podUID="6430cf8a-abc8-4ebb-8c5e-d963fc017ead" containerName="main" containerID="cri-o://f593bb7a57534b554deecca79feb29e2578f576a8aaa2dbed3795f7c6c6a0d5a" gracePeriod=45
Sep 18 18:18:53.425546 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:53.425515    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-03e512/simulator-w4-03e512-kserve-645699d677-9nddq" podUID="0097f764-4e35-4d2d-af6c-8d2a0f496b7e" containerName="main" containerID="cri-o://ed1aa5ae9f0965a7b3f5862be8455ad9726c058bf25f45103772ef7ba12aaadd" gracePeriod=45
Sep 18 18:18:53.460517 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:53.460489    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:53.534747 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:53.534719    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-03e512/distinct-w4-03e512-kserve-b5d574f95-t8m5k" podUID="6430cf8a-abc8-4ebb-8c5e-d963fc017ead" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": dial tcp 10.132.0.72:8000: connect: connection refused"
Sep 18 18:18:53.666424 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:53.666363    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-03e512/simulator-w4-03e512-kserve-645699d677-9nddq"
Sep 18 18:18:53.695268 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:53.695244    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-03e512/distinct-w4-03e512-kserve-b5d574f95-t8m5k"
Sep 18 18:18:53.700699 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:53.700672    2561 generic.go:356] "Generic (PLEG): container finished" podID="0097f764-4e35-4d2d-af6c-8d2a0f496b7e" containerID="ed1aa5ae9f0965a7b3f5862be8455ad9726c058bf25f45103772ef7ba12aaadd" exitCode=0
Sep 18 18:18:53.700803 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:53.700715    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-03e512/simulator-w4-03e512-kserve-645699d677-9nddq"
Sep 18 18:18:53.700803 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:53.700749    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-03e512/simulator-w4-03e512-kserve-645699d677-9nddq" event={"ID":"0097f764-4e35-4d2d-af6c-8d2a0f496b7e","Type":"ContainerDied","Data":"ed1aa5ae9f0965a7b3f5862be8455ad9726c058bf25f45103772ef7ba12aaadd"}
Sep 18 18:18:53.700803 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:53.700791    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-03e512/simulator-w4-03e512-kserve-645699d677-9nddq" event={"ID":"0097f764-4e35-4d2d-af6c-8d2a0f496b7e","Type":"ContainerDied","Data":"32ff466c81cea38349634f6c167e9f4905e9207ed1167b6f85d6b9579bf1a1b9"}
Sep 18 18:18:53.700924 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:53.700813    2561 scope.go:117] "RemoveContainer" containerID="ed1aa5ae9f0965a7b3f5862be8455ad9726c058bf25f45103772ef7ba12aaadd"
Sep 18 18:18:53.702969 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:53.702941    2561 generic.go:356] "Generic (PLEG): container finished" podID="6430cf8a-abc8-4ebb-8c5e-d963fc017ead" containerID="f593bb7a57534b554deecca79feb29e2578f576a8aaa2dbed3795f7c6c6a0d5a" exitCode=0
Sep 18 18:18:53.703078 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:53.703003    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-03e512/distinct-w4-03e512-kserve-b5d574f95-t8m5k" event={"ID":"6430cf8a-abc8-4ebb-8c5e-d963fc017ead","Type":"ContainerDied","Data":"f593bb7a57534b554deecca79feb29e2578f576a8aaa2dbed3795f7c6c6a0d5a"}
Sep 18 18:18:53.703078 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:53.703030    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-03e512/distinct-w4-03e512-kserve-b5d574f95-t8m5k" event={"ID":"6430cf8a-abc8-4ebb-8c5e-d963fc017ead","Type":"ContainerDied","Data":"3824e8e9b2e93f56bb6f49aa4d0ee4856925ed2bb8f59d5f4b617ead31b1cd16"}
Sep 18 18:18:53.703078 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:53.703063    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-03e512/distinct-w4-03e512-kserve-b5d574f95-t8m5k"
Sep 18 18:18:53.712442 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:53.712417    2561 scope.go:117] "RemoveContainer" containerID="ed1aa5ae9f0965a7b3f5862be8455ad9726c058bf25f45103772ef7ba12aaadd"
Sep 18 18:18:53.712807 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:18:53.712780    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed1aa5ae9f0965a7b3f5862be8455ad9726c058bf25f45103772ef7ba12aaadd\": container with ID starting with ed1aa5ae9f0965a7b3f5862be8455ad9726c058bf25f45103772ef7ba12aaadd not found: ID does not exist" containerID="ed1aa5ae9f0965a7b3f5862be8455ad9726c058bf25f45103772ef7ba12aaadd"
Sep 18 18:18:53.712889 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:53.712818    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed1aa5ae9f0965a7b3f5862be8455ad9726c058bf25f45103772ef7ba12aaadd"} err="failed to get container status \"ed1aa5ae9f0965a7b3f5862be8455ad9726c058bf25f45103772ef7ba12aaadd\": rpc error: code = NotFound desc = could not find container \"ed1aa5ae9f0965a7b3f5862be8455ad9726c058bf25f45103772ef7ba12aaadd\": container with ID starting with ed1aa5ae9f0965a7b3f5862be8455ad9726c058bf25f45103772ef7ba12aaadd not found: ID does not exist"
Sep 18 18:18:53.712889 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:53.712841    2561 scope.go:117] "RemoveContainer" containerID="f593bb7a57534b554deecca79feb29e2578f576a8aaa2dbed3795f7c6c6a0d5a"
Sep 18 18:18:53.724165 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:53.724048    2561 scope.go:117] "RemoveContainer" containerID="f593bb7a57534b554deecca79feb29e2578f576a8aaa2dbed3795f7c6c6a0d5a"
Sep 18 18:18:53.724909 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:18:53.724882    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f593bb7a57534b554deecca79feb29e2578f576a8aaa2dbed3795f7c6c6a0d5a\": container with ID starting with f593bb7a57534b554deecca79feb29e2578f576a8aaa2dbed3795f7c6c6a0d5a not found: ID does not exist" containerID="f593bb7a57534b554deecca79feb29e2578f576a8aaa2dbed3795f7c6c6a0d5a"
Sep 18 18:18:53.725015 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:53.724919    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f593bb7a57534b554deecca79feb29e2578f576a8aaa2dbed3795f7c6c6a0d5a"} err="failed to get container status \"f593bb7a57534b554deecca79feb29e2578f576a8aaa2dbed3795f7c6c6a0d5a\": rpc error: code = NotFound desc = could not find container \"f593bb7a57534b554deecca79feb29e2578f576a8aaa2dbed3795f7c6c6a0d5a\": container with ID starting with f593bb7a57534b554deecca79feb29e2578f576a8aaa2dbed3795f7c6c6a0d5a not found: ID does not exist"
Sep 18 18:18:53.842901 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:53.842875    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0097f764-4e35-4d2d-af6c-8d2a0f496b7e-tmp-dir\") pod \"0097f764-4e35-4d2d-af6c-8d2a0f496b7e\" (UID: \"0097f764-4e35-4d2d-af6c-8d2a0f496b7e\") "
Sep 18 18:18:53.842901 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:53.842907    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5qv52\" (UniqueName: \"kubernetes.io/projected/0097f764-4e35-4d2d-af6c-8d2a0f496b7e-kube-api-access-5qv52\") pod \"0097f764-4e35-4d2d-af6c-8d2a0f496b7e\" (UID: \"0097f764-4e35-4d2d-af6c-8d2a0f496b7e\") "
Sep 18 18:18:53.843191 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:53.842944    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6430cf8a-abc8-4ebb-8c5e-d963fc017ead-dshm\") pod \"6430cf8a-abc8-4ebb-8c5e-d963fc017ead\" (UID: \"6430cf8a-abc8-4ebb-8c5e-d963fc017ead\") "
Sep 18 18:18:53.843191 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:53.842984    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0097f764-4e35-4d2d-af6c-8d2a0f496b7e-tls-certs\") pod \"0097f764-4e35-4d2d-af6c-8d2a0f496b7e\" (UID: \"0097f764-4e35-4d2d-af6c-8d2a0f496b7e\") "
Sep 18 18:18:53.843191 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:53.843058    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0097f764-4e35-4d2d-af6c-8d2a0f496b7e-home\") pod \"0097f764-4e35-4d2d-af6c-8d2a0f496b7e\" (UID: \"0097f764-4e35-4d2d-af6c-8d2a0f496b7e\") "
Sep 18 18:18:53.843191 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:53.843083    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0097f764-4e35-4d2d-af6c-8d2a0f496b7e-model-cache\") pod \"0097f764-4e35-4d2d-af6c-8d2a0f496b7e\" (UID: \"0097f764-4e35-4d2d-af6c-8d2a0f496b7e\") "
Sep 18 18:18:53.843191 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:53.843106    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6430cf8a-abc8-4ebb-8c5e-d963fc017ead-model-cache\") pod \"6430cf8a-abc8-4ebb-8c5e-d963fc017ead\" (UID: \"6430cf8a-abc8-4ebb-8c5e-d963fc017ead\") "
Sep 18 18:18:53.843191 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:53.843164    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6430cf8a-abc8-4ebb-8c5e-d963fc017ead-tmp-dir\") pod \"6430cf8a-abc8-4ebb-8c5e-d963fc017ead\" (UID: \"6430cf8a-abc8-4ebb-8c5e-d963fc017ead\") "
Sep 18 18:18:53.843191 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:53.843193    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9qz8k\" (UniqueName: \"kubernetes.io/projected/6430cf8a-abc8-4ebb-8c5e-d963fc017ead-kube-api-access-9qz8k\") pod \"6430cf8a-abc8-4ebb-8c5e-d963fc017ead\" (UID: \"6430cf8a-abc8-4ebb-8c5e-d963fc017ead\") "
Sep 18 18:18:53.843560 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:53.843229    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6430cf8a-abc8-4ebb-8c5e-d963fc017ead-tls-certs\") pod \"6430cf8a-abc8-4ebb-8c5e-d963fc017ead\" (UID: \"6430cf8a-abc8-4ebb-8c5e-d963fc017ead\") "
Sep 18 18:18:53.843560 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:53.843256    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6430cf8a-abc8-4ebb-8c5e-d963fc017ead-home\") pod \"6430cf8a-abc8-4ebb-8c5e-d963fc017ead\" (UID: \"6430cf8a-abc8-4ebb-8c5e-d963fc017ead\") "
Sep 18 18:18:53.843560 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:53.843265    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0097f764-4e35-4d2d-af6c-8d2a0f496b7e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0097f764-4e35-4d2d-af6c-8d2a0f496b7e" (UID: "0097f764-4e35-4d2d-af6c-8d2a0f496b7e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:18:53.843560 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:53.843314    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0097f764-4e35-4d2d-af6c-8d2a0f496b7e-dshm\") pod \"0097f764-4e35-4d2d-af6c-8d2a0f496b7e\" (UID: \"0097f764-4e35-4d2d-af6c-8d2a0f496b7e\") "
Sep 18 18:18:53.843560 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:53.843539    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0097f764-4e35-4d2d-af6c-8d2a0f496b7e-model-cache" (OuterVolumeSpecName: "model-cache") pod "0097f764-4e35-4d2d-af6c-8d2a0f496b7e" (UID: "0097f764-4e35-4d2d-af6c-8d2a0f496b7e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:18:53.843819 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:53.843555    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0097f764-4e35-4d2d-af6c-8d2a0f496b7e-home" (OuterVolumeSpecName: "home") pod "0097f764-4e35-4d2d-af6c-8d2a0f496b7e" (UID: "0097f764-4e35-4d2d-af6c-8d2a0f496b7e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:18:53.843819 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:53.843614    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6430cf8a-abc8-4ebb-8c5e-d963fc017ead-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6430cf8a-abc8-4ebb-8c5e-d963fc017ead" (UID: "6430cf8a-abc8-4ebb-8c5e-d963fc017ead"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:18:53.843819 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:53.843736    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0097f764-4e35-4d2d-af6c-8d2a0f496b7e-tmp-dir\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:18:53.843819 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:53.843810    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6430cf8a-abc8-4ebb-8c5e-d963fc017ead-model-cache" (OuterVolumeSpecName: "model-cache") pod "6430cf8a-abc8-4ebb-8c5e-d963fc017ead" (UID: "6430cf8a-abc8-4ebb-8c5e-d963fc017ead"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:18:53.844038 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:53.843756    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0097f764-4e35-4d2d-af6c-8d2a0f496b7e-home\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:18:53.844038 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:53.843957    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0097f764-4e35-4d2d-af6c-8d2a0f496b7e-model-cache\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:18:53.844038 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:53.843971    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6430cf8a-abc8-4ebb-8c5e-d963fc017ead-tmp-dir\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:18:53.844038 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:53.843993    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6430cf8a-abc8-4ebb-8c5e-d963fc017ead-home" (OuterVolumeSpecName: "home") pod "6430cf8a-abc8-4ebb-8c5e-d963fc017ead" (UID: "6430cf8a-abc8-4ebb-8c5e-d963fc017ead"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:18:53.845891 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:53.845866    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0097f764-4e35-4d2d-af6c-8d2a0f496b7e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0097f764-4e35-4d2d-af6c-8d2a0f496b7e" (UID: "0097f764-4e35-4d2d-af6c-8d2a0f496b7e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:18:53.845998 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:53.845923    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6430cf8a-abc8-4ebb-8c5e-d963fc017ead-kube-api-access-9qz8k" (OuterVolumeSpecName: "kube-api-access-9qz8k") pod "6430cf8a-abc8-4ebb-8c5e-d963fc017ead" (UID: "6430cf8a-abc8-4ebb-8c5e-d963fc017ead"). InnerVolumeSpecName "kube-api-access-9qz8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:18:53.845998 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:53.845960    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6430cf8a-abc8-4ebb-8c5e-d963fc017ead-dshm" (OuterVolumeSpecName: "dshm") pod "6430cf8a-abc8-4ebb-8c5e-d963fc017ead" (UID: "6430cf8a-abc8-4ebb-8c5e-d963fc017ead"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:18:53.845998 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:53.845984    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6430cf8a-abc8-4ebb-8c5e-d963fc017ead-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6430cf8a-abc8-4ebb-8c5e-d963fc017ead" (UID: "6430cf8a-abc8-4ebb-8c5e-d963fc017ead"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:18:53.846111 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:53.846006    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0097f764-4e35-4d2d-af6c-8d2a0f496b7e-dshm" (OuterVolumeSpecName: "dshm") pod "0097f764-4e35-4d2d-af6c-8d2a0f496b7e" (UID: "0097f764-4e35-4d2d-af6c-8d2a0f496b7e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:18:53.846111 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:53.846030    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0097f764-4e35-4d2d-af6c-8d2a0f496b7e-kube-api-access-5qv52" (OuterVolumeSpecName: "kube-api-access-5qv52") pod "0097f764-4e35-4d2d-af6c-8d2a0f496b7e" (UID: "0097f764-4e35-4d2d-af6c-8d2a0f496b7e"). InnerVolumeSpecName "kube-api-access-5qv52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:18:53.945178 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:53.945087    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0097f764-4e35-4d2d-af6c-8d2a0f496b7e-dshm\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:18:53.945178 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:53.945133    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qv52\" (UniqueName: \"kubernetes.io/projected/0097f764-4e35-4d2d-af6c-8d2a0f496b7e-kube-api-access-5qv52\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:18:53.945178 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:53.945144    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6430cf8a-abc8-4ebb-8c5e-d963fc017ead-dshm\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:18:53.945178 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:53.945152    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0097f764-4e35-4d2d-af6c-8d2a0f496b7e-tls-certs\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:18:53.945178 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:53.945160    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6430cf8a-abc8-4ebb-8c5e-d963fc017ead-model-cache\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:18:53.945178 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:53.945168    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qz8k\" (UniqueName: \"kubernetes.io/projected/6430cf8a-abc8-4ebb-8c5e-d963fc017ead-kube-api-access-9qz8k\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:18:53.945178 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:53.945178    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6430cf8a-abc8-4ebb-8c5e-d963fc017ead-tls-certs\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:18:53.945502 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:53.945191    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6430cf8a-abc8-4ebb-8c5e-d963fc017ead-home\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:18:54.019492 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:54.019461    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-03e512/simulator-w4-03e512-kserve-645699d677-9nddq"]
Sep 18 18:18:54.025074 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:54.025051    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-03e512/simulator-w4-03e512-kserve-645699d677-9nddq"]
Sep 18 18:18:54.042321 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:54.042297    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-03e512/distinct-w4-03e512-kserve-b5d574f95-t8m5k"]
Sep 18 18:18:54.050998 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:54.050971    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-03e512/distinct-w4-03e512-kserve-b5d574f95-t8m5k"]
Sep 18 18:18:54.432066 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:54.432034    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:55.228130 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:55.228091    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:55.863469 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:55.863440    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0097f764-4e35-4d2d-af6c-8d2a0f496b7e" path="/var/lib/kubelet/pods/0097f764-4e35-4d2d-af6c-8d2a0f496b7e/volumes"
Sep 18 18:18:55.863825 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:55.863811    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6430cf8a-abc8-4ebb-8c5e-d963fc017ead" path="/var/lib/kubelet/pods/6430cf8a-abc8-4ebb-8c5e-d963fc017ead/volumes"
Sep 18 18:18:56.045144 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:56.045099    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:56.482351 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:56.482313    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:56.583064 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:56.583033    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:56.708799 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:56.708771    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:57.769700 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:57.769674    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:57.969281 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:57.969253    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:58.070290 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:58.070212    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:58.231205 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:58.231159    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:58.856131 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:58.856091    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:18:59.754919 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:18:59.754887    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:00.521491 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:00.521461    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:01.299521 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:01.299483    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:01.400666 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:01.400632    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:02.108411 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:02.108383    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:02.406994 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:02.406957    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7602-8a7deb"]
Sep 18 18:19:02.407398 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:02.407383    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9606a239-122e-4814-9048-ab09b406183f" containerName="payload-processing"
Sep 18 18:19:02.407467 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:02.407400    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9606a239-122e-4814-9048-ab09b406183f" containerName="payload-processing"
Sep 18 18:19:02.407467 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:02.407409    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0097f764-4e35-4d2d-af6c-8d2a0f496b7e" containerName="main"
Sep 18 18:19:02.407467 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:02.407415    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0097f764-4e35-4d2d-af6c-8d2a0f496b7e" containerName="main"
Sep 18 18:19:02.407467 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:02.407431    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6430cf8a-abc8-4ebb-8c5e-d963fc017ead" containerName="main"
Sep 18 18:19:02.407467 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:02.407437    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6430cf8a-abc8-4ebb-8c5e-d963fc017ead" containerName="main"
Sep 18 18:19:02.407467 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:02.407445    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33b62f5a-0f4d-4687-b901-05874f328161" containerName="payload-pre-processing"
Sep 18 18:19:02.407467 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:02.407451    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="33b62f5a-0f4d-4687-b901-05874f328161" containerName="payload-pre-processing"
Sep 18 18:19:02.407467 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:02.407457    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50f1cb39-8933-498e-9789-9d4bfa0416f7" containerName="payload-processing"
Sep 18 18:19:02.407467 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:02.407463    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="50f1cb39-8933-498e-9789-9d4bfa0416f7" containerName="payload-processing"
Sep 18 18:19:02.407728 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:02.407472    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a5f114b-5a1f-48bf-800a-f3d70cb2fd5b" containerName="maas-api"
Sep 18 18:19:02.407728 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:02.407478    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a5f114b-5a1f-48bf-800a-f3d70cb2fd5b" containerName="maas-api"
Sep 18 18:19:02.407728 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:02.407535    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="33b62f5a-0f4d-4687-b901-05874f328161" containerName="payload-pre-processing"
Sep 18 18:19:02.407728 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:02.407544    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0097f764-4e35-4d2d-af6c-8d2a0f496b7e" containerName="main"
Sep 18 18:19:02.407728 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:02.407554    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="50f1cb39-8933-498e-9789-9d4bfa0416f7" containerName="payload-processing"
Sep 18 18:19:02.407728 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:02.407559    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a5f114b-5a1f-48bf-800a-f3d70cb2fd5b" containerName="maas-api"
Sep 18 18:19:02.407728 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:02.407566    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9606a239-122e-4814-9048-ab09b406183f" containerName="payload-processing"
Sep 18 18:19:02.407728 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:02.407571    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="6430cf8a-abc8-4ebb-8c5e-d963fc017ead" containerName="main"
Sep 18 18:19:02.410489 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:02.410473    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7602-8a7deb"
Sep 18 18:19:02.412858 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:02.412832    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-fz228\""
Sep 18 18:19:02.422984 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:02.422961    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7602-8a7deb"]
Sep 18 18:19:02.423492 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:02.423471    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdbl7\" (UniqueName: \"kubernetes.io/projected/6149bf52-4d94-4edf-a2dd-7071b0fe91c5-kube-api-access-vdbl7\") pod \"test-curl-7602-8a7deb\" (UID: \"6149bf52-4d94-4edf-a2dd-7071b0fe91c5\") " pod="opendatahub/test-curl-7602-8a7deb"
Sep 18 18:19:02.524181 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:02.524147    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdbl7\" (UniqueName: \"kubernetes.io/projected/6149bf52-4d94-4edf-a2dd-7071b0fe91c5-kube-api-access-vdbl7\") pod \"test-curl-7602-8a7deb\" (UID: \"6149bf52-4d94-4edf-a2dd-7071b0fe91c5\") " pod="opendatahub/test-curl-7602-8a7deb"
Sep 18 18:19:02.535838 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:02.535811    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdbl7\" (UniqueName: \"kubernetes.io/projected/6149bf52-4d94-4edf-a2dd-7071b0fe91c5-kube-api-access-vdbl7\") pod \"test-curl-7602-8a7deb\" (UID: \"6149bf52-4d94-4edf-a2dd-7071b0fe91c5\") " pod="opendatahub/test-curl-7602-8a7deb"
Sep 18 18:19:02.719901 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:02.719820    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7602-8a7deb"
Sep 18 18:19:02.842306 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:02.842278    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:02.996695 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:02.996671    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7602-8a7deb"]
Sep 18 18:19:02.998938 ip-10-0-140-54 kubenswrapper[2561]: W0918 18:19:02.998907    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6149bf52_4d94_4edf_a2dd_7071b0fe91c5.slice/crio-3deb9eeeb69862035272dd21cb18f04c2c9874ab8845e8c2ced8e36b68c40558 WatchSource:0}: Error finding container 3deb9eeeb69862035272dd21cb18f04c2c9874ab8845e8c2ced8e36b68c40558: Status 404 returned error can't find the container with id 3deb9eeeb69862035272dd21cb18f04c2c9874ab8845e8c2ced8e36b68c40558
Sep 18 18:19:03.757984 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:03.757947    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7602-8a7deb" event={"ID":"6149bf52-4d94-4edf-a2dd-7071b0fe91c5","Type":"ContainerStarted","Data":"ee4aca18d98732ecd612396b45ee767915d5742d5522bd0d7345e9476085dc20"}
Sep 18 18:19:03.757984 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:03.757988    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7602-8a7deb" event={"ID":"6149bf52-4d94-4edf-a2dd-7071b0fe91c5","Type":"ContainerStarted","Data":"3deb9eeeb69862035272dd21cb18f04c2c9874ab8845e8c2ced8e36b68c40558"}
Sep 18 18:19:03.772367 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:03.772290    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7602-8a7deb" podStartSLOduration=1.7722710990000001 podStartE2EDuration="1.772271099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:19:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:19:03.771438247 +0000 UTC m=+1748.398569890" watchObservedRunningTime="2026-09-18 18:19:03.772271099 +0000 UTC m=+1748.399402719"
Sep 18 18:19:04.338369 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:04.338329    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7602-8a7deb"]
Sep 18 18:19:04.343430 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:04.343401    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7602-8a7deb"]
Sep 18 18:19:04.763247 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:19:04.763216    2561 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7602-8a7deb" container="test-curl-7602-8a7deb"
Sep 18 18:19:05.768231 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:05.768192    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7602-8a7deb" podUID="6149bf52-4d94-4edf-a2dd-7071b0fe91c5" containerName="test-curl-7602-8a7deb" containerID="cri-o://ee4aca18d98732ecd612396b45ee767915d5742d5522bd0d7345e9476085dc20" gracePeriod=2
Sep 18 18:19:05.770244 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:19:05.770215    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7602-8a7deb\" is forbidden: User \"system:node:ip-10-0-140-54.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-54.ec2.internal' and this object" podUID="6149bf52-4d94-4edf-a2dd-7071b0fe91c5" pod="opendatahub/test-curl-7602-8a7deb"
Sep 18 18:19:05.863972 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:19:05.863930    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7602-8a7deb\" is forbidden: User \"system:node:ip-10-0-140-54.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-54.ec2.internal' and this object" podUID="6149bf52-4d94-4edf-a2dd-7071b0fe91c5" pod="opendatahub/test-curl-7602-8a7deb"
Sep 18 18:19:06.071932 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:06.071840    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-53cde3/auth-test-model-53cde3-kserve-75dcb68b95-d4j87"]
Sep 18 18:19:06.090810 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:06.090770    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:07.942889 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:07.942857    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-341bd4/auth-test-model-341bd4-kserve-5c4967bb4c-mqf6c"]
Sep 18 18:19:07.954781 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:07.954758    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7602-8a7deb"
Sep 18 18:19:07.961494 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:07.961472    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:07.971663 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:07.971644    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vdbl7\" (UniqueName: \"kubernetes.io/projected/6149bf52-4d94-4edf-a2dd-7071b0fe91c5-kube-api-access-vdbl7\") pod \"6149bf52-4d94-4edf-a2dd-7071b0fe91c5\" (UID: \"6149bf52-4d94-4edf-a2dd-7071b0fe91c5\") "
Sep 18 18:19:07.973471 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:07.973448    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6149bf52-4d94-4edf-a2dd-7071b0fe91c5-kube-api-access-vdbl7" (OuterVolumeSpecName: "kube-api-access-vdbl7") pod "6149bf52-4d94-4edf-a2dd-7071b0fe91c5" (UID: "6149bf52-4d94-4edf-a2dd-7071b0fe91c5"). InnerVolumeSpecName "kube-api-access-vdbl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:19:08.073250 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:08.073173    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdbl7\" (UniqueName: \"kubernetes.io/projected/6149bf52-4d94-4edf-a2dd-7071b0fe91c5-kube-api-access-vdbl7\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:19:08.785364 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:08.785328    2561 generic.go:356] "Generic (PLEG): container finished" podID="6149bf52-4d94-4edf-a2dd-7071b0fe91c5" containerID="ee4aca18d98732ecd612396b45ee767915d5742d5522bd0d7345e9476085dc20" exitCode=137
Sep 18 18:19:08.785364 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:08.785368    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7602-8a7deb"
Sep 18 18:19:08.785598 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:08.785405    2561 scope.go:117] "RemoveContainer" containerID="ee4aca18d98732ecd612396b45ee767915d5742d5522bd0d7345e9476085dc20"
Sep 18 18:19:08.795057 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:08.795035    2561 scope.go:117] "RemoveContainer" containerID="ee4aca18d98732ecd612396b45ee767915d5742d5522bd0d7345e9476085dc20"
Sep 18 18:19:08.795356 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:19:08.795335    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee4aca18d98732ecd612396b45ee767915d5742d5522bd0d7345e9476085dc20\": container with ID starting with ee4aca18d98732ecd612396b45ee767915d5742d5522bd0d7345e9476085dc20 not found: ID does not exist" containerID="ee4aca18d98732ecd612396b45ee767915d5742d5522bd0d7345e9476085dc20"
Sep 18 18:19:08.795424 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:08.795364    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee4aca18d98732ecd612396b45ee767915d5742d5522bd0d7345e9476085dc20"} err="failed to get container status \"ee4aca18d98732ecd612396b45ee767915d5742d5522bd0d7345e9476085dc20\": rpc error: code = NotFound desc = could not find container \"ee4aca18d98732ecd612396b45ee767915d5742d5522bd0d7345e9476085dc20\": container with ID starting with ee4aca18d98732ecd612396b45ee767915d5742d5522bd0d7345e9476085dc20 not found: ID does not exist"
Sep 18 18:19:08.796286 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:19:08.796261    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7602-8a7deb\" is forbidden: User \"system:node:ip-10-0-140-54.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-54.ec2.internal' and this object" podUID="6149bf52-4d94-4edf-a2dd-7071b0fe91c5" pod="opendatahub/test-curl-7602-8a7deb"
Sep 18 18:19:09.023430 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:09.023400    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:09.123088 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:09.122986    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:09.214205 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:09.214175    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:09.270647 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:09.270620    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:09.586497 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:09.586465    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:09.674086 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:09.674051    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:09.861356 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:09.861278    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6149bf52-4d94-4edf-a2dd-7071b0fe91c5" path="/var/lib/kubelet/pods/6149bf52-4d94-4edf-a2dd-7071b0fe91c5/volumes"
Sep 18 18:19:10.574414 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:10.574383    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:11.217272 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:11.217244    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:11.924301 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:11.924271    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:16.867983 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:16.867937    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:16.996160 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:16.996129    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:17.325473 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:17.325444    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:17.441283 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:17.441250    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:17.949871 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:17.949833    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:18.585793 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:18.585759    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:19.652669 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:19.652614    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:20.638694 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:20.638666    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:21.091273 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:21.091231    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-53cde3/auth-test-model-53cde3-kserve-75dcb68b95-d4j87" podUID="536e0d16-7967-4753-9123-f335ac1a763f" containerName="main" containerID="cri-o://b2f1ceebf586d1ed6a1f20f1d98bb4d930cb95dc5900c516e77993bd1f0832da" gracePeriod=45
Sep 18 18:19:21.353865 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:21.353797    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-53cde3/auth-test-model-53cde3-kserve-75dcb68b95-d4j87"
Sep 18 18:19:21.499584 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:21.499548    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/536e0d16-7967-4753-9123-f335ac1a763f-model-cache\") pod \"536e0d16-7967-4753-9123-f335ac1a763f\" (UID: \"536e0d16-7967-4753-9123-f335ac1a763f\") "
Sep 18 18:19:21.499784 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:21.499606    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/536e0d16-7967-4753-9123-f335ac1a763f-home\") pod \"536e0d16-7967-4753-9123-f335ac1a763f\" (UID: \"536e0d16-7967-4753-9123-f335ac1a763f\") "
Sep 18 18:19:21.499784 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:21.499660    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/536e0d16-7967-4753-9123-f335ac1a763f-tls-certs\") pod \"536e0d16-7967-4753-9123-f335ac1a763f\" (UID: \"536e0d16-7967-4753-9123-f335ac1a763f\") "
Sep 18 18:19:21.499784 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:21.499685    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b6x4k\" (UniqueName: \"kubernetes.io/projected/536e0d16-7967-4753-9123-f335ac1a763f-kube-api-access-b6x4k\") pod \"536e0d16-7967-4753-9123-f335ac1a763f\" (UID: \"536e0d16-7967-4753-9123-f335ac1a763f\") "
Sep 18 18:19:21.499784 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:21.499721    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/536e0d16-7967-4753-9123-f335ac1a763f-dshm\") pod \"536e0d16-7967-4753-9123-f335ac1a763f\" (UID: \"536e0d16-7967-4753-9123-f335ac1a763f\") "
Sep 18 18:19:21.500002 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:21.499779    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/536e0d16-7967-4753-9123-f335ac1a763f-model-cache" (OuterVolumeSpecName: "model-cache") pod "536e0d16-7967-4753-9123-f335ac1a763f" (UID: "536e0d16-7967-4753-9123-f335ac1a763f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:19:21.500002 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:21.499860    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/536e0d16-7967-4753-9123-f335ac1a763f-tmp-dir\") pod \"536e0d16-7967-4753-9123-f335ac1a763f\" (UID: \"536e0d16-7967-4753-9123-f335ac1a763f\") "
Sep 18 18:19:21.500002 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:21.499928    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/536e0d16-7967-4753-9123-f335ac1a763f-home" (OuterVolumeSpecName: "home") pod "536e0d16-7967-4753-9123-f335ac1a763f" (UID: "536e0d16-7967-4753-9123-f335ac1a763f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:19:21.500205 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:21.500087    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/536e0d16-7967-4753-9123-f335ac1a763f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "536e0d16-7967-4753-9123-f335ac1a763f" (UID: "536e0d16-7967-4753-9123-f335ac1a763f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:19:21.500262 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:21.500242    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/536e0d16-7967-4753-9123-f335ac1a763f-tmp-dir\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:19:21.500262 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:21.500260    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/536e0d16-7967-4753-9123-f335ac1a763f-model-cache\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:19:21.500336 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:21.500270    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/536e0d16-7967-4753-9123-f335ac1a763f-home\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:19:21.501976 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:21.501947    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/536e0d16-7967-4753-9123-f335ac1a763f-dshm" (OuterVolumeSpecName: "dshm") pod "536e0d16-7967-4753-9123-f335ac1a763f" (UID: "536e0d16-7967-4753-9123-f335ac1a763f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:19:21.502085 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:21.502013    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/536e0d16-7967-4753-9123-f335ac1a763f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "536e0d16-7967-4753-9123-f335ac1a763f" (UID: "536e0d16-7967-4753-9123-f335ac1a763f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:19:21.502085 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:21.502027    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/536e0d16-7967-4753-9123-f335ac1a763f-kube-api-access-b6x4k" (OuterVolumeSpecName: "kube-api-access-b6x4k") pod "536e0d16-7967-4753-9123-f335ac1a763f" (UID: "536e0d16-7967-4753-9123-f335ac1a763f"). InnerVolumeSpecName "kube-api-access-b6x4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:19:21.601268 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:21.601233    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/536e0d16-7967-4753-9123-f335ac1a763f-tls-certs\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:19:21.601268 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:21.601262    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6x4k\" (UniqueName: \"kubernetes.io/projected/536e0d16-7967-4753-9123-f335ac1a763f-kube-api-access-b6x4k\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:19:21.601268 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:21.601270    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/536e0d16-7967-4753-9123-f335ac1a763f-dshm\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:19:21.695291 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:21.695244    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:21.851759 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:21.851723    2561 generic.go:356] "Generic (PLEG): container finished" podID="536e0d16-7967-4753-9123-f335ac1a763f" containerID="b2f1ceebf586d1ed6a1f20f1d98bb4d930cb95dc5900c516e77993bd1f0832da" exitCode=0
Sep 18 18:19:21.851932 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:21.851767    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-53cde3/auth-test-model-53cde3-kserve-75dcb68b95-d4j87"
Sep 18 18:19:21.851932 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:21.851821    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-53cde3/auth-test-model-53cde3-kserve-75dcb68b95-d4j87" event={"ID":"536e0d16-7967-4753-9123-f335ac1a763f","Type":"ContainerDied","Data":"b2f1ceebf586d1ed6a1f20f1d98bb4d930cb95dc5900c516e77993bd1f0832da"}
Sep 18 18:19:21.851932 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:21.851858    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-53cde3/auth-test-model-53cde3-kserve-75dcb68b95-d4j87" event={"ID":"536e0d16-7967-4753-9123-f335ac1a763f","Type":"ContainerDied","Data":"12ed042b70573d9110e54fd80311859b1e3738907eb2bf2a8303de43c7804c89"}
Sep 18 18:19:21.851932 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:21.851880    2561 scope.go:117] "RemoveContainer" containerID="b2f1ceebf586d1ed6a1f20f1d98bb4d930cb95dc5900c516e77993bd1f0832da"
Sep 18 18:19:21.861409 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:21.861387    2561 scope.go:117] "RemoveContainer" containerID="b2f1ceebf586d1ed6a1f20f1d98bb4d930cb95dc5900c516e77993bd1f0832da"
Sep 18 18:19:21.861658 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:19:21.861639    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2f1ceebf586d1ed6a1f20f1d98bb4d930cb95dc5900c516e77993bd1f0832da\": container with ID starting with b2f1ceebf586d1ed6a1f20f1d98bb4d930cb95dc5900c516e77993bd1f0832da not found: ID does not exist" containerID="b2f1ceebf586d1ed6a1f20f1d98bb4d930cb95dc5900c516e77993bd1f0832da"
Sep 18 18:19:21.861715 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:21.861666    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2f1ceebf586d1ed6a1f20f1d98bb4d930cb95dc5900c516e77993bd1f0832da"} err="failed to get container status \"b2f1ceebf586d1ed6a1f20f1d98bb4d930cb95dc5900c516e77993bd1f0832da\": rpc error: code = NotFound desc = could not find container \"b2f1ceebf586d1ed6a1f20f1d98bb4d930cb95dc5900c516e77993bd1f0832da\": container with ID starting with b2f1ceebf586d1ed6a1f20f1d98bb4d930cb95dc5900c516e77993bd1f0832da not found: ID does not exist"
Sep 18 18:19:21.885970 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:21.885947    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-53cde3/auth-test-model-53cde3-kserve-75dcb68b95-d4j87"]
Sep 18 18:19:21.891442 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:21.891420    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-53cde3/auth-test-model-53cde3-kserve-75dcb68b95-d4j87"]
Sep 18 18:19:22.963278 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:22.963241    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-341bd4/auth-test-model-341bd4-kserve-5c4967bb4c-mqf6c" podUID="0864e158-2933-4867-b3b9-8ef5d9749ad5" containerName="main" containerID="cri-o://58675b48e308be18cbe39aa419a9f1837d5a66f965b4c61e316d21eb84b29766" gracePeriod=45
Sep 18 18:19:23.204396 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:23.204370    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-341bd4/auth-test-model-341bd4-kserve-5c4967bb4c-mqf6c"
Sep 18 18:19:23.319172 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:23.319092    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0864e158-2933-4867-b3b9-8ef5d9749ad5-dshm\") pod \"0864e158-2933-4867-b3b9-8ef5d9749ad5\" (UID: \"0864e158-2933-4867-b3b9-8ef5d9749ad5\") "
Sep 18 18:19:23.319172 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:23.319158    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0864e158-2933-4867-b3b9-8ef5d9749ad5-model-cache\") pod \"0864e158-2933-4867-b3b9-8ef5d9749ad5\" (UID: \"0864e158-2933-4867-b3b9-8ef5d9749ad5\") "
Sep 18 18:19:23.319410 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:23.319207    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0864e158-2933-4867-b3b9-8ef5d9749ad5-home\") pod \"0864e158-2933-4867-b3b9-8ef5d9749ad5\" (UID: \"0864e158-2933-4867-b3b9-8ef5d9749ad5\") "
Sep 18 18:19:23.319410 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:23.319264    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0864e158-2933-4867-b3b9-8ef5d9749ad5-tmp-dir\") pod \"0864e158-2933-4867-b3b9-8ef5d9749ad5\" (UID: \"0864e158-2933-4867-b3b9-8ef5d9749ad5\") "
Sep 18 18:19:23.319410 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:23.319297    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cmrwf\" (UniqueName: \"kubernetes.io/projected/0864e158-2933-4867-b3b9-8ef5d9749ad5-kube-api-access-cmrwf\") pod \"0864e158-2933-4867-b3b9-8ef5d9749ad5\" (UID: \"0864e158-2933-4867-b3b9-8ef5d9749ad5\") "
Sep 18 18:19:23.319410 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:23.319316    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0864e158-2933-4867-b3b9-8ef5d9749ad5-tls-certs\") pod \"0864e158-2933-4867-b3b9-8ef5d9749ad5\" (UID: \"0864e158-2933-4867-b3b9-8ef5d9749ad5\") "
Sep 18 18:19:23.319614 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:23.319484    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0864e158-2933-4867-b3b9-8ef5d9749ad5-model-cache" (OuterVolumeSpecName: "model-cache") pod "0864e158-2933-4867-b3b9-8ef5d9749ad5" (UID: "0864e158-2933-4867-b3b9-8ef5d9749ad5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:19:23.319614 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:23.319523    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0864e158-2933-4867-b3b9-8ef5d9749ad5-home" (OuterVolumeSpecName: "home") pod "0864e158-2933-4867-b3b9-8ef5d9749ad5" (UID: "0864e158-2933-4867-b3b9-8ef5d9749ad5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:19:23.319614 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:23.319583    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0864e158-2933-4867-b3b9-8ef5d9749ad5-model-cache\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:19:23.319711 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:23.319610    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0864e158-2933-4867-b3b9-8ef5d9749ad5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0864e158-2933-4867-b3b9-8ef5d9749ad5" (UID: "0864e158-2933-4867-b3b9-8ef5d9749ad5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:19:23.321397 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:23.321375    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0864e158-2933-4867-b3b9-8ef5d9749ad5-dshm" (OuterVolumeSpecName: "dshm") pod "0864e158-2933-4867-b3b9-8ef5d9749ad5" (UID: "0864e158-2933-4867-b3b9-8ef5d9749ad5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:19:23.321512 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:23.321494    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0864e158-2933-4867-b3b9-8ef5d9749ad5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0864e158-2933-4867-b3b9-8ef5d9749ad5" (UID: "0864e158-2933-4867-b3b9-8ef5d9749ad5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:19:23.321557 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:23.321513    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0864e158-2933-4867-b3b9-8ef5d9749ad5-kube-api-access-cmrwf" (OuterVolumeSpecName: "kube-api-access-cmrwf") pod "0864e158-2933-4867-b3b9-8ef5d9749ad5" (UID: "0864e158-2933-4867-b3b9-8ef5d9749ad5"). InnerVolumeSpecName "kube-api-access-cmrwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:19:23.420999 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:23.420970    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0864e158-2933-4867-b3b9-8ef5d9749ad5-home\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:19:23.420999 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:23.420994    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0864e158-2933-4867-b3b9-8ef5d9749ad5-tmp-dir\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:19:23.420999 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:23.421003    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmrwf\" (UniqueName: \"kubernetes.io/projected/0864e158-2933-4867-b3b9-8ef5d9749ad5-kube-api-access-cmrwf\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:19:23.421214 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:23.421013    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0864e158-2933-4867-b3b9-8ef5d9749ad5-tls-certs\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:19:23.421214 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:23.421021    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0864e158-2933-4867-b3b9-8ef5d9749ad5-dshm\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:19:23.862626 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:23.862584    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="536e0d16-7967-4753-9123-f335ac1a763f" path="/var/lib/kubelet/pods/536e0d16-7967-4753-9123-f335ac1a763f/volumes"
Sep 18 18:19:23.864144 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:23.864104    2561 generic.go:356] "Generic (PLEG): container finished" podID="0864e158-2933-4867-b3b9-8ef5d9749ad5" containerID="58675b48e308be18cbe39aa419a9f1837d5a66f965b4c61e316d21eb84b29766" exitCode=0
Sep 18 18:19:23.864310 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:23.864154    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-341bd4/auth-test-model-341bd4-kserve-5c4967bb4c-mqf6c"
Sep 18 18:19:23.864310 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:23.864187    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-341bd4/auth-test-model-341bd4-kserve-5c4967bb4c-mqf6c" event={"ID":"0864e158-2933-4867-b3b9-8ef5d9749ad5","Type":"ContainerDied","Data":"58675b48e308be18cbe39aa419a9f1837d5a66f965b4c61e316d21eb84b29766"}
Sep 18 18:19:23.864310 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:23.864223    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-341bd4/auth-test-model-341bd4-kserve-5c4967bb4c-mqf6c" event={"ID":"0864e158-2933-4867-b3b9-8ef5d9749ad5","Type":"ContainerDied","Data":"d66a1c44bb198a763d61dd2b08b26645b387c76522cd6ede96c857204560082b"}
Sep 18 18:19:23.864310 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:23.864240    2561 scope.go:117] "RemoveContainer" containerID="58675b48e308be18cbe39aa419a9f1837d5a66f965b4c61e316d21eb84b29766"
Sep 18 18:19:23.873521 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:23.873501    2561 scope.go:117] "RemoveContainer" containerID="58675b48e308be18cbe39aa419a9f1837d5a66f965b4c61e316d21eb84b29766"
Sep 18 18:19:23.873830 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:19:23.873796    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58675b48e308be18cbe39aa419a9f1837d5a66f965b4c61e316d21eb84b29766\": container with ID starting with 58675b48e308be18cbe39aa419a9f1837d5a66f965b4c61e316d21eb84b29766 not found: ID does not exist" containerID="58675b48e308be18cbe39aa419a9f1837d5a66f965b4c61e316d21eb84b29766"
Sep 18 18:19:23.873958 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:23.873837    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58675b48e308be18cbe39aa419a9f1837d5a66f965b4c61e316d21eb84b29766"} err="failed to get container status \"58675b48e308be18cbe39aa419a9f1837d5a66f965b4c61e316d21eb84b29766\": rpc error: code = NotFound desc = could not find container \"58675b48e308be18cbe39aa419a9f1837d5a66f965b4c61e316d21eb84b29766\": container with ID starting with 58675b48e308be18cbe39aa419a9f1837d5a66f965b4c61e316d21eb84b29766 not found: ID does not exist"
Sep 18 18:19:23.886208 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:23.886187    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-341bd4/auth-test-model-341bd4-kserve-5c4967bb4c-mqf6c"]
Sep 18 18:19:23.891755 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:23.891736    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-341bd4/auth-test-model-341bd4-kserve-5c4967bb4c-mqf6c"]
Sep 18 18:19:24.014997 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:24.014964    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-53cde3/sub-test-model-53cde3-kserve-5bfb84c797-qs67n"]
Sep 18 18:19:24.015700 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:24.015681    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6149bf52-4d94-4edf-a2dd-7071b0fe91c5" containerName="test-curl-7602-8a7deb"
Sep 18 18:19:24.015753 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:24.015704    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6149bf52-4d94-4edf-a2dd-7071b0fe91c5" containerName="test-curl-7602-8a7deb"
Sep 18 18:19:24.015753 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:24.015728    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0864e158-2933-4867-b3b9-8ef5d9749ad5" containerName="main"
Sep 18 18:19:24.015753 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:24.015736    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0864e158-2933-4867-b3b9-8ef5d9749ad5" containerName="main"
Sep 18 18:19:24.015847 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:24.015769    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="536e0d16-7967-4753-9123-f335ac1a763f" containerName="main"
Sep 18 18:19:24.015847 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:24.015779    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="536e0d16-7967-4753-9123-f335ac1a763f" containerName="main"
Sep 18 18:19:24.015912 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:24.015866    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0864e158-2933-4867-b3b9-8ef5d9749ad5" containerName="main"
Sep 18 18:19:24.015912 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:24.015879    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="536e0d16-7967-4753-9123-f335ac1a763f" containerName="main"
Sep 18 18:19:24.015912 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:24.015890    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="6149bf52-4d94-4edf-a2dd-7071b0fe91c5" containerName="test-curl-7602-8a7deb"
Sep 18 18:19:24.020553 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:24.020531    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-53cde3/sub-test-model-53cde3-kserve-5bfb84c797-qs67n"
Sep 18 18:19:24.024773 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:24.023713    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-53cde3\"/\"kube-root-ca.crt\""
Sep 18 18:19:24.024773 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:24.023952    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-53cde3\"/\"sub-test-model-53cde3-kserve-self-signed-certs\""
Sep 18 18:19:24.024773 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:24.023719    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-53cde3\"/\"default-dockercfg-rks92\""
Sep 18 18:19:24.024773 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:24.024445    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-53cde3\"/\"openshift-service-ca.crt\""
Sep 18 18:19:24.026863 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:24.026838    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-53cde3/sub-test-model-53cde3-kserve-5bfb84c797-qs67n"]
Sep 18 18:19:24.045580 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:24.045558    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:24.127699 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:24.127632    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bc5b8d4-c09d-4db0-b483-f409d3783a42-model-cache\") pod \"sub-test-model-53cde3-kserve-5bfb84c797-qs67n\" (UID: \"9bc5b8d4-c09d-4db0-b483-f409d3783a42\") " pod="ai-tenant-e2e-shared-a-w3-53cde3/sub-test-model-53cde3-kserve-5bfb84c797-qs67n"
Sep 18 18:19:24.127699 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:24.127691    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxtfm\" (UniqueName: \"kubernetes.io/projected/9bc5b8d4-c09d-4db0-b483-f409d3783a42-kube-api-access-cxtfm\") pod \"sub-test-model-53cde3-kserve-5bfb84c797-qs67n\" (UID: \"9bc5b8d4-c09d-4db0-b483-f409d3783a42\") " pod="ai-tenant-e2e-shared-a-w3-53cde3/sub-test-model-53cde3-kserve-5bfb84c797-qs67n"
Sep 18 18:19:24.127871 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:24.127722    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bc5b8d4-c09d-4db0-b483-f409d3783a42-home\") pod \"sub-test-model-53cde3-kserve-5bfb84c797-qs67n\" (UID: \"9bc5b8d4-c09d-4db0-b483-f409d3783a42\") " pod="ai-tenant-e2e-shared-a-w3-53cde3/sub-test-model-53cde3-kserve-5bfb84c797-qs67n"
Sep 18 18:19:24.127871 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:24.127749    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bc5b8d4-c09d-4db0-b483-f409d3783a42-tls-certs\") pod \"sub-test-model-53cde3-kserve-5bfb84c797-qs67n\" (UID: \"9bc5b8d4-c09d-4db0-b483-f409d3783a42\") " pod="ai-tenant-e2e-shared-a-w3-53cde3/sub-test-model-53cde3-kserve-5bfb84c797-qs67n"
Sep 18 18:19:24.127871 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:24.127783    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bc5b8d4-c09d-4db0-b483-f409d3783a42-dshm\") pod \"sub-test-model-53cde3-kserve-5bfb84c797-qs67n\" (UID: \"9bc5b8d4-c09d-4db0-b483-f409d3783a42\") " pod="ai-tenant-e2e-shared-a-w3-53cde3/sub-test-model-53cde3-kserve-5bfb84c797-qs67n"
Sep 18 18:19:24.127968 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:24.127905    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bc5b8d4-c09d-4db0-b483-f409d3783a42-tmp-dir\") pod \"sub-test-model-53cde3-kserve-5bfb84c797-qs67n\" (UID: \"9bc5b8d4-c09d-4db0-b483-f409d3783a42\") " pod="ai-tenant-e2e-shared-a-w3-53cde3/sub-test-model-53cde3-kserve-5bfb84c797-qs67n"
Sep 18 18:19:24.228508 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:24.228475    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bc5b8d4-c09d-4db0-b483-f409d3783a42-dshm\") pod \"sub-test-model-53cde3-kserve-5bfb84c797-qs67n\" (UID: \"9bc5b8d4-c09d-4db0-b483-f409d3783a42\") " pod="ai-tenant-e2e-shared-a-w3-53cde3/sub-test-model-53cde3-kserve-5bfb84c797-qs67n"
Sep 18 18:19:24.228627 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:24.228559    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bc5b8d4-c09d-4db0-b483-f409d3783a42-tmp-dir\") pod \"sub-test-model-53cde3-kserve-5bfb84c797-qs67n\" (UID: \"9bc5b8d4-c09d-4db0-b483-f409d3783a42\") " pod="ai-tenant-e2e-shared-a-w3-53cde3/sub-test-model-53cde3-kserve-5bfb84c797-qs67n"
Sep 18 18:19:24.228627 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:24.228590    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bc5b8d4-c09d-4db0-b483-f409d3783a42-model-cache\") pod \"sub-test-model-53cde3-kserve-5bfb84c797-qs67n\" (UID: \"9bc5b8d4-c09d-4db0-b483-f409d3783a42\") " pod="ai-tenant-e2e-shared-a-w3-53cde3/sub-test-model-53cde3-kserve-5bfb84c797-qs67n"
Sep 18 18:19:24.228715 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:24.228639    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxtfm\" (UniqueName: \"kubernetes.io/projected/9bc5b8d4-c09d-4db0-b483-f409d3783a42-kube-api-access-cxtfm\") pod \"sub-test-model-53cde3-kserve-5bfb84c797-qs67n\" (UID: \"9bc5b8d4-c09d-4db0-b483-f409d3783a42\") " pod="ai-tenant-e2e-shared-a-w3-53cde3/sub-test-model-53cde3-kserve-5bfb84c797-qs67n"
Sep 18 18:19:24.228715 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:24.228668    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bc5b8d4-c09d-4db0-b483-f409d3783a42-home\") pod \"sub-test-model-53cde3-kserve-5bfb84c797-qs67n\" (UID: \"9bc5b8d4-c09d-4db0-b483-f409d3783a42\") " pod="ai-tenant-e2e-shared-a-w3-53cde3/sub-test-model-53cde3-kserve-5bfb84c797-qs67n"
Sep 18 18:19:24.228715 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:24.228693    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bc5b8d4-c09d-4db0-b483-f409d3783a42-tls-certs\") pod \"sub-test-model-53cde3-kserve-5bfb84c797-qs67n\" (UID: \"9bc5b8d4-c09d-4db0-b483-f409d3783a42\") " pod="ai-tenant-e2e-shared-a-w3-53cde3/sub-test-model-53cde3-kserve-5bfb84c797-qs67n"
Sep 18 18:19:24.228987 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:24.228966    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bc5b8d4-c09d-4db0-b483-f409d3783a42-tmp-dir\") pod \"sub-test-model-53cde3-kserve-5bfb84c797-qs67n\" (UID: \"9bc5b8d4-c09d-4db0-b483-f409d3783a42\") " pod="ai-tenant-e2e-shared-a-w3-53cde3/sub-test-model-53cde3-kserve-5bfb84c797-qs67n"
Sep 18 18:19:24.229063 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:24.228985    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bc5b8d4-c09d-4db0-b483-f409d3783a42-model-cache\") pod \"sub-test-model-53cde3-kserve-5bfb84c797-qs67n\" (UID: \"9bc5b8d4-c09d-4db0-b483-f409d3783a42\") " pod="ai-tenant-e2e-shared-a-w3-53cde3/sub-test-model-53cde3-kserve-5bfb84c797-qs67n"
Sep 18 18:19:24.229178 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:24.229158    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bc5b8d4-c09d-4db0-b483-f409d3783a42-home\") pod \"sub-test-model-53cde3-kserve-5bfb84c797-qs67n\" (UID: \"9bc5b8d4-c09d-4db0-b483-f409d3783a42\") " pod="ai-tenant-e2e-shared-a-w3-53cde3/sub-test-model-53cde3-kserve-5bfb84c797-qs67n"
Sep 18 18:19:24.230689 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:24.230660    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bc5b8d4-c09d-4db0-b483-f409d3783a42-dshm\") pod \"sub-test-model-53cde3-kserve-5bfb84c797-qs67n\" (UID: \"9bc5b8d4-c09d-4db0-b483-f409d3783a42\") " pod="ai-tenant-e2e-shared-a-w3-53cde3/sub-test-model-53cde3-kserve-5bfb84c797-qs67n"
Sep 18 18:19:24.230970 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:24.230953    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bc5b8d4-c09d-4db0-b483-f409d3783a42-tls-certs\") pod \"sub-test-model-53cde3-kserve-5bfb84c797-qs67n\" (UID: \"9bc5b8d4-c09d-4db0-b483-f409d3783a42\") " pod="ai-tenant-e2e-shared-a-w3-53cde3/sub-test-model-53cde3-kserve-5bfb84c797-qs67n"
Sep 18 18:19:24.236250 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:24.236231    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxtfm\" (UniqueName: \"kubernetes.io/projected/9bc5b8d4-c09d-4db0-b483-f409d3783a42-kube-api-access-cxtfm\") pod \"sub-test-model-53cde3-kserve-5bfb84c797-qs67n\" (UID: \"9bc5b8d4-c09d-4db0-b483-f409d3783a42\") " pod="ai-tenant-e2e-shared-a-w3-53cde3/sub-test-model-53cde3-kserve-5bfb84c797-qs67n"
Sep 18 18:19:24.333016 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:24.332975    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-53cde3/sub-test-model-53cde3-kserve-5bfb84c797-qs67n"
Sep 18 18:19:24.607764 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:24.607738    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-53cde3/sub-test-model-53cde3-kserve-5bfb84c797-qs67n"]
Sep 18 18:19:24.871063 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:24.871024    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-53cde3/sub-test-model-53cde3-kserve-5bfb84c797-qs67n" event={"ID":"9bc5b8d4-c09d-4db0-b483-f409d3783a42","Type":"ContainerStarted","Data":"102a0c6f223e7305b9a2086f839b067713e08a04430480f73ddfd5550a773080"}
Sep 18 18:19:24.871063 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:24.871066    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-53cde3/sub-test-model-53cde3-kserve-5bfb84c797-qs67n" event={"ID":"9bc5b8d4-c09d-4db0-b483-f409d3783a42","Type":"ContainerStarted","Data":"648baa8772dc86d635727a93ddf6871dc2ade4ee0af2007f5722761c5f9ddbfa"}
Sep 18 18:19:24.889773 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:24.889724    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-53cde3/sub-test-model-53cde3-kserve-5bfb84c797-qs67n" podStartSLOduration=1.889705544 podStartE2EDuration="1.889705544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:19:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:19:24.889206391 +0000 UTC m=+1769.516338011" watchObservedRunningTime="2026-09-18 18:19:24.889705544 +0000 UTC m=+1769.516837166"
Sep 18 18:19:25.861797 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:25.861768    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0864e158-2933-4867-b3b9-8ef5d9749ad5" path="/var/lib/kubelet/pods/0864e158-2933-4867-b3b9-8ef5d9749ad5/volumes"
Sep 18 18:19:28.236466 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:28.236429    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-130a80-777c8489f4-tdchm"]
Sep 18 18:19:28.236885 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:28.236675    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-130a80-777c8489f4-tdchm" podUID="b9e1f09c-7857-49ae-8efe-78b2bad8cf8d" containerName="maas-api" containerID="cri-o://b2905502d00bb7cef9ac03136f2f516815de4c44603fd5ab2d922652bb1db806" gracePeriod=30
Sep 18 18:19:28.278687 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:28.278662    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:28.379524 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:28.379491    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:28.480382 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:28.480355    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:28.488459 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:28.488407    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-130a80-777c8489f4-tdchm"
Sep 18 18:19:28.571771 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:28.571738    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j2846\" (UniqueName: \"kubernetes.io/projected/b9e1f09c-7857-49ae-8efe-78b2bad8cf8d-kube-api-access-j2846\") pod \"b9e1f09c-7857-49ae-8efe-78b2bad8cf8d\" (UID: \"b9e1f09c-7857-49ae-8efe-78b2bad8cf8d\") "
Sep 18 18:19:28.571959 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:28.571798    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9e1f09c-7857-49ae-8efe-78b2bad8cf8d-metrics-certs\") pod \"b9e1f09c-7857-49ae-8efe-78b2bad8cf8d\" (UID: \"b9e1f09c-7857-49ae-8efe-78b2bad8cf8d\") "
Sep 18 18:19:28.571959 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:28.571833    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9e1f09c-7857-49ae-8efe-78b2bad8cf8d-openshift-service-ca\") pod \"b9e1f09c-7857-49ae-8efe-78b2bad8cf8d\" (UID: \"b9e1f09c-7857-49ae-8efe-78b2bad8cf8d\") "
Sep 18 18:19:28.571959 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:28.571894    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9e1f09c-7857-49ae-8efe-78b2bad8cf8d-maas-api-tls\") pod \"b9e1f09c-7857-49ae-8efe-78b2bad8cf8d\" (UID: \"b9e1f09c-7857-49ae-8efe-78b2bad8cf8d\") "
Sep 18 18:19:28.572278 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:28.572252    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9e1f09c-7857-49ae-8efe-78b2bad8cf8d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b9e1f09c-7857-49ae-8efe-78b2bad8cf8d" (UID: "b9e1f09c-7857-49ae-8efe-78b2bad8cf8d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:19:28.573913 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:28.573892    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9e1f09c-7857-49ae-8efe-78b2bad8cf8d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b9e1f09c-7857-49ae-8efe-78b2bad8cf8d" (UID: "b9e1f09c-7857-49ae-8efe-78b2bad8cf8d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:19:28.573997 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:28.573940    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9e1f09c-7857-49ae-8efe-78b2bad8cf8d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b9e1f09c-7857-49ae-8efe-78b2bad8cf8d" (UID: "b9e1f09c-7857-49ae-8efe-78b2bad8cf8d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:19:28.574033 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:28.574008    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9e1f09c-7857-49ae-8efe-78b2bad8cf8d-kube-api-access-j2846" (OuterVolumeSpecName: "kube-api-access-j2846") pod "b9e1f09c-7857-49ae-8efe-78b2bad8cf8d" (UID: "b9e1f09c-7857-49ae-8efe-78b2bad8cf8d"). InnerVolumeSpecName "kube-api-access-j2846". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:19:28.578510 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:28.578490    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:28.673184 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:28.673154    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9e1f09c-7857-49ae-8efe-78b2bad8cf8d-maas-api-tls\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:19:28.673184 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:28.673181    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2846\" (UniqueName: \"kubernetes.io/projected/b9e1f09c-7857-49ae-8efe-78b2bad8cf8d-kube-api-access-j2846\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:19:28.673184 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:28.673191    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9e1f09c-7857-49ae-8efe-78b2bad8cf8d-metrics-certs\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:19:28.673403 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:28.673200    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9e1f09c-7857-49ae-8efe-78b2bad8cf8d-openshift-service-ca\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:19:28.894522 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:28.894443    2561 generic.go:356] "Generic (PLEG): container finished" podID="b9e1f09c-7857-49ae-8efe-78b2bad8cf8d" containerID="b2905502d00bb7cef9ac03136f2f516815de4c44603fd5ab2d922652bb1db806" exitCode=0
Sep 18 18:19:28.894522 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:28.894477    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-130a80-777c8489f4-tdchm"
Sep 18 18:19:28.894522 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:28.894494    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-130a80-777c8489f4-tdchm" event={"ID":"b9e1f09c-7857-49ae-8efe-78b2bad8cf8d","Type":"ContainerDied","Data":"b2905502d00bb7cef9ac03136f2f516815de4c44603fd5ab2d922652bb1db806"}
Sep 18 18:19:28.894750 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:28.894530    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-130a80-777c8489f4-tdchm" event={"ID":"b9e1f09c-7857-49ae-8efe-78b2bad8cf8d","Type":"ContainerDied","Data":"9909cf46822293bba750cc962384fcfd1253c7b2c2a7cd0ccce26041de7b8ffc"}
Sep 18 18:19:28.894750 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:28.894551    2561 scope.go:117] "RemoveContainer" containerID="b2905502d00bb7cef9ac03136f2f516815de4c44603fd5ab2d922652bb1db806"
Sep 18 18:19:28.904173 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:28.904151    2561 scope.go:117] "RemoveContainer" containerID="b2905502d00bb7cef9ac03136f2f516815de4c44603fd5ab2d922652bb1db806"
Sep 18 18:19:28.904418 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:19:28.904401    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2905502d00bb7cef9ac03136f2f516815de4c44603fd5ab2d922652bb1db806\": container with ID starting with b2905502d00bb7cef9ac03136f2f516815de4c44603fd5ab2d922652bb1db806 not found: ID does not exist" containerID="b2905502d00bb7cef9ac03136f2f516815de4c44603fd5ab2d922652bb1db806"
Sep 18 18:19:28.904466 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:28.904426    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2905502d00bb7cef9ac03136f2f516815de4c44603fd5ab2d922652bb1db806"} err="failed to get container status \"b2905502d00bb7cef9ac03136f2f516815de4c44603fd5ab2d922652bb1db806\": rpc error: code = NotFound desc = could not find container \"b2905502d00bb7cef9ac03136f2f516815de4c44603fd5ab2d922652bb1db806\": container with ID starting with b2905502d00bb7cef9ac03136f2f516815de4c44603fd5ab2d922652bb1db806 not found: ID does not exist"
Sep 18 18:19:28.908267 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:28.908249    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:28.922337 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:28.922317    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-130a80-777c8489f4-tdchm"]
Sep 18 18:19:28.928806 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:28.928784    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-130a80-777c8489f4-tdchm"]
Sep 18 18:19:29.008759 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:29.008730    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:29.862437 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:29.862395    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9e1f09c-7857-49ae-8efe-78b2bad8cf8d" path="/var/lib/kubelet/pods/b9e1f09c-7857-49ae-8efe-78b2bad8cf8d/volumes"
Sep 18 18:19:29.960478 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:29.960440    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:30.855251 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:30.855216    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:32.489583 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:32.489550    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:34.333450 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:34.333411    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-53cde3/sub-test-model-53cde3-kserve-5bfb84c797-qs67n"
Sep 18 18:19:34.333450 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:34.333457    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-53cde3/sub-test-model-53cde3-kserve-5bfb84c797-qs67n"
Sep 18 18:19:34.346304 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:34.346278    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-53cde3/sub-test-model-53cde3-kserve-5bfb84c797-qs67n"
Sep 18 18:19:34.750870 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:34.750818    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:34.889249 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:34.889223    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:34.936918 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:34.936883    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-53cde3/sub-test-model-53cde3-kserve-5bfb84c797-qs67n"
Sep 18 18:19:35.054783 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:35.054706    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:36.701535 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:36.701503    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:37.357711 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:37.357673    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:55.936315 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:55.936284    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-5kjlz_75f2a1f7-95a7-462e-9ea7-23c51dd29599/console-operator/1.log"
Sep 18 18:19:55.941941 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:55.941921    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-5kjlz_75f2a1f7-95a7-462e-9ea7-23c51dd29599/console-operator/1.log"
Sep 18 18:19:55.943538 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:55.943516    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkcgk_c5d026b7-af6b-4899-85f4-6554c0f7cd52/ovn-acl-logging/0.log"
Sep 18 18:19:55.948868 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:55.948852    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkcgk_c5d026b7-af6b-4899-85f4-6554c0f7cd52/ovn-acl-logging/0.log"
Sep 18 18:19:58.623229 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:58.623195    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-341bd4/sub-test-model-341bd4-kserve-7fd8ccc679-mb8b9"]
Sep 18 18:19:58.623759 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:58.623636    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9e1f09c-7857-49ae-8efe-78b2bad8cf8d" containerName="maas-api"
Sep 18 18:19:58.623759 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:58.623649    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9e1f09c-7857-49ae-8efe-78b2bad8cf8d" containerName="maas-api"
Sep 18 18:19:58.623759 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:58.623735    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9e1f09c-7857-49ae-8efe-78b2bad8cf8d" containerName="maas-api"
Sep 18 18:19:58.626715 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:58.626690    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-341bd4/sub-test-model-341bd4-kserve-7fd8ccc679-mb8b9"
Sep 18 18:19:58.629060 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:58.629035    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-341bd4\"/\"kube-root-ca.crt\""
Sep 18 18:19:58.629790 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:58.629769    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-341bd4\"/\"default-dockercfg-j284w\""
Sep 18 18:19:58.629790 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:58.629788    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-341bd4\"/\"openshift-service-ca.crt\""
Sep 18 18:19:58.630001 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:58.629789    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-341bd4\"/\"sub-test-model-341bd4-kserve-self-signed-certs\""
Sep 18 18:19:58.637322 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:58.637298    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-341bd4/sub-test-model-341bd4-kserve-7fd8ccc679-mb8b9"]
Sep 18 18:19:58.655636 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:58.655607    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:19:58.746982 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:58.746953    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mknds\" (UniqueName: \"kubernetes.io/projected/e3e05d01-c331-44a9-a8d5-b021cb59c0bf-kube-api-access-mknds\") pod \"sub-test-model-341bd4-kserve-7fd8ccc679-mb8b9\" (UID: \"e3e05d01-c331-44a9-a8d5-b021cb59c0bf\") " pod="ai-tenant-e2e-shared-b-w3-341bd4/sub-test-model-341bd4-kserve-7fd8ccc679-mb8b9"
Sep 18 18:19:58.747143 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:58.747009    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3e05d01-c331-44a9-a8d5-b021cb59c0bf-home\") pod \"sub-test-model-341bd4-kserve-7fd8ccc679-mb8b9\" (UID: \"e3e05d01-c331-44a9-a8d5-b021cb59c0bf\") " pod="ai-tenant-e2e-shared-b-w3-341bd4/sub-test-model-341bd4-kserve-7fd8ccc679-mb8b9"
Sep 18 18:19:58.747143 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:58.747060    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3e05d01-c331-44a9-a8d5-b021cb59c0bf-model-cache\") pod \"sub-test-model-341bd4-kserve-7fd8ccc679-mb8b9\" (UID: \"e3e05d01-c331-44a9-a8d5-b021cb59c0bf\") " pod="ai-tenant-e2e-shared-b-w3-341bd4/sub-test-model-341bd4-kserve-7fd8ccc679-mb8b9"
Sep 18 18:19:58.747143 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:58.747080    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3e05d01-c331-44a9-a8d5-b021cb59c0bf-tls-certs\") pod \"sub-test-model-341bd4-kserve-7fd8ccc679-mb8b9\" (UID: \"e3e05d01-c331-44a9-a8d5-b021cb59c0bf\") " pod="ai-tenant-e2e-shared-b-w3-341bd4/sub-test-model-341bd4-kserve-7fd8ccc679-mb8b9"
Sep 18 18:19:58.747143 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:58.747104    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3e05d01-c331-44a9-a8d5-b021cb59c0bf-tmp-dir\") pod \"sub-test-model-341bd4-kserve-7fd8ccc679-mb8b9\" (UID: \"e3e05d01-c331-44a9-a8d5-b021cb59c0bf\") " pod="ai-tenant-e2e-shared-b-w3-341bd4/sub-test-model-341bd4-kserve-7fd8ccc679-mb8b9"
Sep 18 18:19:58.747289 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:58.747150    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3e05d01-c331-44a9-a8d5-b021cb59c0bf-dshm\") pod \"sub-test-model-341bd4-kserve-7fd8ccc679-mb8b9\" (UID: \"e3e05d01-c331-44a9-a8d5-b021cb59c0bf\") " pod="ai-tenant-e2e-shared-b-w3-341bd4/sub-test-model-341bd4-kserve-7fd8ccc679-mb8b9"
Sep 18 18:19:58.847572 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:58.847542    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3e05d01-c331-44a9-a8d5-b021cb59c0bf-home\") pod \"sub-test-model-341bd4-kserve-7fd8ccc679-mb8b9\" (UID: \"e3e05d01-c331-44a9-a8d5-b021cb59c0bf\") " pod="ai-tenant-e2e-shared-b-w3-341bd4/sub-test-model-341bd4-kserve-7fd8ccc679-mb8b9"
Sep 18 18:19:58.847722 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:58.847589    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3e05d01-c331-44a9-a8d5-b021cb59c0bf-model-cache\") pod \"sub-test-model-341bd4-kserve-7fd8ccc679-mb8b9\" (UID: \"e3e05d01-c331-44a9-a8d5-b021cb59c0bf\") " pod="ai-tenant-e2e-shared-b-w3-341bd4/sub-test-model-341bd4-kserve-7fd8ccc679-mb8b9"
Sep 18 18:19:58.847722 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:58.847613    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3e05d01-c331-44a9-a8d5-b021cb59c0bf-tls-certs\") pod \"sub-test-model-341bd4-kserve-7fd8ccc679-mb8b9\" (UID: \"e3e05d01-c331-44a9-a8d5-b021cb59c0bf\") " pod="ai-tenant-e2e-shared-b-w3-341bd4/sub-test-model-341bd4-kserve-7fd8ccc679-mb8b9"
Sep 18 18:19:58.847722 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:58.847651    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3e05d01-c331-44a9-a8d5-b021cb59c0bf-tmp-dir\") pod \"sub-test-model-341bd4-kserve-7fd8ccc679-mb8b9\" (UID: \"e3e05d01-c331-44a9-a8d5-b021cb59c0bf\") " pod="ai-tenant-e2e-shared-b-w3-341bd4/sub-test-model-341bd4-kserve-7fd8ccc679-mb8b9"
Sep 18 18:19:58.847722 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:58.847669    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3e05d01-c331-44a9-a8d5-b021cb59c0bf-dshm\") pod \"sub-test-model-341bd4-kserve-7fd8ccc679-mb8b9\" (UID: \"e3e05d01-c331-44a9-a8d5-b021cb59c0bf\") " pod="ai-tenant-e2e-shared-b-w3-341bd4/sub-test-model-341bd4-kserve-7fd8ccc679-mb8b9"
Sep 18 18:19:58.847722 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:58.847695    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mknds\" (UniqueName: \"kubernetes.io/projected/e3e05d01-c331-44a9-a8d5-b021cb59c0bf-kube-api-access-mknds\") pod \"sub-test-model-341bd4-kserve-7fd8ccc679-mb8b9\" (UID: \"e3e05d01-c331-44a9-a8d5-b021cb59c0bf\") " pod="ai-tenant-e2e-shared-b-w3-341bd4/sub-test-model-341bd4-kserve-7fd8ccc679-mb8b9"
Sep 18 18:19:58.848009 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:58.847984    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3e05d01-c331-44a9-a8d5-b021cb59c0bf-home\") pod \"sub-test-model-341bd4-kserve-7fd8ccc679-mb8b9\" (UID: \"e3e05d01-c331-44a9-a8d5-b021cb59c0bf\") " pod="ai-tenant-e2e-shared-b-w3-341bd4/sub-test-model-341bd4-kserve-7fd8ccc679-mb8b9"
Sep 18 18:19:58.848073 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:58.848007    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3e05d01-c331-44a9-a8d5-b021cb59c0bf-model-cache\") pod \"sub-test-model-341bd4-kserve-7fd8ccc679-mb8b9\" (UID: \"e3e05d01-c331-44a9-a8d5-b021cb59c0bf\") " pod="ai-tenant-e2e-shared-b-w3-341bd4/sub-test-model-341bd4-kserve-7fd8ccc679-mb8b9"
Sep 18 18:19:58.848157 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:58.848080    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3e05d01-c331-44a9-a8d5-b021cb59c0bf-tmp-dir\") pod \"sub-test-model-341bd4-kserve-7fd8ccc679-mb8b9\" (UID: \"e3e05d01-c331-44a9-a8d5-b021cb59c0bf\") " pod="ai-tenant-e2e-shared-b-w3-341bd4/sub-test-model-341bd4-kserve-7fd8ccc679-mb8b9"
Sep 18 18:19:58.849948 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:58.849929    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3e05d01-c331-44a9-a8d5-b021cb59c0bf-dshm\") pod \"sub-test-model-341bd4-kserve-7fd8ccc679-mb8b9\" (UID: \"e3e05d01-c331-44a9-a8d5-b021cb59c0bf\") " pod="ai-tenant-e2e-shared-b-w3-341bd4/sub-test-model-341bd4-kserve-7fd8ccc679-mb8b9"
Sep 18 18:19:58.850247 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:58.850229    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3e05d01-c331-44a9-a8d5-b021cb59c0bf-tls-certs\") pod \"sub-test-model-341bd4-kserve-7fd8ccc679-mb8b9\" (UID: \"e3e05d01-c331-44a9-a8d5-b021cb59c0bf\") " pod="ai-tenant-e2e-shared-b-w3-341bd4/sub-test-model-341bd4-kserve-7fd8ccc679-mb8b9"
Sep 18 18:19:58.856003 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:58.855979    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mknds\" (UniqueName: \"kubernetes.io/projected/e3e05d01-c331-44a9-a8d5-b021cb59c0bf-kube-api-access-mknds\") pod \"sub-test-model-341bd4-kserve-7fd8ccc679-mb8b9\" (UID: \"e3e05d01-c331-44a9-a8d5-b021cb59c0bf\") " pod="ai-tenant-e2e-shared-b-w3-341bd4/sub-test-model-341bd4-kserve-7fd8ccc679-mb8b9"
Sep 18 18:19:58.937986 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:58.937963    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-341bd4/sub-test-model-341bd4-kserve-7fd8ccc679-mb8b9"
Sep 18 18:19:59.209690 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:19:59.209662    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-341bd4/sub-test-model-341bd4-kserve-7fd8ccc679-mb8b9"]
Sep 18 18:19:59.211939 ip-10-0-140-54 kubenswrapper[2561]: W0918 18:19:59.211911    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode3e05d01_c331_44a9_a8d5_b021cb59c0bf.slice/crio-11210cf2b79b3a40809f09b3a8a95004e259e5c8a3e3edc4a926f5d536f08230 WatchSource:0}: Error finding container 11210cf2b79b3a40809f09b3a8a95004e259e5c8a3e3edc4a926f5d536f08230: Status 404 returned error can't find the container with id 11210cf2b79b3a40809f09b3a8a95004e259e5c8a3e3edc4a926f5d536f08230
Sep 18 18:20:00.051763 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:00.051718    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-341bd4/sub-test-model-341bd4-kserve-7fd8ccc679-mb8b9" event={"ID":"e3e05d01-c331-44a9-a8d5-b021cb59c0bf","Type":"ContainerStarted","Data":"34bb35492e42e3d61c66948b2dd4613d5da0ab70e77ad27e30c3ddec04fa73f8"}
Sep 18 18:20:00.052135 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:00.051769    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-341bd4/sub-test-model-341bd4-kserve-7fd8ccc679-mb8b9" event={"ID":"e3e05d01-c331-44a9-a8d5-b021cb59c0bf","Type":"ContainerStarted","Data":"11210cf2b79b3a40809f09b3a8a95004e259e5c8a3e3edc4a926f5d536f08230"}
Sep 18 18:20:00.069070 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:00.069021    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-341bd4/sub-test-model-341bd4-kserve-7fd8ccc679-mb8b9" podStartSLOduration=2.06900681 podStartE2EDuration="2.06900681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:19:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:20:00.067314014 +0000 UTC m=+1804.694445635" watchObservedRunningTime="2026-09-18 18:20:00.06900681 +0000 UTC m=+1804.696138429"
Sep 18 18:20:08.938831 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:08.938796    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-341bd4/sub-test-model-341bd4-kserve-7fd8ccc679-mb8b9"
Sep 18 18:20:08.939226 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:08.938871    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-341bd4/sub-test-model-341bd4-kserve-7fd8ccc679-mb8b9"
Sep 18 18:20:08.951595 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:08.951571    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-341bd4/sub-test-model-341bd4-kserve-7fd8ccc679-mb8b9"
Sep 18 18:20:09.106182 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:09.106149    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-341bd4/sub-test-model-341bd4-kserve-7fd8ccc679-mb8b9"
Sep 18 18:20:11.119367 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:11.119330    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:11.249104 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:11.249045    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:36.491479 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:36.491439    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7602-006dae"]
Sep 18 18:20:36.494944 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:36.494926    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7602-006dae"
Sep 18 18:20:36.497162 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:36.497142    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-fz228\""
Sep 18 18:20:36.500868 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:36.500844    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7602-006dae"]
Sep 18 18:20:36.596555 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:36.596508    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjkst\" (UniqueName: \"kubernetes.io/projected/b77a0b6a-dc8a-4472-ab85-dd13ffa5959b-kube-api-access-rjkst\") pod \"test-curl-7602-006dae\" (UID: \"b77a0b6a-dc8a-4472-ab85-dd13ffa5959b\") " pod="opendatahub/test-curl-7602-006dae"
Sep 18 18:20:36.697143 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:36.697098    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjkst\" (UniqueName: \"kubernetes.io/projected/b77a0b6a-dc8a-4472-ab85-dd13ffa5959b-kube-api-access-rjkst\") pod \"test-curl-7602-006dae\" (UID: \"b77a0b6a-dc8a-4472-ab85-dd13ffa5959b\") " pod="opendatahub/test-curl-7602-006dae"
Sep 18 18:20:36.705296 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:36.705276    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjkst\" (UniqueName: \"kubernetes.io/projected/b77a0b6a-dc8a-4472-ab85-dd13ffa5959b-kube-api-access-rjkst\") pod \"test-curl-7602-006dae\" (UID: \"b77a0b6a-dc8a-4472-ab85-dd13ffa5959b\") " pod="opendatahub/test-curl-7602-006dae"
Sep 18 18:20:36.806724 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:36.806632    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7602-006dae"
Sep 18 18:20:37.076243 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:37.076219    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7602-006dae"]
Sep 18 18:20:37.078220 ip-10-0-140-54 kubenswrapper[2561]: W0918 18:20:37.078192    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb77a0b6a_dc8a_4472_ab85_dd13ffa5959b.slice/crio-595616a70a0a73b84b3afa42419e297cc5c8a50360af441cad233c7e4d7b48bc WatchSource:0}: Error finding container 595616a70a0a73b84b3afa42419e297cc5c8a50360af441cad233c7e4d7b48bc: Status 404 returned error can't find the container with id 595616a70a0a73b84b3afa42419e297cc5c8a50360af441cad233c7e4d7b48bc
Sep 18 18:20:37.238165 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:37.238129    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7602-006dae" event={"ID":"b77a0b6a-dc8a-4472-ab85-dd13ffa5959b","Type":"ContainerStarted","Data":"de4af20b54f3c06ba580ed003a80bee538146037e5ef75ea178ffeb3f7f414cc"}
Sep 18 18:20:37.238331 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:37.238180    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7602-006dae" event={"ID":"b77a0b6a-dc8a-4472-ab85-dd13ffa5959b","Type":"ContainerStarted","Data":"595616a70a0a73b84b3afa42419e297cc5c8a50360af441cad233c7e4d7b48bc"}
Sep 18 18:20:37.253005 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:37.252960    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7602-006dae" podStartSLOduration=1.252945961 podStartE2EDuration="1.252945961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:20:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:20:37.251831937 +0000 UTC m=+1841.878963557" watchObservedRunningTime="2026-09-18 18:20:37.252945961 +0000 UTC m=+1841.880077580"
Sep 18 18:20:37.815729 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:37.815693    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7602-006dae"]
Sep 18 18:20:37.820312 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:37.820285    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7602-006dae"]
Sep 18 18:20:37.928343 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:37.928308    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7602-d184bf"]
Sep 18 18:20:37.931450 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:37.930020    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b77a0b6a-dc8a-4472-ab85-dd13ffa5959b" containerName="test-curl-7602-006dae"
Sep 18 18:20:37.931450 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:37.930049    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b77a0b6a-dc8a-4472-ab85-dd13ffa5959b" containerName="test-curl-7602-006dae"
Sep 18 18:20:37.931450 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:37.930251    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b77a0b6a-dc8a-4472-ab85-dd13ffa5959b" containerName="test-curl-7602-006dae"
Sep 18 18:20:37.936726 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:37.936704    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7602-d184bf"
Sep 18 18:20:37.941953 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:37.941931    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7602-d184bf"]
Sep 18 18:20:38.009573 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:38.009536    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc8nx\" (UniqueName: \"kubernetes.io/projected/943841a0-be31-472c-942b-b3ffa4b7c8b8-kube-api-access-hc8nx\") pod \"test-curl-7602-d184bf\" (UID: \"943841a0-be31-472c-942b-b3ffa4b7c8b8\") " pod="opendatahub/test-curl-7602-d184bf"
Sep 18 18:20:38.110765 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:38.110671    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc8nx\" (UniqueName: \"kubernetes.io/projected/943841a0-be31-472c-942b-b3ffa4b7c8b8-kube-api-access-hc8nx\") pod \"test-curl-7602-d184bf\" (UID: \"943841a0-be31-472c-942b-b3ffa4b7c8b8\") " pod="opendatahub/test-curl-7602-d184bf"
Sep 18 18:20:38.118391 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:38.118368    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc8nx\" (UniqueName: \"kubernetes.io/projected/943841a0-be31-472c-942b-b3ffa4b7c8b8-kube-api-access-hc8nx\") pod \"test-curl-7602-d184bf\" (UID: \"943841a0-be31-472c-942b-b3ffa4b7c8b8\") " pod="opendatahub/test-curl-7602-d184bf"
Sep 18 18:20:38.242872 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:20:38.242842    2561 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7602-006dae" container="test-curl-7602-006dae"
Sep 18 18:20:38.246818 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:38.246802    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7602-d184bf"
Sep 18 18:20:38.516049 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:38.516017    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7602-d184bf"]
Sep 18 18:20:38.517915 ip-10-0-140-54 kubenswrapper[2561]: W0918 18:20:38.517873    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod943841a0_be31_472c_942b_b3ffa4b7c8b8.slice/crio-e87933f61583da22c06c86b4c38f3e2c0cfdcdf221b212965772b18f642b8ffc WatchSource:0}: Error finding container e87933f61583da22c06c86b4c38f3e2c0cfdcdf221b212965772b18f642b8ffc: Status 404 returned error can't find the container with id e87933f61583da22c06c86b4c38f3e2c0cfdcdf221b212965772b18f642b8ffc
Sep 18 18:20:39.249517 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:39.249477    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7602-d184bf" event={"ID":"943841a0-be31-472c-942b-b3ffa4b7c8b8","Type":"ContainerStarted","Data":"404af69ee4de199447f9be7b8ebf7b846126779bbb140fb447a0d504f247b09a"}
Sep 18 18:20:39.249947 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:39.249530    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7602-d184bf" event={"ID":"943841a0-be31-472c-942b-b3ffa4b7c8b8","Type":"ContainerStarted","Data":"e87933f61583da22c06c86b4c38f3e2c0cfdcdf221b212965772b18f642b8ffc"}
Sep 18 18:20:39.249947 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:39.249571    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7602-006dae" podUID="b77a0b6a-dc8a-4472-ab85-dd13ffa5959b" containerName="test-curl-7602-006dae" containerID="cri-o://de4af20b54f3c06ba580ed003a80bee538146037e5ef75ea178ffeb3f7f414cc" gracePeriod=2
Sep 18 18:20:39.263700 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:39.263661    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7602-d184bf" podStartSLOduration=2.263648551 podStartE2EDuration="2.263648551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:20:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:20:39.261899938 +0000 UTC m=+1843.889031559" watchObservedRunningTime="2026-09-18 18:20:39.263648551 +0000 UTC m=+1843.890780172"
Sep 18 18:20:39.824023 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:39.823988    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7602-d184bf"]
Sep 18 18:20:39.827606 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:39.827581    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7602-d184bf"]
Sep 18 18:20:40.253849 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:20:40.253813    2561 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7602-d184bf" container="test-curl-7602-d184bf"
Sep 18 18:20:41.259262 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:41.259211    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7602-d184bf" podUID="943841a0-be31-472c-942b-b3ffa4b7c8b8" containerName="test-curl-7602-d184bf" containerID="cri-o://404af69ee4de199447f9be7b8ebf7b846126779bbb140fb447a0d504f247b09a" gracePeriod=2
Sep 18 18:20:41.426558 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:41.426516    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:41.432167 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:41.432147    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7602-006dae"
Sep 18 18:20:41.542213 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:41.542110    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rjkst\" (UniqueName: \"kubernetes.io/projected/b77a0b6a-dc8a-4472-ab85-dd13ffa5959b-kube-api-access-rjkst\") pod \"b77a0b6a-dc8a-4472-ab85-dd13ffa5959b\" (UID: \"b77a0b6a-dc8a-4472-ab85-dd13ffa5959b\") "
Sep 18 18:20:41.544152 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:41.544109    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b77a0b6a-dc8a-4472-ab85-dd13ffa5959b-kube-api-access-rjkst" (OuterVolumeSpecName: "kube-api-access-rjkst") pod "b77a0b6a-dc8a-4472-ab85-dd13ffa5959b" (UID: "b77a0b6a-dc8a-4472-ab85-dd13ffa5959b"). InnerVolumeSpecName "kube-api-access-rjkst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:20:41.642945 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:41.642912    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjkst\" (UniqueName: \"kubernetes.io/projected/b77a0b6a-dc8a-4472-ab85-dd13ffa5959b-kube-api-access-rjkst\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:20:41.764888 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:41.764853    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-53cde3/sub-test-model-53cde3-kserve-5bfb84c797-qs67n"]
Sep 18 18:20:41.861764 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:41.861687    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b77a0b6a-dc8a-4472-ab85-dd13ffa5959b" path="/var/lib/kubelet/pods/b77a0b6a-dc8a-4472-ab85-dd13ffa5959b/volumes"
Sep 18 18:20:42.269813 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:42.269780    2561 generic.go:356] "Generic (PLEG): container finished" podID="b77a0b6a-dc8a-4472-ab85-dd13ffa5959b" containerID="de4af20b54f3c06ba580ed003a80bee538146037e5ef75ea178ffeb3f7f414cc" exitCode=137
Sep 18 18:20:42.270295 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:42.269845    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7602-006dae"
Sep 18 18:20:42.270295 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:42.269889    2561 scope.go:117] "RemoveContainer" containerID="de4af20b54f3c06ba580ed003a80bee538146037e5ef75ea178ffeb3f7f414cc"
Sep 18 18:20:42.278294 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:42.278278    2561 scope.go:117] "RemoveContainer" containerID="de4af20b54f3c06ba580ed003a80bee538146037e5ef75ea178ffeb3f7f414cc"
Sep 18 18:20:42.278573 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:20:42.278554    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de4af20b54f3c06ba580ed003a80bee538146037e5ef75ea178ffeb3f7f414cc\": container with ID starting with de4af20b54f3c06ba580ed003a80bee538146037e5ef75ea178ffeb3f7f414cc not found: ID does not exist" containerID="de4af20b54f3c06ba580ed003a80bee538146037e5ef75ea178ffeb3f7f414cc"
Sep 18 18:20:42.278647 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:42.278579    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de4af20b54f3c06ba580ed003a80bee538146037e5ef75ea178ffeb3f7f414cc"} err="failed to get container status \"de4af20b54f3c06ba580ed003a80bee538146037e5ef75ea178ffeb3f7f414cc\": rpc error: code = NotFound desc = could not find container \"de4af20b54f3c06ba580ed003a80bee538146037e5ef75ea178ffeb3f7f414cc\": container with ID starting with de4af20b54f3c06ba580ed003a80bee538146037e5ef75ea178ffeb3f7f414cc not found: ID does not exist"
Sep 18 18:20:42.730087 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:42.730047    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-341bd4/sub-test-model-341bd4-kserve-7fd8ccc679-mb8b9"]
Sep 18 18:20:42.771304 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:42.771271    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:43.426866 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:43.426845    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7602-d184bf"
Sep 18 18:20:43.562100 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:43.562007    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hc8nx\" (UniqueName: \"kubernetes.io/projected/943841a0-be31-472c-942b-b3ffa4b7c8b8-kube-api-access-hc8nx\") pod \"943841a0-be31-472c-942b-b3ffa4b7c8b8\" (UID: \"943841a0-be31-472c-942b-b3ffa4b7c8b8\") "
Sep 18 18:20:43.564081 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:43.564053    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/943841a0-be31-472c-942b-b3ffa4b7c8b8-kube-api-access-hc8nx" (OuterVolumeSpecName: "kube-api-access-hc8nx") pod "943841a0-be31-472c-942b-b3ffa4b7c8b8" (UID: "943841a0-be31-472c-942b-b3ffa4b7c8b8"). InnerVolumeSpecName "kube-api-access-hc8nx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:20:43.663094 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:43.663064    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hc8nx\" (UniqueName: \"kubernetes.io/projected/943841a0-be31-472c-942b-b3ffa4b7c8b8-kube-api-access-hc8nx\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:20:43.861432 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:43.861356    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="943841a0-be31-472c-942b-b3ffa4b7c8b8" path="/var/lib/kubelet/pods/943841a0-be31-472c-942b-b3ffa4b7c8b8/volumes"
Sep 18 18:20:44.282391 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:44.282353    2561 generic.go:356] "Generic (PLEG): container finished" podID="943841a0-be31-472c-942b-b3ffa4b7c8b8" containerID="404af69ee4de199447f9be7b8ebf7b846126779bbb140fb447a0d504f247b09a" exitCode=137
Sep 18 18:20:44.282600 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:44.282393    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7602-d184bf"
Sep 18 18:20:44.282600 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:44.282449    2561 scope.go:117] "RemoveContainer" containerID="404af69ee4de199447f9be7b8ebf7b846126779bbb140fb447a0d504f247b09a"
Sep 18 18:20:44.291198 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:44.290982    2561 scope.go:117] "RemoveContainer" containerID="404af69ee4de199447f9be7b8ebf7b846126779bbb140fb447a0d504f247b09a"
Sep 18 18:20:44.291278 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:20:44.291257    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"404af69ee4de199447f9be7b8ebf7b846126779bbb140fb447a0d504f247b09a\": container with ID starting with 404af69ee4de199447f9be7b8ebf7b846126779bbb140fb447a0d504f247b09a not found: ID does not exist" containerID="404af69ee4de199447f9be7b8ebf7b846126779bbb140fb447a0d504f247b09a"
Sep 18 18:20:44.291321 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:44.291287    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"404af69ee4de199447f9be7b8ebf7b846126779bbb140fb447a0d504f247b09a"} err="failed to get container status \"404af69ee4de199447f9be7b8ebf7b846126779bbb140fb447a0d504f247b09a\": rpc error: code = NotFound desc = could not find container \"404af69ee4de199447f9be7b8ebf7b846126779bbb140fb447a0d504f247b09a\": container with ID starting with 404af69ee4de199447f9be7b8ebf7b846126779bbb140fb447a0d504f247b09a not found: ID does not exist"
Sep 18 18:20:55.755859 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:55.755828    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:20:56.784785 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:56.784742    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-53cde3/sub-test-model-53cde3-kserve-5bfb84c797-qs67n" podUID="9bc5b8d4-c09d-4db0-b483-f409d3783a42" containerName="main" containerID="cri-o://102a0c6f223e7305b9a2086f839b067713e08a04430480f73ddfd5550a773080" gracePeriod=45
Sep 18 18:20:57.052468 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:57.052443    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-53cde3/sub-test-model-53cde3-kserve-5bfb84c797-qs67n"
Sep 18 18:20:57.194841 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:57.194803    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bc5b8d4-c09d-4db0-b483-f409d3783a42-dshm\") pod \"9bc5b8d4-c09d-4db0-b483-f409d3783a42\" (UID: \"9bc5b8d4-c09d-4db0-b483-f409d3783a42\") "
Sep 18 18:20:57.195043 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:57.194875    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bc5b8d4-c09d-4db0-b483-f409d3783a42-home\") pod \"9bc5b8d4-c09d-4db0-b483-f409d3783a42\" (UID: \"9bc5b8d4-c09d-4db0-b483-f409d3783a42\") "
Sep 18 18:20:57.195043 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:57.194937    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bc5b8d4-c09d-4db0-b483-f409d3783a42-model-cache\") pod \"9bc5b8d4-c09d-4db0-b483-f409d3783a42\" (UID: \"9bc5b8d4-c09d-4db0-b483-f409d3783a42\") "
Sep 18 18:20:57.195043 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:57.194951    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bc5b8d4-c09d-4db0-b483-f409d3783a42-tmp-dir\") pod \"9bc5b8d4-c09d-4db0-b483-f409d3783a42\" (UID: \"9bc5b8d4-c09d-4db0-b483-f409d3783a42\") "
Sep 18 18:20:57.195043 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:57.194997    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cxtfm\" (UniqueName: \"kubernetes.io/projected/9bc5b8d4-c09d-4db0-b483-f409d3783a42-kube-api-access-cxtfm\") pod \"9bc5b8d4-c09d-4db0-b483-f409d3783a42\" (UID: \"9bc5b8d4-c09d-4db0-b483-f409d3783a42\") "
Sep 18 18:20:57.195043 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:57.195023    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bc5b8d4-c09d-4db0-b483-f409d3783a42-tls-certs\") pod \"9bc5b8d4-c09d-4db0-b483-f409d3783a42\" (UID: \"9bc5b8d4-c09d-4db0-b483-f409d3783a42\") "
Sep 18 18:20:57.195368 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:57.195106    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bc5b8d4-c09d-4db0-b483-f409d3783a42-home" (OuterVolumeSpecName: "home") pod "9bc5b8d4-c09d-4db0-b483-f409d3783a42" (UID: "9bc5b8d4-c09d-4db0-b483-f409d3783a42"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:20:57.195368 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:57.195253    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bc5b8d4-c09d-4db0-b483-f409d3783a42-model-cache" (OuterVolumeSpecName: "model-cache") pod "9bc5b8d4-c09d-4db0-b483-f409d3783a42" (UID: "9bc5b8d4-c09d-4db0-b483-f409d3783a42"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:20:57.195368 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:57.195262    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bc5b8d4-c09d-4db0-b483-f409d3783a42-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9bc5b8d4-c09d-4db0-b483-f409d3783a42" (UID: "9bc5b8d4-c09d-4db0-b483-f409d3783a42"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:20:57.195474 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:57.195429    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bc5b8d4-c09d-4db0-b483-f409d3783a42-home\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:20:57.195474 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:57.195449    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bc5b8d4-c09d-4db0-b483-f409d3783a42-model-cache\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:20:57.195474 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:57.195460    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bc5b8d4-c09d-4db0-b483-f409d3783a42-tmp-dir\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:20:57.197040 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:57.197014    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bc5b8d4-c09d-4db0-b483-f409d3783a42-dshm" (OuterVolumeSpecName: "dshm") pod "9bc5b8d4-c09d-4db0-b483-f409d3783a42" (UID: "9bc5b8d4-c09d-4db0-b483-f409d3783a42"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:20:57.197152 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:57.197045    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bc5b8d4-c09d-4db0-b483-f409d3783a42-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9bc5b8d4-c09d-4db0-b483-f409d3783a42" (UID: "9bc5b8d4-c09d-4db0-b483-f409d3783a42"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:20:57.197152 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:57.197057    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bc5b8d4-c09d-4db0-b483-f409d3783a42-kube-api-access-cxtfm" (OuterVolumeSpecName: "kube-api-access-cxtfm") pod "9bc5b8d4-c09d-4db0-b483-f409d3783a42" (UID: "9bc5b8d4-c09d-4db0-b483-f409d3783a42"). InnerVolumeSpecName "kube-api-access-cxtfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:20:57.296363 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:57.296279    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxtfm\" (UniqueName: \"kubernetes.io/projected/9bc5b8d4-c09d-4db0-b483-f409d3783a42-kube-api-access-cxtfm\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:20:57.296363 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:57.296317    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bc5b8d4-c09d-4db0-b483-f409d3783a42-tls-certs\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:20:57.296363 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:57.296328    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bc5b8d4-c09d-4db0-b483-f409d3783a42-dshm\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:20:57.349144 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:57.349094    2561 generic.go:356] "Generic (PLEG): container finished" podID="9bc5b8d4-c09d-4db0-b483-f409d3783a42" containerID="102a0c6f223e7305b9a2086f839b067713e08a04430480f73ddfd5550a773080" exitCode=0
Sep 18 18:20:57.349292 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:57.349168    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-53cde3/sub-test-model-53cde3-kserve-5bfb84c797-qs67n"
Sep 18 18:20:57.349292 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:57.349180    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-53cde3/sub-test-model-53cde3-kserve-5bfb84c797-qs67n" event={"ID":"9bc5b8d4-c09d-4db0-b483-f409d3783a42","Type":"ContainerDied","Data":"102a0c6f223e7305b9a2086f839b067713e08a04430480f73ddfd5550a773080"}
Sep 18 18:20:57.349292 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:57.349227    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-53cde3/sub-test-model-53cde3-kserve-5bfb84c797-qs67n" event={"ID":"9bc5b8d4-c09d-4db0-b483-f409d3783a42","Type":"ContainerDied","Data":"648baa8772dc86d635727a93ddf6871dc2ade4ee0af2007f5722761c5f9ddbfa"}
Sep 18 18:20:57.349292 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:57.349247    2561 scope.go:117] "RemoveContainer" containerID="102a0c6f223e7305b9a2086f839b067713e08a04430480f73ddfd5550a773080"
Sep 18 18:20:57.358356 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:57.358336    2561 scope.go:117] "RemoveContainer" containerID="102a0c6f223e7305b9a2086f839b067713e08a04430480f73ddfd5550a773080"
Sep 18 18:20:57.358619 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:20:57.358602    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"102a0c6f223e7305b9a2086f839b067713e08a04430480f73ddfd5550a773080\": container with ID starting with 102a0c6f223e7305b9a2086f839b067713e08a04430480f73ddfd5550a773080 not found: ID does not exist" containerID="102a0c6f223e7305b9a2086f839b067713e08a04430480f73ddfd5550a773080"
Sep 18 18:20:57.358668 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:57.358628    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"102a0c6f223e7305b9a2086f839b067713e08a04430480f73ddfd5550a773080"} err="failed to get container status \"102a0c6f223e7305b9a2086f839b067713e08a04430480f73ddfd5550a773080\": rpc error: code = NotFound desc = could not find container \"102a0c6f223e7305b9a2086f839b067713e08a04430480f73ddfd5550a773080\": container with ID starting with 102a0c6f223e7305b9a2086f839b067713e08a04430480f73ddfd5550a773080 not found: ID does not exist"
Sep 18 18:20:57.369793 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:57.369769    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-53cde3/sub-test-model-53cde3-kserve-5bfb84c797-qs67n"]
Sep 18 18:20:57.373923 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:57.373903    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-53cde3/sub-test-model-53cde3-kserve-5bfb84c797-qs67n"]
Sep 18 18:20:57.749362 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:57.749324    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-341bd4/sub-test-model-341bd4-kserve-7fd8ccc679-mb8b9" podUID="e3e05d01-c331-44a9-a8d5-b021cb59c0bf" containerName="main" containerID="cri-o://34bb35492e42e3d61c66948b2dd4613d5da0ab70e77ad27e30c3ddec04fa73f8" gracePeriod=45
Sep 18 18:20:57.861265 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:57.861233    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9bc5b8d4-c09d-4db0-b483-f409d3783a42" path="/var/lib/kubelet/pods/9bc5b8d4-c09d-4db0-b483-f409d3783a42/volumes"
Sep 18 18:20:57.998659 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:57.998633    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-341bd4/sub-test-model-341bd4-kserve-7fd8ccc679-mb8b9"
Sep 18 18:20:58.103142 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:58.103042    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3e05d01-c331-44a9-a8d5-b021cb59c0bf-model-cache\") pod \"e3e05d01-c331-44a9-a8d5-b021cb59c0bf\" (UID: \"e3e05d01-c331-44a9-a8d5-b021cb59c0bf\") "
Sep 18 18:20:58.103142 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:58.103083    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3e05d01-c331-44a9-a8d5-b021cb59c0bf-dshm\") pod \"e3e05d01-c331-44a9-a8d5-b021cb59c0bf\" (UID: \"e3e05d01-c331-44a9-a8d5-b021cb59c0bf\") "
Sep 18 18:20:58.103382 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:58.103162    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3e05d01-c331-44a9-a8d5-b021cb59c0bf-tls-certs\") pod \"e3e05d01-c331-44a9-a8d5-b021cb59c0bf\" (UID: \"e3e05d01-c331-44a9-a8d5-b021cb59c0bf\") "
Sep 18 18:20:58.103382 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:58.103198    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3e05d01-c331-44a9-a8d5-b021cb59c0bf-home\") pod \"e3e05d01-c331-44a9-a8d5-b021cb59c0bf\" (UID: \"e3e05d01-c331-44a9-a8d5-b021cb59c0bf\") "
Sep 18 18:20:58.103382 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:58.103224    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3e05d01-c331-44a9-a8d5-b021cb59c0bf-tmp-dir\") pod \"e3e05d01-c331-44a9-a8d5-b021cb59c0bf\" (UID: \"e3e05d01-c331-44a9-a8d5-b021cb59c0bf\") "
Sep 18 18:20:58.103382 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:58.103299    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mknds\" (UniqueName: \"kubernetes.io/projected/e3e05d01-c331-44a9-a8d5-b021cb59c0bf-kube-api-access-mknds\") pod \"e3e05d01-c331-44a9-a8d5-b021cb59c0bf\" (UID: \"e3e05d01-c331-44a9-a8d5-b021cb59c0bf\") "
Sep 18 18:20:58.103382 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:58.103329    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3e05d01-c331-44a9-a8d5-b021cb59c0bf-model-cache" (OuterVolumeSpecName: "model-cache") pod "e3e05d01-c331-44a9-a8d5-b021cb59c0bf" (UID: "e3e05d01-c331-44a9-a8d5-b021cb59c0bf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:20:58.103601 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:58.103467    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3e05d01-c331-44a9-a8d5-b021cb59c0bf-home" (OuterVolumeSpecName: "home") pod "e3e05d01-c331-44a9-a8d5-b021cb59c0bf" (UID: "e3e05d01-c331-44a9-a8d5-b021cb59c0bf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:20:58.103601 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:58.103500    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3e05d01-c331-44a9-a8d5-b021cb59c0bf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e3e05d01-c331-44a9-a8d5-b021cb59c0bf" (UID: "e3e05d01-c331-44a9-a8d5-b021cb59c0bf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:20:58.103676 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:58.103603    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3e05d01-c331-44a9-a8d5-b021cb59c0bf-model-cache\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:20:58.103676 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:58.103615    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3e05d01-c331-44a9-a8d5-b021cb59c0bf-home\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:20:58.103676 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:58.103624    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3e05d01-c331-44a9-a8d5-b021cb59c0bf-tmp-dir\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:20:58.105288 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:58.105265    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3e05d01-c331-44a9-a8d5-b021cb59c0bf-dshm" (OuterVolumeSpecName: "dshm") pod "e3e05d01-c331-44a9-a8d5-b021cb59c0bf" (UID: "e3e05d01-c331-44a9-a8d5-b021cb59c0bf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:20:58.105390 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:58.105305    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3e05d01-c331-44a9-a8d5-b021cb59c0bf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e3e05d01-c331-44a9-a8d5-b021cb59c0bf" (UID: "e3e05d01-c331-44a9-a8d5-b021cb59c0bf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:20:58.105390 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:58.105331    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3e05d01-c331-44a9-a8d5-b021cb59c0bf-kube-api-access-mknds" (OuterVolumeSpecName: "kube-api-access-mknds") pod "e3e05d01-c331-44a9-a8d5-b021cb59c0bf" (UID: "e3e05d01-c331-44a9-a8d5-b021cb59c0bf"). InnerVolumeSpecName "kube-api-access-mknds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:20:58.204336 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:58.204306    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mknds\" (UniqueName: \"kubernetes.io/projected/e3e05d01-c331-44a9-a8d5-b021cb59c0bf-kube-api-access-mknds\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:20:58.204336 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:58.204332    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3e05d01-c331-44a9-a8d5-b021cb59c0bf-dshm\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:20:58.204336 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:58.204342    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3e05d01-c331-44a9-a8d5-b021cb59c0bf-tls-certs\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:20:58.354963 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:58.354872    2561 generic.go:356] "Generic (PLEG): container finished" podID="e3e05d01-c331-44a9-a8d5-b021cb59c0bf" containerID="34bb35492e42e3d61c66948b2dd4613d5da0ab70e77ad27e30c3ddec04fa73f8" exitCode=0
Sep 18 18:20:58.354963 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:58.354919    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-341bd4/sub-test-model-341bd4-kserve-7fd8ccc679-mb8b9"
Sep 18 18:20:58.355227 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:58.354956    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-341bd4/sub-test-model-341bd4-kserve-7fd8ccc679-mb8b9" event={"ID":"e3e05d01-c331-44a9-a8d5-b021cb59c0bf","Type":"ContainerDied","Data":"34bb35492e42e3d61c66948b2dd4613d5da0ab70e77ad27e30c3ddec04fa73f8"}
Sep 18 18:20:58.355227 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:58.354994    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-341bd4/sub-test-model-341bd4-kserve-7fd8ccc679-mb8b9" event={"ID":"e3e05d01-c331-44a9-a8d5-b021cb59c0bf","Type":"ContainerDied","Data":"11210cf2b79b3a40809f09b3a8a95004e259e5c8a3e3edc4a926f5d536f08230"}
Sep 18 18:20:58.355227 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:58.355012    2561 scope.go:117] "RemoveContainer" containerID="34bb35492e42e3d61c66948b2dd4613d5da0ab70e77ad27e30c3ddec04fa73f8"
Sep 18 18:20:58.364460 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:58.364444    2561 scope.go:117] "RemoveContainer" containerID="34bb35492e42e3d61c66948b2dd4613d5da0ab70e77ad27e30c3ddec04fa73f8"
Sep 18 18:20:58.364722 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:20:58.364704    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34bb35492e42e3d61c66948b2dd4613d5da0ab70e77ad27e30c3ddec04fa73f8\": container with ID starting with 34bb35492e42e3d61c66948b2dd4613d5da0ab70e77ad27e30c3ddec04fa73f8 not found: ID does not exist" containerID="34bb35492e42e3d61c66948b2dd4613d5da0ab70e77ad27e30c3ddec04fa73f8"
Sep 18 18:20:58.364772 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:58.364732    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34bb35492e42e3d61c66948b2dd4613d5da0ab70e77ad27e30c3ddec04fa73f8"} err="failed to get container status \"34bb35492e42e3d61c66948b2dd4613d5da0ab70e77ad27e30c3ddec04fa73f8\": rpc error: code = NotFound desc = could not find container \"34bb35492e42e3d61c66948b2dd4613d5da0ab70e77ad27e30c3ddec04fa73f8\": container with ID starting with 34bb35492e42e3d61c66948b2dd4613d5da0ab70e77ad27e30c3ddec04fa73f8 not found: ID does not exist"
Sep 18 18:20:58.375379 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:58.375354    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-341bd4/sub-test-model-341bd4-kserve-7fd8ccc679-mb8b9"]
Sep 18 18:20:58.380972 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:58.380953    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-341bd4/sub-test-model-341bd4-kserve-7fd8ccc679-mb8b9"]
Sep 18 18:20:59.861565 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:20:59.861531    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3e05d01-c331-44a9-a8d5-b021cb59c0bf" path="/var/lib/kubelet/pods/e3e05d01-c331-44a9-a8d5-b021cb59c0bf/volumes"
Sep 18 18:21:09.810925 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:21:09.810892    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:21:20.929333 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:21:20.929256    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-341bd4/rate-test-model-341bd4-kserve-55c7b677f9-f7dgb"]
Sep 18 18:21:20.929816 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:21:20.929798    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bc5b8d4-c09d-4db0-b483-f409d3783a42" containerName="main"
Sep 18 18:21:20.929870 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:21:20.929820    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bc5b8d4-c09d-4db0-b483-f409d3783a42" containerName="main"
Sep 18 18:21:20.929870 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:21:20.929839    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3e05d01-c331-44a9-a8d5-b021cb59c0bf" containerName="main"
Sep 18 18:21:20.929870 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:21:20.929845    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3e05d01-c331-44a9-a8d5-b021cb59c0bf" containerName="main"
Sep 18 18:21:20.929870 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:21:20.929857    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="943841a0-be31-472c-942b-b3ffa4b7c8b8" containerName="test-curl-7602-d184bf"
Sep 18 18:21:20.929870 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:21:20.929863    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="943841a0-be31-472c-942b-b3ffa4b7c8b8" containerName="test-curl-7602-d184bf"
Sep 18 18:21:20.930035 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:21:20.929927    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="943841a0-be31-472c-942b-b3ffa4b7c8b8" containerName="test-curl-7602-d184bf"
Sep 18 18:21:20.930035 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:21:20.929935    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bc5b8d4-c09d-4db0-b483-f409d3783a42" containerName="main"
Sep 18 18:21:20.930035 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:21:20.929943    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3e05d01-c331-44a9-a8d5-b021cb59c0bf" containerName="main"
Sep 18 18:21:20.939798 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:21:20.939771    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-341bd4/rate-test-model-341bd4-kserve-55c7b677f9-f7dgb"
Sep 18 18:21:20.942790 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:21:20.942758    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-341bd4\"/\"rate-test-model-341bd4-kserve-self-signed-certs\""
Sep 18 18:21:20.942926 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:21:20.942808    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-341bd4\"/\"openshift-service-ca.crt\""
Sep 18 18:21:20.942926 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:21:20.942852    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-341bd4\"/\"kube-root-ca.crt\""
Sep 18 18:21:20.943499 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:21:20.943481    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-341bd4\"/\"default-dockercfg-j284w\""
Sep 18 18:21:20.944748 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:21:20.944713    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-341bd4/rate-test-model-341bd4-kserve-55c7b677f9-f7dgb"]
Sep 18 18:21:21.019218 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:21:21.019187    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:21:21.028299 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:21:21.028275    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65d873b4-882a-47c8-b076-10fdbec49ae7-model-cache\") pod \"rate-test-model-341bd4-kserve-55c7b677f9-f7dgb\" (UID: \"65d873b4-882a-47c8-b076-10fdbec49ae7\") " pod="ai-tenant-e2e-shared-b-w3-341bd4/rate-test-model-341bd4-kserve-55c7b677f9-f7dgb"
Sep 18 18:21:21.028401 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:21:21.028306    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65d873b4-882a-47c8-b076-10fdbec49ae7-dshm\") pod \"rate-test-model-341bd4-kserve-55c7b677f9-f7dgb\" (UID: \"65d873b4-882a-47c8-b076-10fdbec49ae7\") " pod="ai-tenant-e2e-shared-b-w3-341bd4/rate-test-model-341bd4-kserve-55c7b677f9-f7dgb"
Sep 18 18:21:21.028401 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:21:21.028332    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65d873b4-882a-47c8-b076-10fdbec49ae7-home\") pod \"rate-test-model-341bd4-kserve-55c7b677f9-f7dgb\" (UID: \"65d873b4-882a-47c8-b076-10fdbec49ae7\") " pod="ai-tenant-e2e-shared-b-w3-341bd4/rate-test-model-341bd4-kserve-55c7b677f9-f7dgb"
Sep 18 18:21:21.028472 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:21:21.028430    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65d873b4-882a-47c8-b076-10fdbec49ae7-tmp-dir\") pod \"rate-test-model-341bd4-kserve-55c7b677f9-f7dgb\" (UID: \"65d873b4-882a-47c8-b076-10fdbec49ae7\") " pod="ai-tenant-e2e-shared-b-w3-341bd4/rate-test-model-341bd4-kserve-55c7b677f9-f7dgb"
Sep 18 18:21:21.028472 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:21:21.028462    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65d873b4-882a-47c8-b076-10fdbec49ae7-tls-certs\") pod \"rate-test-model-341bd4-kserve-55c7b677f9-f7dgb\" (UID: \"65d873b4-882a-47c8-b076-10fdbec49ae7\") " pod="ai-tenant-e2e-shared-b-w3-341bd4/rate-test-model-341bd4-kserve-55c7b677f9-f7dgb"
Sep 18 18:21:21.028541 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:21:21.028497    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvk2g\" (UniqueName: \"kubernetes.io/projected/65d873b4-882a-47c8-b076-10fdbec49ae7-kube-api-access-rvk2g\") pod \"rate-test-model-341bd4-kserve-55c7b677f9-f7dgb\" (UID: \"65d873b4-882a-47c8-b076-10fdbec49ae7\") " pod="ai-tenant-e2e-shared-b-w3-341bd4/rate-test-model-341bd4-kserve-55c7b677f9-f7dgb"
Sep 18 18:21:21.129102 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:21:21.129069    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65d873b4-882a-47c8-b076-10fdbec49ae7-model-cache\") pod \"rate-test-model-341bd4-kserve-55c7b677f9-f7dgb\" (UID: \"65d873b4-882a-47c8-b076-10fdbec49ae7\") " pod="ai-tenant-e2e-shared-b-w3-341bd4/rate-test-model-341bd4-kserve-55c7b677f9-f7dgb"
Sep 18 18:21:21.129102 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:21:21.129102    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65d873b4-882a-47c8-b076-10fdbec49ae7-dshm\") pod \"rate-test-model-341bd4-kserve-55c7b677f9-f7dgb\" (UID: \"65d873b4-882a-47c8-b076-10fdbec49ae7\") " pod="ai-tenant-e2e-shared-b-w3-341bd4/rate-test-model-341bd4-kserve-55c7b677f9-f7dgb"
Sep 18 18:21:21.129344 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:21:21.129143    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65d873b4-882a-47c8-b076-10fdbec49ae7-home\") pod \"rate-test-model-341bd4-kserve-55c7b677f9-f7dgb\" (UID: \"65d873b4-882a-47c8-b076-10fdbec49ae7\") " pod="ai-tenant-e2e-shared-b-w3-341bd4/rate-test-model-341bd4-kserve-55c7b677f9-f7dgb"
Sep 18 18:21:21.129344 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:21:21.129174    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65d873b4-882a-47c8-b076-10fdbec49ae7-tmp-dir\") pod \"rate-test-model-341bd4-kserve-55c7b677f9-f7dgb\" (UID: \"65d873b4-882a-47c8-b076-10fdbec49ae7\") " pod="ai-tenant-e2e-shared-b-w3-341bd4/rate-test-model-341bd4-kserve-55c7b677f9-f7dgb"
Sep 18 18:21:21.129344 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:21:21.129195    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65d873b4-882a-47c8-b076-10fdbec49ae7-tls-certs\") pod \"rate-test-model-341bd4-kserve-55c7b677f9-f7dgb\" (UID: \"65d873b4-882a-47c8-b076-10fdbec49ae7\") " pod="ai-tenant-e2e-shared-b-w3-341bd4/rate-test-model-341bd4-kserve-55c7b677f9-f7dgb"
Sep 18 18:21:21.129344 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:21:21.129238    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvk2g\" (UniqueName: \"kubernetes.io/projected/65d873b4-882a-47c8-b076-10fdbec49ae7-kube-api-access-rvk2g\") pod \"rate-test-model-341bd4-kserve-55c7b677f9-f7dgb\" (UID: \"65d873b4-882a-47c8-b076-10fdbec49ae7\") " pod="ai-tenant-e2e-shared-b-w3-341bd4/rate-test-model-341bd4-kserve-55c7b677f9-f7dgb"
Sep 18 18:21:21.129579 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:21:21.129496    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65d873b4-882a-47c8-b076-10fdbec49ae7-model-cache\") pod \"rate-test-model-341bd4-kserve-55c7b677f9-f7dgb\" (UID: \"65d873b4-882a-47c8-b076-10fdbec49ae7\") " pod="ai-tenant-e2e-shared-b-w3-341bd4/rate-test-model-341bd4-kserve-55c7b677f9-f7dgb"
Sep 18 18:21:21.129579 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:21:21.129516    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65d873b4-882a-47c8-b076-10fdbec49ae7-tmp-dir\") pod \"rate-test-model-341bd4-kserve-55c7b677f9-f7dgb\" (UID: \"65d873b4-882a-47c8-b076-10fdbec49ae7\") " pod="ai-tenant-e2e-shared-b-w3-341bd4/rate-test-model-341bd4-kserve-55c7b677f9-f7dgb"
Sep 18 18:21:21.129665 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:21:21.129651    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65d873b4-882a-47c8-b076-10fdbec49ae7-home\") pod \"rate-test-model-341bd4-kserve-55c7b677f9-f7dgb\" (UID: \"65d873b4-882a-47c8-b076-10fdbec49ae7\") " pod="ai-tenant-e2e-shared-b-w3-341bd4/rate-test-model-341bd4-kserve-55c7b677f9-f7dgb"
Sep 18 18:21:21.131423 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:21:21.131393    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65d873b4-882a-47c8-b076-10fdbec49ae7-dshm\") pod \"rate-test-model-341bd4-kserve-55c7b677f9-f7dgb\" (UID: \"65d873b4-882a-47c8-b076-10fdbec49ae7\") " pod="ai-tenant-e2e-shared-b-w3-341bd4/rate-test-model-341bd4-kserve-55c7b677f9-f7dgb"
Sep 18 18:21:21.131670 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:21:21.131651    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65d873b4-882a-47c8-b076-10fdbec49ae7-tls-certs\") pod \"rate-test-model-341bd4-kserve-55c7b677f9-f7dgb\" (UID: \"65d873b4-882a-47c8-b076-10fdbec49ae7\") " pod="ai-tenant-e2e-shared-b-w3-341bd4/rate-test-model-341bd4-kserve-55c7b677f9-f7dgb"
Sep 18 18:21:21.137522 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:21:21.137502    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvk2g\" (UniqueName: \"kubernetes.io/projected/65d873b4-882a-47c8-b076-10fdbec49ae7-kube-api-access-rvk2g\") pod \"rate-test-model-341bd4-kserve-55c7b677f9-f7dgb\" (UID: \"65d873b4-882a-47c8-b076-10fdbec49ae7\") " pod="ai-tenant-e2e-shared-b-w3-341bd4/rate-test-model-341bd4-kserve-55c7b677f9-f7dgb"
Sep 18 18:21:21.250965 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:21:21.250886    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-341bd4/rate-test-model-341bd4-kserve-55c7b677f9-f7dgb"
Sep 18 18:21:21.521798 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:21:21.521775    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-341bd4/rate-test-model-341bd4-kserve-55c7b677f9-f7dgb"]
Sep 18 18:21:21.524503 ip-10-0-140-54 kubenswrapper[2561]: W0918 18:21:21.524470    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod65d873b4_882a_47c8_b076_10fdbec49ae7.slice/crio-3eeabe6f9a234bfae5920bc6c414c33bd8dcf691ae535aa823915ba40ddc7be3 WatchSource:0}: Error finding container 3eeabe6f9a234bfae5920bc6c414c33bd8dcf691ae535aa823915ba40ddc7be3: Status 404 returned error can't find the container with id 3eeabe6f9a234bfae5920bc6c414c33bd8dcf691ae535aa823915ba40ddc7be3
Sep 18 18:21:22.478542 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:21:22.478508    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-341bd4/rate-test-model-341bd4-kserve-55c7b677f9-f7dgb" event={"ID":"65d873b4-882a-47c8-b076-10fdbec49ae7","Type":"ContainerStarted","Data":"775046462f5d5572a55e098304aab8dd0c7941035813748d69f1e0fbbc9b7719"}
Sep 18 18:21:22.478923 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:21:22.478554    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-341bd4/rate-test-model-341bd4-kserve-55c7b677f9-f7dgb" event={"ID":"65d873b4-882a-47c8-b076-10fdbec49ae7","Type":"ContainerStarted","Data":"3eeabe6f9a234bfae5920bc6c414c33bd8dcf691ae535aa823915ba40ddc7be3"}
Sep 18 18:21:22.495912 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:21:22.495860    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-341bd4/rate-test-model-341bd4-kserve-55c7b677f9-f7dgb" podStartSLOduration=2.495845084 podStartE2EDuration="2.495845084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:21:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:21:22.494880778 +0000 UTC m=+1887.122012400" watchObservedRunningTime="2026-09-18 18:21:22.495845084 +0000 UTC m=+1887.122976702"
Sep 18 18:21:31.251838 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:21:31.251807    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-341bd4/rate-test-model-341bd4-kserve-55c7b677f9-f7dgb"
Sep 18 18:21:31.252228 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:21:31.252151    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-341bd4/rate-test-model-341bd4-kserve-55c7b677f9-f7dgb"
Sep 18 18:21:31.264308 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:21:31.264286    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-341bd4/rate-test-model-341bd4-kserve-55c7b677f9-f7dgb"
Sep 18 18:21:31.535449 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:21:31.535362    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-341bd4/rate-test-model-341bd4-kserve-55c7b677f9-f7dgb"
Sep 18 18:21:40.373556 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:21:40.373519    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:21:55.963569 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:21:55.963542    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:21:56.935398 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:21:56.935366    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-341bd4/rate-test-model-341bd4-kserve-55c7b677f9-f7dgb"]
Sep 18 18:21:56.955298 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:21:56.955271    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:21:57.370780 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:21:57.370695    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:21:57.696026 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:21:57.695977    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:21:57.792196 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:21:57.792161    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:22:00.961265 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:00.961234    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:22:01.312698 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:01.312613    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:22:01.412941 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:01.412911    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:22:01.904521 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:01.904489    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:22:02.636314 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:02.636273    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-66c663-74bd6d9bf6-m5k85"]
Sep 18 18:22:02.639937 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:02.639906    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-66c663-74bd6d9bf6-m5k85"
Sep 18 18:22:02.643310 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:02.643272    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-66c663-serving-cert\""
Sep 18 18:22:02.655579 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:02.655552    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-66c663-74bd6d9bf6-m5k85"]
Sep 18 18:22:02.685689 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:02.685652    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:22:02.713242 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:02.713218    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c49vr\" (UniqueName: \"kubernetes.io/projected/acf839df-86e6-47e0-9cd8-c2d155ae6d6a-kube-api-access-c49vr\") pod \"maas-api-e2e-ipp-a-66c663-74bd6d9bf6-m5k85\" (UID: \"acf839df-86e6-47e0-9cd8-c2d155ae6d6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-66c663-74bd6d9bf6-m5k85"
Sep 18 18:22:02.713369 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:02.713353    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acf839df-86e6-47e0-9cd8-c2d155ae6d6a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-66c663-74bd6d9bf6-m5k85\" (UID: \"acf839df-86e6-47e0-9cd8-c2d155ae6d6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-66c663-74bd6d9bf6-m5k85"
Sep 18 18:22:02.713431 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:02.713398    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acf839df-86e6-47e0-9cd8-c2d155ae6d6a-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-66c663-74bd6d9bf6-m5k85\" (UID: \"acf839df-86e6-47e0-9cd8-c2d155ae6d6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-66c663-74bd6d9bf6-m5k85"
Sep 18 18:22:02.713431 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:02.713420    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acf839df-86e6-47e0-9cd8-c2d155ae6d6a-metrics-certs\") pod \"maas-api-e2e-ipp-a-66c663-74bd6d9bf6-m5k85\" (UID: \"acf839df-86e6-47e0-9cd8-c2d155ae6d6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-66c663-74bd6d9bf6-m5k85"
Sep 18 18:22:02.784290 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:02.784259    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:22:02.814398 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:02.814376    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acf839df-86e6-47e0-9cd8-c2d155ae6d6a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-66c663-74bd6d9bf6-m5k85\" (UID: \"acf839df-86e6-47e0-9cd8-c2d155ae6d6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-66c663-74bd6d9bf6-m5k85"
Sep 18 18:22:02.814513 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:02.814413    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acf839df-86e6-47e0-9cd8-c2d155ae6d6a-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-66c663-74bd6d9bf6-m5k85\" (UID: \"acf839df-86e6-47e0-9cd8-c2d155ae6d6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-66c663-74bd6d9bf6-m5k85"
Sep 18 18:22:02.814513 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:02.814432    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acf839df-86e6-47e0-9cd8-c2d155ae6d6a-metrics-certs\") pod \"maas-api-e2e-ipp-a-66c663-74bd6d9bf6-m5k85\" (UID: \"acf839df-86e6-47e0-9cd8-c2d155ae6d6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-66c663-74bd6d9bf6-m5k85"
Sep 18 18:22:02.814513 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:02.814466    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c49vr\" (UniqueName: \"kubernetes.io/projected/acf839df-86e6-47e0-9cd8-c2d155ae6d6a-kube-api-access-c49vr\") pod \"maas-api-e2e-ipp-a-66c663-74bd6d9bf6-m5k85\" (UID: \"acf839df-86e6-47e0-9cd8-c2d155ae6d6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-66c663-74bd6d9bf6-m5k85"
Sep 18 18:22:02.814643 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:22:02.814525    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-66c663-serving-cert: secret "maas-api-e2e-ipp-a-66c663-serving-cert" not found
Sep 18 18:22:02.814643 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:22:02.814605    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acf839df-86e6-47e0-9cd8-c2d155ae6d6a-maas-api-tls podName:acf839df-86e6-47e0-9cd8-c2d155ae6d6a nodeName:}" failed. No retries permitted until 2026-09-18 18:22:03.314586959 +0000 UTC m=+1927.941718557 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/acf839df-86e6-47e0-9cd8-c2d155ae6d6a-maas-api-tls") pod "maas-api-e2e-ipp-a-66c663-74bd6d9bf6-m5k85" (UID: "acf839df-86e6-47e0-9cd8-c2d155ae6d6a") : secret "maas-api-e2e-ipp-a-66c663-serving-cert" not found
Sep 18 18:22:02.815176 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:02.815155    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acf839df-86e6-47e0-9cd8-c2d155ae6d6a-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-66c663-74bd6d9bf6-m5k85\" (UID: \"acf839df-86e6-47e0-9cd8-c2d155ae6d6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-66c663-74bd6d9bf6-m5k85"
Sep 18 18:22:02.816882 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:02.816851    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acf839df-86e6-47e0-9cd8-c2d155ae6d6a-metrics-certs\") pod \"maas-api-e2e-ipp-a-66c663-74bd6d9bf6-m5k85\" (UID: \"acf839df-86e6-47e0-9cd8-c2d155ae6d6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-66c663-74bd6d9bf6-m5k85"
Sep 18 18:22:02.823162 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:02.823137    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c49vr\" (UniqueName: \"kubernetes.io/projected/acf839df-86e6-47e0-9cd8-c2d155ae6d6a-kube-api-access-c49vr\") pod \"maas-api-e2e-ipp-a-66c663-74bd6d9bf6-m5k85\" (UID: \"acf839df-86e6-47e0-9cd8-c2d155ae6d6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-66c663-74bd6d9bf6-m5k85"
Sep 18 18:22:02.953574 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:02.953548    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:22:03.155844 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:03.155805    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:22:03.243883 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:03.243796    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:22:03.289864 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:03.289827    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-66c663-f7dc8b74f-tvnq5"]
Sep 18 18:22:03.293727 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:03.293704    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-66c663-f7dc8b74f-tvnq5"
Sep 18 18:22:03.296166 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:03.296144    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-66c663-dockercfg-5zpjl\""
Sep 18 18:22:03.296273 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:03.296148    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-66c663\""
Sep 18 18:22:03.302960 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:03.302939    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-66c663-f7dc8b74f-tvnq5"]
Sep 18 18:22:03.319551 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:03.319531    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acf839df-86e6-47e0-9cd8-c2d155ae6d6a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-66c663-74bd6d9bf6-m5k85\" (UID: \"acf839df-86e6-47e0-9cd8-c2d155ae6d6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-66c663-74bd6d9bf6-m5k85"
Sep 18 18:22:03.321863 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:03.321843    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acf839df-86e6-47e0-9cd8-c2d155ae6d6a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-66c663-74bd6d9bf6-m5k85\" (UID: \"acf839df-86e6-47e0-9cd8-c2d155ae6d6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-66c663-74bd6d9bf6-m5k85"
Sep 18 18:22:03.420128 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:03.420080    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klj24\" (UniqueName: \"kubernetes.io/projected/d517f3ac-da19-4bef-bf4d-23aef7d01c29-kube-api-access-klj24\") pod \"payload-processing-e2e-ipp-a-66c663-f7dc8b74f-tvnq5\" (UID: \"d517f3ac-da19-4bef-bf4d-23aef7d01c29\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-66c663-f7dc8b74f-tvnq5"
Sep 18 18:22:03.420299 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:03.420212    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d517f3ac-da19-4bef-bf4d-23aef7d01c29-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-66c663-f7dc8b74f-tvnq5\" (UID: \"d517f3ac-da19-4bef-bf4d-23aef7d01c29\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-66c663-f7dc8b74f-tvnq5"
Sep 18 18:22:03.520992 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:03.520911    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klj24\" (UniqueName: \"kubernetes.io/projected/d517f3ac-da19-4bef-bf4d-23aef7d01c29-kube-api-access-klj24\") pod \"payload-processing-e2e-ipp-a-66c663-f7dc8b74f-tvnq5\" (UID: \"d517f3ac-da19-4bef-bf4d-23aef7d01c29\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-66c663-f7dc8b74f-tvnq5"
Sep 18 18:22:03.520992 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:03.520964    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d517f3ac-da19-4bef-bf4d-23aef7d01c29-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-66c663-f7dc8b74f-tvnq5\" (UID: \"d517f3ac-da19-4bef-bf4d-23aef7d01c29\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-66c663-f7dc8b74f-tvnq5"
Sep 18 18:22:03.521647 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:03.521629    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d517f3ac-da19-4bef-bf4d-23aef7d01c29-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-66c663-f7dc8b74f-tvnq5\" (UID: \"d517f3ac-da19-4bef-bf4d-23aef7d01c29\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-66c663-f7dc8b74f-tvnq5"
Sep 18 18:22:03.529906 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:03.529884    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-klj24\" (UniqueName: \"kubernetes.io/projected/d517f3ac-da19-4bef-bf4d-23aef7d01c29-kube-api-access-klj24\") pod \"payload-processing-e2e-ipp-a-66c663-f7dc8b74f-tvnq5\" (UID: \"d517f3ac-da19-4bef-bf4d-23aef7d01c29\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-66c663-f7dc8b74f-tvnq5"
Sep 18 18:22:03.550768 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:03.550737    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-66c663-74bd6d9bf6-m5k85"
Sep 18 18:22:03.605495 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:03.605411    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-66c663-f7dc8b74f-tvnq5"
Sep 18 18:22:03.824503 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:03.824462    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-66c663-74bd6d9bf6-m5k85"]
Sep 18 18:22:03.826896 ip-10-0-140-54 kubenswrapper[2561]: W0918 18:22:03.826863    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacf839df_86e6_47e0_9cd8_c2d155ae6d6a.slice/crio-cb31fe1c6053a4784dd3dcf9cd79c5097b71e9a760100bc41403caa4fcd63d9e WatchSource:0}: Error finding container cb31fe1c6053a4784dd3dcf9cd79c5097b71e9a760100bc41403caa4fcd63d9e: Status 404 returned error can't find the container with id cb31fe1c6053a4784dd3dcf9cd79c5097b71e9a760100bc41403caa4fcd63d9e
Sep 18 18:22:03.828715 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:03.828694    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 18:22:03.883598 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:03.883576    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-66c663-f7dc8b74f-tvnq5"]
Sep 18 18:22:03.885724 ip-10-0-140-54 kubenswrapper[2561]: W0918 18:22:03.885702    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd517f3ac_da19_4bef_bf4d_23aef7d01c29.slice/crio-3558d3f8d1757eebdb472a5b78a5b5f21cd39618f5b6e5d47e5d557a5bcae355 WatchSource:0}: Error finding container 3558d3f8d1757eebdb472a5b78a5b5f21cd39618f5b6e5d47e5d557a5bcae355: Status 404 returned error can't find the container with id 3558d3f8d1757eebdb472a5b78a5b5f21cd39618f5b6e5d47e5d557a5bcae355
Sep 18 18:22:04.689548 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:04.689510    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-66c663-74bd6d9bf6-m5k85" event={"ID":"acf839df-86e6-47e0-9cd8-c2d155ae6d6a","Type":"ContainerStarted","Data":"9fcce50571b6a5dd670e226fbc26b92ed7da1f29afb7355faf1f2f3e77e8d0a8"}
Sep 18 18:22:04.689752 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:04.689571    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-66c663-74bd6d9bf6-m5k85" event={"ID":"acf839df-86e6-47e0-9cd8-c2d155ae6d6a","Type":"ContainerStarted","Data":"cb31fe1c6053a4784dd3dcf9cd79c5097b71e9a760100bc41403caa4fcd63d9e"}
Sep 18 18:22:04.689752 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:04.689599    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-66c663-74bd6d9bf6-m5k85"
Sep 18 18:22:04.691174 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:04.691149    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-66c663-f7dc8b74f-tvnq5" event={"ID":"d517f3ac-da19-4bef-bf4d-23aef7d01c29","Type":"ContainerStarted","Data":"c1751f4ac2a8ebf593e57cfd7d1d22b4c5adb3849901ed8d5468b2f980e4ed71"}
Sep 18 18:22:04.691289 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:04.691189    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-66c663-f7dc8b74f-tvnq5" event={"ID":"d517f3ac-da19-4bef-bf4d-23aef7d01c29","Type":"ContainerStarted","Data":"3558d3f8d1757eebdb472a5b78a5b5f21cd39618f5b6e5d47e5d557a5bcae355"}
Sep 18 18:22:04.691289 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:04.691210    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-66c663-f7dc8b74f-tvnq5"
Sep 18 18:22:04.705733 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:04.705677    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-66c663-74bd6d9bf6-m5k85" podStartSLOduration=2.281761809 podStartE2EDuration="2.705659796s" totalImagesPullingTime="423.897987ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:22:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:22:04.704959264 +0000 UTC m=+1929.332090884" watchObservedRunningTime="2026-09-18 18:22:04.705659796 +0000 UTC m=+1929.332791418"
Sep 18 18:22:04.721933 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:04.721889    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-66c663-f7dc8b74f-tvnq5" podStartSLOduration=1.72187489 podStartE2EDuration="1.72187489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:22:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:22:04.720836084 +0000 UTC m=+1929.347967705" watchObservedRunningTime="2026-09-18 18:22:04.72187489 +0000 UTC m=+1929.349006509"
Sep 18 18:22:10.702431 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:10.702401    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-66c663-74bd6d9bf6-m5k85"
Sep 18 18:22:11.956559 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:11.956518    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-341bd4/rate-test-model-341bd4-kserve-55c7b677f9-f7dgb" podUID="65d873b4-882a-47c8-b076-10fdbec49ae7" containerName="main" containerID="cri-o://775046462f5d5572a55e098304aab8dd0c7941035813748d69f1e0fbbc9b7719" gracePeriod=45
Sep 18 18:22:12.201664 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:12.201636    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-341bd4/rate-test-model-341bd4-kserve-55c7b677f9-f7dgb"
Sep 18 18:22:12.303468 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:12.303385    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65d873b4-882a-47c8-b076-10fdbec49ae7-dshm\") pod \"65d873b4-882a-47c8-b076-10fdbec49ae7\" (UID: \"65d873b4-882a-47c8-b076-10fdbec49ae7\") "
Sep 18 18:22:12.303468 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:12.303431    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rvk2g\" (UniqueName: \"kubernetes.io/projected/65d873b4-882a-47c8-b076-10fdbec49ae7-kube-api-access-rvk2g\") pod \"65d873b4-882a-47c8-b076-10fdbec49ae7\" (UID: \"65d873b4-882a-47c8-b076-10fdbec49ae7\") "
Sep 18 18:22:12.303708 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:12.303497    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65d873b4-882a-47c8-b076-10fdbec49ae7-tmp-dir\") pod \"65d873b4-882a-47c8-b076-10fdbec49ae7\" (UID: \"65d873b4-882a-47c8-b076-10fdbec49ae7\") "
Sep 18 18:22:12.303708 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:12.303521    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65d873b4-882a-47c8-b076-10fdbec49ae7-home\") pod \"65d873b4-882a-47c8-b076-10fdbec49ae7\" (UID: \"65d873b4-882a-47c8-b076-10fdbec49ae7\") "
Sep 18 18:22:12.303708 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:12.303545    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65d873b4-882a-47c8-b076-10fdbec49ae7-tls-certs\") pod \"65d873b4-882a-47c8-b076-10fdbec49ae7\" (UID: \"65d873b4-882a-47c8-b076-10fdbec49ae7\") "
Sep 18 18:22:12.303708 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:12.303588    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65d873b4-882a-47c8-b076-10fdbec49ae7-model-cache\") pod \"65d873b4-882a-47c8-b076-10fdbec49ae7\" (UID: \"65d873b4-882a-47c8-b076-10fdbec49ae7\") "
Sep 18 18:22:12.303906 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:12.303784    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65d873b4-882a-47c8-b076-10fdbec49ae7-home" (OuterVolumeSpecName: "home") pod "65d873b4-882a-47c8-b076-10fdbec49ae7" (UID: "65d873b4-882a-47c8-b076-10fdbec49ae7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:22:12.303906 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:12.303799    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65d873b4-882a-47c8-b076-10fdbec49ae7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "65d873b4-882a-47c8-b076-10fdbec49ae7" (UID: "65d873b4-882a-47c8-b076-10fdbec49ae7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:22:12.304006 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:12.303935    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65d873b4-882a-47c8-b076-10fdbec49ae7-model-cache" (OuterVolumeSpecName: "model-cache") pod "65d873b4-882a-47c8-b076-10fdbec49ae7" (UID: "65d873b4-882a-47c8-b076-10fdbec49ae7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:22:12.304006 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:12.303955    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65d873b4-882a-47c8-b076-10fdbec49ae7-tmp-dir\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:22:12.304006 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:12.303988    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65d873b4-882a-47c8-b076-10fdbec49ae7-home\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:22:12.305571 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:12.305539    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65d873b4-882a-47c8-b076-10fdbec49ae7-dshm" (OuterVolumeSpecName: "dshm") pod "65d873b4-882a-47c8-b076-10fdbec49ae7" (UID: "65d873b4-882a-47c8-b076-10fdbec49ae7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:22:12.305687 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:12.305606    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65d873b4-882a-47c8-b076-10fdbec49ae7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "65d873b4-882a-47c8-b076-10fdbec49ae7" (UID: "65d873b4-882a-47c8-b076-10fdbec49ae7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:22:12.305687 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:12.305619    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65d873b4-882a-47c8-b076-10fdbec49ae7-kube-api-access-rvk2g" (OuterVolumeSpecName: "kube-api-access-rvk2g") pod "65d873b4-882a-47c8-b076-10fdbec49ae7" (UID: "65d873b4-882a-47c8-b076-10fdbec49ae7"). InnerVolumeSpecName "kube-api-access-rvk2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:22:12.405479 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:12.405439    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65d873b4-882a-47c8-b076-10fdbec49ae7-tls-certs\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:22:12.405479 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:12.405472    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65d873b4-882a-47c8-b076-10fdbec49ae7-model-cache\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:22:12.405479 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:12.405480    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65d873b4-882a-47c8-b076-10fdbec49ae7-dshm\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:22:12.405479 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:12.405489    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvk2g\" (UniqueName: \"kubernetes.io/projected/65d873b4-882a-47c8-b076-10fdbec49ae7-kube-api-access-rvk2g\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:22:12.736731 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:12.736690    2561 generic.go:356] "Generic (PLEG): container finished" podID="65d873b4-882a-47c8-b076-10fdbec49ae7" containerID="775046462f5d5572a55e098304aab8dd0c7941035813748d69f1e0fbbc9b7719" exitCode=0
Sep 18 18:22:12.736899 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:12.736734    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-341bd4/rate-test-model-341bd4-kserve-55c7b677f9-f7dgb"
Sep 18 18:22:12.736899 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:12.736742    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-341bd4/rate-test-model-341bd4-kserve-55c7b677f9-f7dgb" event={"ID":"65d873b4-882a-47c8-b076-10fdbec49ae7","Type":"ContainerDied","Data":"775046462f5d5572a55e098304aab8dd0c7941035813748d69f1e0fbbc9b7719"}
Sep 18 18:22:12.736899 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:12.736775    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-341bd4/rate-test-model-341bd4-kserve-55c7b677f9-f7dgb" event={"ID":"65d873b4-882a-47c8-b076-10fdbec49ae7","Type":"ContainerDied","Data":"3eeabe6f9a234bfae5920bc6c414c33bd8dcf691ae535aa823915ba40ddc7be3"}
Sep 18 18:22:12.736899 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:12.736790    2561 scope.go:117] "RemoveContainer" containerID="775046462f5d5572a55e098304aab8dd0c7941035813748d69f1e0fbbc9b7719"
Sep 18 18:22:12.746012 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:12.745995    2561 scope.go:117] "RemoveContainer" containerID="775046462f5d5572a55e098304aab8dd0c7941035813748d69f1e0fbbc9b7719"
Sep 18 18:22:12.746305 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:22:12.746287    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"775046462f5d5572a55e098304aab8dd0c7941035813748d69f1e0fbbc9b7719\": container with ID starting with 775046462f5d5572a55e098304aab8dd0c7941035813748d69f1e0fbbc9b7719 not found: ID does not exist" containerID="775046462f5d5572a55e098304aab8dd0c7941035813748d69f1e0fbbc9b7719"
Sep 18 18:22:12.746396 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:12.746311    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"775046462f5d5572a55e098304aab8dd0c7941035813748d69f1e0fbbc9b7719"} err="failed to get container status \"775046462f5d5572a55e098304aab8dd0c7941035813748d69f1e0fbbc9b7719\": rpc error: code = NotFound desc = could not find container \"775046462f5d5572a55e098304aab8dd0c7941035813748d69f1e0fbbc9b7719\": container with ID starting with 775046462f5d5572a55e098304aab8dd0c7941035813748d69f1e0fbbc9b7719 not found: ID does not exist"
Sep 18 18:22:12.757804 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:12.757778    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-341bd4/rate-test-model-341bd4-kserve-55c7b677f9-f7dgb"]
Sep 18 18:22:12.760830 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:12.760804    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-341bd4/rate-test-model-341bd4-kserve-55c7b677f9-f7dgb"]
Sep 18 18:22:13.861621 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:13.861589    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65d873b4-882a-47c8-b076-10fdbec49ae7" path="/var/lib/kubelet/pods/65d873b4-882a-47c8-b076-10fdbec49ae7/volumes"
Sep 18 18:22:15.700072 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:15.700046    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-66c663-f7dc8b74f-tvnq5"
Sep 18 18:22:24.644509 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:24.644454    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:22:25.328942 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:25.328905    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:22:26.104936 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:26.104903    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:22:26.205258 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:26.205221    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:22:26.770413 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:26.770380    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:22:26.871478 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:26.871417    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:22:27.160626 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:27.160596    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:22:27.327512 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:27.327482    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:22:29.736537 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:29.736506    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:22:30.370097 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:30.370058    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:22:30.470289 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:30.470250    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:22:30.628913 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:30.628836    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:22:31.408200 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:31.408153    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:22:32.120368 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:32.120339    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:22:32.923699 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:32.923669    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-949a32-5c4c4d9c45-zrj6t"]
Sep 18 18:22:32.924799 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:32.924769    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65d873b4-882a-47c8-b076-10fdbec49ae7" containerName="main"
Sep 18 18:22:32.924799 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:32.924795    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="65d873b4-882a-47c8-b076-10fdbec49ae7" containerName="main"
Sep 18 18:22:32.925010 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:32.924913    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="65d873b4-882a-47c8-b076-10fdbec49ae7" containerName="main"
Sep 18 18:22:32.927071 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:32.927050    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-949a32-5c4c4d9c45-zrj6t"
Sep 18 18:22:32.929542 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:32.929522    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-949a32-serving-cert\""
Sep 18 18:22:32.940776 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:32.940751    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-949a32-5c4c4d9c45-zrj6t"]
Sep 18 18:22:32.961144 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:32.961097    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:22:33.000456 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:33.000426    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0525c780-301e-4a8c-9896-f34a912e0f3b-metrics-certs\") pod \"maas-api-e2e-ipp-b-949a32-5c4c4d9c45-zrj6t\" (UID: \"0525c780-301e-4a8c-9896-f34a912e0f3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-949a32-5c4c4d9c45-zrj6t"
Sep 18 18:22:33.000594 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:33.000547    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0525c780-301e-4a8c-9896-f34a912e0f3b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-949a32-5c4c4d9c45-zrj6t\" (UID: \"0525c780-301e-4a8c-9896-f34a912e0f3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-949a32-5c4c4d9c45-zrj6t"
Sep 18 18:22:33.000594 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:33.000575    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h6x2\" (UniqueName: \"kubernetes.io/projected/0525c780-301e-4a8c-9896-f34a912e0f3b-kube-api-access-6h6x2\") pod \"maas-api-e2e-ipp-b-949a32-5c4c4d9c45-zrj6t\" (UID: \"0525c780-301e-4a8c-9896-f34a912e0f3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-949a32-5c4c4d9c45-zrj6t"
Sep 18 18:22:33.000697 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:33.000599    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0525c780-301e-4a8c-9896-f34a912e0f3b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-949a32-5c4c4d9c45-zrj6t\" (UID: \"0525c780-301e-4a8c-9896-f34a912e0f3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-949a32-5c4c4d9c45-zrj6t"
Sep 18 18:22:33.060329 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:33.060285    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:22:33.101642 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:33.101611    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0525c780-301e-4a8c-9896-f34a912e0f3b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-949a32-5c4c4d9c45-zrj6t\" (UID: \"0525c780-301e-4a8c-9896-f34a912e0f3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-949a32-5c4c4d9c45-zrj6t"
Sep 18 18:22:33.101818 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:33.101674    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0525c780-301e-4a8c-9896-f34a912e0f3b-metrics-certs\") pod \"maas-api-e2e-ipp-b-949a32-5c4c4d9c45-zrj6t\" (UID: \"0525c780-301e-4a8c-9896-f34a912e0f3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-949a32-5c4c4d9c45-zrj6t"
Sep 18 18:22:33.101818 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:33.101756    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0525c780-301e-4a8c-9896-f34a912e0f3b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-949a32-5c4c4d9c45-zrj6t\" (UID: \"0525c780-301e-4a8c-9896-f34a912e0f3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-949a32-5c4c4d9c45-zrj6t"
Sep 18 18:22:33.101818 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:33.101778    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h6x2\" (UniqueName: \"kubernetes.io/projected/0525c780-301e-4a8c-9896-f34a912e0f3b-kube-api-access-6h6x2\") pod \"maas-api-e2e-ipp-b-949a32-5c4c4d9c45-zrj6t\" (UID: \"0525c780-301e-4a8c-9896-f34a912e0f3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-949a32-5c4c4d9c45-zrj6t"
Sep 18 18:22:33.102000 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:22:33.101896    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-949a32-serving-cert: secret "maas-api-e2e-ipp-b-949a32-serving-cert" not found
Sep 18 18:22:33.102000 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:22:33.101975    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0525c780-301e-4a8c-9896-f34a912e0f3b-maas-api-tls podName:0525c780-301e-4a8c-9896-f34a912e0f3b nodeName:}" failed. No retries permitted until 2026-09-18 18:22:33.601953621 +0000 UTC m=+1958.229085221 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0525c780-301e-4a8c-9896-f34a912e0f3b-maas-api-tls") pod "maas-api-e2e-ipp-b-949a32-5c4c4d9c45-zrj6t" (UID: "0525c780-301e-4a8c-9896-f34a912e0f3b") : secret "maas-api-e2e-ipp-b-949a32-serving-cert" not found
Sep 18 18:22:33.102381 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:33.102358    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0525c780-301e-4a8c-9896-f34a912e0f3b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-949a32-5c4c4d9c45-zrj6t\" (UID: \"0525c780-301e-4a8c-9896-f34a912e0f3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-949a32-5c4c4d9c45-zrj6t"
Sep 18 18:22:33.104004 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:33.103981    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0525c780-301e-4a8c-9896-f34a912e0f3b-metrics-certs\") pod \"maas-api-e2e-ipp-b-949a32-5c4c4d9c45-zrj6t\" (UID: \"0525c780-301e-4a8c-9896-f34a912e0f3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-949a32-5c4c4d9c45-zrj6t"
Sep 18 18:22:33.110038 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:33.110021    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h6x2\" (UniqueName: \"kubernetes.io/projected/0525c780-301e-4a8c-9896-f34a912e0f3b-kube-api-access-6h6x2\") pod \"maas-api-e2e-ipp-b-949a32-5c4c4d9c45-zrj6t\" (UID: \"0525c780-301e-4a8c-9896-f34a912e0f3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-949a32-5c4c4d9c45-zrj6t"
Sep 18 18:22:33.279605 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:33.279530    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:22:33.431505 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:33.431471    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:22:33.531881 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:33.531792    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:22:33.558179 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:33.558151    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-949a32-6c86dfd9c5-8jcxc"]
Sep 18 18:22:33.560987 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:33.560971    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-949a32-6c86dfd9c5-8jcxc"
Sep 18 18:22:33.564891 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:33.564868    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-949a32\""
Sep 18 18:22:33.564999 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:33.564920    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-949a32-dockercfg-bv5v4\""
Sep 18 18:22:33.575991 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:33.575966    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-949a32-6c86dfd9c5-8jcxc"]
Sep 18 18:22:33.604933 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:33.604905    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0525c780-301e-4a8c-9896-f34a912e0f3b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-949a32-5c4c4d9c45-zrj6t\" (UID: \"0525c780-301e-4a8c-9896-f34a912e0f3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-949a32-5c4c4d9c45-zrj6t"
Sep 18 18:22:33.605065 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:33.604948    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mmr8\" (UniqueName: \"kubernetes.io/projected/fdfbb3be-537d-413c-869e-61e01cb0e86c-kube-api-access-6mmr8\") pod \"payload-processing-e2e-ipp-b-949a32-6c86dfd9c5-8jcxc\" (UID: \"fdfbb3be-537d-413c-869e-61e01cb0e86c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-949a32-6c86dfd9c5-8jcxc"
Sep 18 18:22:33.605065 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:33.604994    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdfbb3be-537d-413c-869e-61e01cb0e86c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-949a32-6c86dfd9c5-8jcxc\" (UID: \"fdfbb3be-537d-413c-869e-61e01cb0e86c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-949a32-6c86dfd9c5-8jcxc"
Sep 18 18:22:33.607252 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:33.607232    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0525c780-301e-4a8c-9896-f34a912e0f3b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-949a32-5c4c4d9c45-zrj6t\" (UID: \"0525c780-301e-4a8c-9896-f34a912e0f3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-949a32-5c4c4d9c45-zrj6t"
Sep 18 18:22:33.706046 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:33.706012    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mmr8\" (UniqueName: \"kubernetes.io/projected/fdfbb3be-537d-413c-869e-61e01cb0e86c-kube-api-access-6mmr8\") pod \"payload-processing-e2e-ipp-b-949a32-6c86dfd9c5-8jcxc\" (UID: \"fdfbb3be-537d-413c-869e-61e01cb0e86c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-949a32-6c86dfd9c5-8jcxc"
Sep 18 18:22:33.706194 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:33.706060    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdfbb3be-537d-413c-869e-61e01cb0e86c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-949a32-6c86dfd9c5-8jcxc\" (UID: \"fdfbb3be-537d-413c-869e-61e01cb0e86c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-949a32-6c86dfd9c5-8jcxc"
Sep 18 18:22:33.706799 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:33.706772    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdfbb3be-537d-413c-869e-61e01cb0e86c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-949a32-6c86dfd9c5-8jcxc\" (UID: \"fdfbb3be-537d-413c-869e-61e01cb0e86c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-949a32-6c86dfd9c5-8jcxc"
Sep 18 18:22:33.714534 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:33.714509    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mmr8\" (UniqueName: \"kubernetes.io/projected/fdfbb3be-537d-413c-869e-61e01cb0e86c-kube-api-access-6mmr8\") pod \"payload-processing-e2e-ipp-b-949a32-6c86dfd9c5-8jcxc\" (UID: \"fdfbb3be-537d-413c-869e-61e01cb0e86c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-949a32-6c86dfd9c5-8jcxc"
Sep 18 18:22:33.839140 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:33.839037    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-949a32-5c4c4d9c45-zrj6t"
Sep 18 18:22:33.872730 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:33.872701    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-949a32-6c86dfd9c5-8jcxc"
Sep 18 18:22:34.116085 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:34.116060    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-949a32-5c4c4d9c45-zrj6t"]
Sep 18 18:22:34.118246 ip-10-0-140-54 kubenswrapper[2561]: W0918 18:22:34.118221    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0525c780_301e_4a8c_9896_f34a912e0f3b.slice/crio-0c1e8dfba0ad2a11ab418aab60f016cf5fb2d45dcd9996e1659da4e56fd75cf5 WatchSource:0}: Error finding container 0c1e8dfba0ad2a11ab418aab60f016cf5fb2d45dcd9996e1659da4e56fd75cf5: Status 404 returned error can't find the container with id 0c1e8dfba0ad2a11ab418aab60f016cf5fb2d45dcd9996e1659da4e56fd75cf5
Sep 18 18:22:34.155787 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:34.155761    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-949a32-6c86dfd9c5-8jcxc"]
Sep 18 18:22:34.157384 ip-10-0-140-54 kubenswrapper[2561]: W0918 18:22:34.157354    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdfbb3be_537d_413c_869e_61e01cb0e86c.slice/crio-1a6685044a8437c68399fab24a7a6419deb49b469b7d2f86c617d19d1be9e433 WatchSource:0}: Error finding container 1a6685044a8437c68399fab24a7a6419deb49b469b7d2f86c617d19d1be9e433: Status 404 returned error can't find the container with id 1a6685044a8437c68399fab24a7a6419deb49b469b7d2f86c617d19d1be9e433
Sep 18 18:22:34.849458 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:34.849427    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-949a32-5c4c4d9c45-zrj6t" event={"ID":"0525c780-301e-4a8c-9896-f34a912e0f3b","Type":"ContainerStarted","Data":"7f8e56d9cdf0fdbbc97bbcc5c37a624a820f846c79d304e9a51a08375ea2bb0e"}
Sep 18 18:22:34.849627 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:34.849476    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-949a32-5c4c4d9c45-zrj6t" event={"ID":"0525c780-301e-4a8c-9896-f34a912e0f3b","Type":"ContainerStarted","Data":"0c1e8dfba0ad2a11ab418aab60f016cf5fb2d45dcd9996e1659da4e56fd75cf5"}
Sep 18 18:22:34.849627 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:34.849548    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-949a32-5c4c4d9c45-zrj6t"
Sep 18 18:22:34.851161 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:34.851137    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-949a32-6c86dfd9c5-8jcxc" event={"ID":"fdfbb3be-537d-413c-869e-61e01cb0e86c","Type":"ContainerStarted","Data":"34072393f9b6584222d90b70ddc67b9aac35c2f24c45f61de613143696abe302"}
Sep 18 18:22:34.851275 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:34.851175    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-949a32-6c86dfd9c5-8jcxc" event={"ID":"fdfbb3be-537d-413c-869e-61e01cb0e86c","Type":"ContainerStarted","Data":"1a6685044a8437c68399fab24a7a6419deb49b469b7d2f86c617d19d1be9e433"}
Sep 18 18:22:34.851275 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:34.851196    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-949a32-6c86dfd9c5-8jcxc"
Sep 18 18:22:34.868388 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:34.868342    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-949a32-5c4c4d9c45-zrj6t" podStartSLOduration=2.277746352 podStartE2EDuration="2.86832299s" totalImagesPullingTime="590.576638ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:22:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:22:34.866625618 +0000 UTC m=+1959.493757238" watchObservedRunningTime="2026-09-18 18:22:34.86832299 +0000 UTC m=+1959.495454617"
Sep 18 18:22:34.885347 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:34.885301    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-949a32-6c86dfd9c5-8jcxc" podStartSLOduration=1.885287095 podStartE2EDuration="1.885287095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:22:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:22:34.883867424 +0000 UTC m=+1959.510999045" watchObservedRunningTime="2026-09-18 18:22:34.885287095 +0000 UTC m=+1959.512418714"
Sep 18 18:22:40.863818 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:40.863788    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-949a32-5c4c4d9c45-zrj6t"
Sep 18 18:22:45.861524 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:45.861493    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-949a32-6c86dfd9c5-8jcxc"
Sep 18 18:22:54.832619 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:54.832543    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:22:55.329312 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:55.329246    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:22:55.634310 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:55.634223    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:22:56.671327 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:56.671279    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:22:57.497035 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:57.497001    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:22:58.171299 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:22:58.171268    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:23:00.326533 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:23:00.326498    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-j77pl_d1e487ec-d2f3-45a0-9b0c-0a14e89ea981/payload-processing/0.log"
Sep 18 18:23:00.441609 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:23:00.441581    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-949a32-6c86dfd9c5-8jcxc_fdfbb3be-537d-413c-869e-61e01cb0e86c/payload-processing/0.log"
Sep 18 18:23:09.006732 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:23:09.006704    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:23:16.754259 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:23:16.754187    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:23:36.370195 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:23:36.370162    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-66c663-f7dc8b74f-tvnq5_d517f3ac-da19-4bef-bf4d-23aef7d01c29/payload-processing/0.log"
Sep 18 18:23:36.481049 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:23:36.481022    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-j77pl_d1e487ec-d2f3-45a0-9b0c-0a14e89ea981/payload-processing/0.log"
Sep 18 18:23:36.593484 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:23:36.593457    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-949a32-6c86dfd9c5-8jcxc_fdfbb3be-537d-413c-869e-61e01cb0e86c/payload-processing/0.log"
Sep 18 18:23:36.714991 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:23:36.714967    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-949a32-6c86dfd9c5-8jcxc_fdfbb3be-537d-413c-869e-61e01cb0e86c/payload-processing/0.log"
Sep 18 18:23:36.977741 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:23:36.977659    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:23:37.258612 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:23:37.258536    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:23:37.430820 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:23:37.430791    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:23:39.418215 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:23:39.418178    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:23:40.213193 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:23:40.213164    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:23:40.310936 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:23:40.310903    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:23:41.090695 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:23:41.090663    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:23:41.978799 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:23:41.978764    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:23:42.658810 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:23:42.658780    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:23:43.424402 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:23:43.424368    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:23:43.480919 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:23:43.480887    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:23:43.633871 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:23:43.633835    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:23:43.872864 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:23:43.872795    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:23:43.974251 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:23:43.974218    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:23:44.017623 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:23:44.017594    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-f3eedc-654d65d7c9-d8vt2"]
Sep 18 18:23:44.021341 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:23:44.021324    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f3eedc-654d65d7c9-d8vt2"
Sep 18 18:23:44.023624 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:23:44.023604    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-f3eedc\""
Sep 18 18:23:44.023736 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:23:44.023693    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-f3eedc-dockercfg-q2zl6\""
Sep 18 18:23:44.030224 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:23:44.030202    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-f3eedc-654d65d7c9-d8vt2"]
Sep 18 18:23:44.146508 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:23:44.146478    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4aaee9c-1d5e-4549-9f4e-119114e87e35-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-f3eedc-654d65d7c9-d8vt2\" (UID: \"d4aaee9c-1d5e-4549-9f4e-119114e87e35\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f3eedc-654d65d7c9-d8vt2"
Sep 18 18:23:44.146508 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:23:44.146516    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r89v\" (UniqueName: \"kubernetes.io/projected/d4aaee9c-1d5e-4549-9f4e-119114e87e35-kube-api-access-6r89v\") pod \"payload-processing-e2e-ipp-cleanup-f3eedc-654d65d7c9-d8vt2\" (UID: \"d4aaee9c-1d5e-4549-9f4e-119114e87e35\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f3eedc-654d65d7c9-d8vt2"
Sep 18 18:23:44.247864 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:23:44.247826    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4aaee9c-1d5e-4549-9f4e-119114e87e35-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-f3eedc-654d65d7c9-d8vt2\" (UID: \"d4aaee9c-1d5e-4549-9f4e-119114e87e35\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f3eedc-654d65d7c9-d8vt2"
Sep 18 18:23:44.247864 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:23:44.247862    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r89v\" (UniqueName: \"kubernetes.io/projected/d4aaee9c-1d5e-4549-9f4e-119114e87e35-kube-api-access-6r89v\") pod \"payload-processing-e2e-ipp-cleanup-f3eedc-654d65d7c9-d8vt2\" (UID: \"d4aaee9c-1d5e-4549-9f4e-119114e87e35\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f3eedc-654d65d7c9-d8vt2"
Sep 18 18:23:44.248552 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:23:44.248530    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4aaee9c-1d5e-4549-9f4e-119114e87e35-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-f3eedc-654d65d7c9-d8vt2\" (UID: \"d4aaee9c-1d5e-4549-9f4e-119114e87e35\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f3eedc-654d65d7c9-d8vt2"
Sep 18 18:23:44.255430 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:23:44.255408    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r89v\" (UniqueName: \"kubernetes.io/projected/d4aaee9c-1d5e-4549-9f4e-119114e87e35-kube-api-access-6r89v\") pod \"payload-processing-e2e-ipp-cleanup-f3eedc-654d65d7c9-d8vt2\" (UID: \"d4aaee9c-1d5e-4549-9f4e-119114e87e35\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f3eedc-654d65d7c9-d8vt2"
Sep 18 18:23:44.332631 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:23:44.332596    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f3eedc-654d65d7c9-d8vt2"
Sep 18 18:23:44.603436 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:23:44.603405    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-f3eedc-654d65d7c9-d8vt2"]
Sep 18 18:23:44.606407 ip-10-0-140-54 kubenswrapper[2561]: W0918 18:23:44.606382    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4aaee9c_1d5e_4549_9f4e_119114e87e35.slice/crio-5f47304de88cdc20fb1621a07054442616570b4190d7243b55804e00fcf3833f WatchSource:0}: Error finding container 5f47304de88cdc20fb1621a07054442616570b4190d7243b55804e00fcf3833f: Status 404 returned error can't find the container with id 5f47304de88cdc20fb1621a07054442616570b4190d7243b55804e00fcf3833f
Sep 18 18:23:45.219252 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:23:45.219217    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f3eedc-654d65d7c9-d8vt2" event={"ID":"d4aaee9c-1d5e-4549-9f4e-119114e87e35","Type":"ContainerStarted","Data":"9852780851d2660189eb19bd9453c4ed7b6ffa7dc021818961dd89b986197aa2"}
Sep 18 18:23:45.219725 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:23:45.219266    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f3eedc-654d65d7c9-d8vt2" event={"ID":"d4aaee9c-1d5e-4549-9f4e-119114e87e35","Type":"ContainerStarted","Data":"5f47304de88cdc20fb1621a07054442616570b4190d7243b55804e00fcf3833f"}
Sep 18 18:23:45.219725 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:23:45.219308    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f3eedc-654d65d7c9-d8vt2"
Sep 18 18:23:45.236838 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:23:45.236795    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f3eedc-654d65d7c9-d8vt2" podStartSLOduration=2.236782483 podStartE2EDuration="2.236782483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:23:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:23:45.234496985 +0000 UTC m=+2029.861628611" watchObservedRunningTime="2026-09-18 18:23:45.236782483 +0000 UTC m=+2029.863914103"
Sep 18 18:23:54.888255 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:23:54.888214    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:23:54.936610 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:23:54.936584    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:23:56.226312 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:23:56.226278    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f3eedc-654d65d7c9-d8vt2"
Sep 18 18:23:58.548206 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:23:58.548171    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:04.240318 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:04.240280    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:05.639347 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:05.639309    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:06.190561 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:06.190526    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:06.291947 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:06.291914    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:06.393425 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:06.393394    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:06.932812 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:06.932781    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:07.629622 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:07.629584    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:08.264846 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:08.264809    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:09.048159 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:09.048096    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:10.226712 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:10.226684    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:11.549619 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:11.549586    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:12.346146 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:12.346098    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:13.027367 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:13.027336    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:13.799696 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:13.799668    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:14.428997 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:14.428965    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:15.087928 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:15.087887    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:22.089576 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:22.089502    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:22.130823 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:22.130792    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-f3eedc-654d65d7c9-d8vt2"]
Sep 18 18:24:22.131058 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:22.131031    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f3eedc-654d65d7c9-d8vt2" podUID="d4aaee9c-1d5e-4549-9f4e-119114e87e35" containerName="payload-processing" containerID="cri-o://9852780851d2660189eb19bd9453c4ed7b6ffa7dc021818961dd89b986197aa2" gracePeriod=30
Sep 18 18:24:22.172352 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:22.172324    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:22.273205 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:22.273170    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:22.331670 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:22.331632    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:22.394626 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:22.394598    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f3eedc-654d65d7c9-d8vt2"
Sep 18 18:24:22.415036 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:22.415003    2561 generic.go:356] "Generic (PLEG): container finished" podID="d4aaee9c-1d5e-4549-9f4e-119114e87e35" containerID="9852780851d2660189eb19bd9453c4ed7b6ffa7dc021818961dd89b986197aa2" exitCode=0
Sep 18 18:24:22.415231 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:22.415048    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f3eedc-654d65d7c9-d8vt2" event={"ID":"d4aaee9c-1d5e-4549-9f4e-119114e87e35","Type":"ContainerDied","Data":"9852780851d2660189eb19bd9453c4ed7b6ffa7dc021818961dd89b986197aa2"}
Sep 18 18:24:22.415231 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:22.415096    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f3eedc-654d65d7c9-d8vt2" event={"ID":"d4aaee9c-1d5e-4549-9f4e-119114e87e35","Type":"ContainerDied","Data":"5f47304de88cdc20fb1621a07054442616570b4190d7243b55804e00fcf3833f"}
Sep 18 18:24:22.415231 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:22.415140    2561 scope.go:117] "RemoveContainer" containerID="9852780851d2660189eb19bd9453c4ed7b6ffa7dc021818961dd89b986197aa2"
Sep 18 18:24:22.415432 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:22.415286    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f3eedc-654d65d7c9-d8vt2"
Sep 18 18:24:22.425232 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:22.425211    2561 scope.go:117] "RemoveContainer" containerID="9852780851d2660189eb19bd9453c4ed7b6ffa7dc021818961dd89b986197aa2"
Sep 18 18:24:22.425514 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:24:22.425488    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9852780851d2660189eb19bd9453c4ed7b6ffa7dc021818961dd89b986197aa2\": container with ID starting with 9852780851d2660189eb19bd9453c4ed7b6ffa7dc021818961dd89b986197aa2 not found: ID does not exist" containerID="9852780851d2660189eb19bd9453c4ed7b6ffa7dc021818961dd89b986197aa2"
Sep 18 18:24:22.425585 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:22.425526    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9852780851d2660189eb19bd9453c4ed7b6ffa7dc021818961dd89b986197aa2"} err="failed to get container status \"9852780851d2660189eb19bd9453c4ed7b6ffa7dc021818961dd89b986197aa2\": rpc error: code = NotFound desc = could not find container \"9852780851d2660189eb19bd9453c4ed7b6ffa7dc021818961dd89b986197aa2\": container with ID starting with 9852780851d2660189eb19bd9453c4ed7b6ffa7dc021818961dd89b986197aa2 not found: ID does not exist"
Sep 18 18:24:22.505982 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:22.505949    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6r89v\" (UniqueName: \"kubernetes.io/projected/d4aaee9c-1d5e-4549-9f4e-119114e87e35-kube-api-access-6r89v\") pod \"d4aaee9c-1d5e-4549-9f4e-119114e87e35\" (UID: \"d4aaee9c-1d5e-4549-9f4e-119114e87e35\") "
Sep 18 18:24:22.506172 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:22.506050    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4aaee9c-1d5e-4549-9f4e-119114e87e35-plugins-config-volume\") pod \"d4aaee9c-1d5e-4549-9f4e-119114e87e35\" (UID: \"d4aaee9c-1d5e-4549-9f4e-119114e87e35\") "
Sep 18 18:24:22.506432 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:22.506402    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4aaee9c-1d5e-4549-9f4e-119114e87e35-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d4aaee9c-1d5e-4549-9f4e-119114e87e35" (UID: "d4aaee9c-1d5e-4549-9f4e-119114e87e35"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:24:22.508079 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:22.508058    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4aaee9c-1d5e-4549-9f4e-119114e87e35-kube-api-access-6r89v" (OuterVolumeSpecName: "kube-api-access-6r89v") pod "d4aaee9c-1d5e-4549-9f4e-119114e87e35" (UID: "d4aaee9c-1d5e-4549-9f4e-119114e87e35"). InnerVolumeSpecName "kube-api-access-6r89v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:24:22.607043 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:22.607007    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6r89v\" (UniqueName: \"kubernetes.io/projected/d4aaee9c-1d5e-4549-9f4e-119114e87e35-kube-api-access-6r89v\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:24:22.607043 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:22.607035    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4aaee9c-1d5e-4549-9f4e-119114e87e35-plugins-config-volume\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:24:22.660384 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:22.660347    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:22.739073 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:22.739035    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-f3eedc-654d65d7c9-d8vt2"]
Sep 18 18:24:22.741553 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:22.741527    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-f3eedc-654d65d7c9-d8vt2"]
Sep 18 18:24:22.741988 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:22.741969    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:23.414993 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:23.414958    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:23.861370 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:23.861288    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4aaee9c-1d5e-4549-9f4e-119114e87e35" path="/var/lib/kubelet/pods/d4aaee9c-1d5e-4549-9f4e-119114e87e35/volumes"
Sep 18 18:24:24.076100 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:24.076033    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:24.687410 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:24.687375    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:25.327275 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:25.327230    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:28.564268 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:28.564239    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:28.617343 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:28.617315    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:28.871082 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:28.871004    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:32.626425 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:32.626393    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:33.344398 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:33.344364    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:34.022241 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:34.022213    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:34.767465 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:34.767385    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:35.547047 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:35.547017    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:36.164228 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:36.164198    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:47.817864 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:47.817822    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-66c663-74bd6d9bf6-m5k85"]
Sep 18 18:24:47.818449 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:47.818066    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-66c663-74bd6d9bf6-m5k85" podUID="acf839df-86e6-47e0-9cd8-c2d155ae6d6a" containerName="maas-api" containerID="cri-o://9fcce50571b6a5dd670e226fbc26b92ed7da1f29afb7355faf1f2f3e77e8d0a8" gracePeriod=30
Sep 18 18:24:47.859110 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:47.859081    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:47.870303 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:47.870271    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-66c663-f7dc8b74f-tvnq5"]
Sep 18 18:24:47.870562 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:47.870535    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-66c663-f7dc8b74f-tvnq5" podUID="d517f3ac-da19-4bef-bf4d-23aef7d01c29" containerName="payload-processing" containerID="cri-o://c1751f4ac2a8ebf593e57cfd7d1d22b4c5adb3849901ed8d5468b2f980e4ed71" gracePeriod=30
Sep 18 18:24:47.959070 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:47.959035    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:48.031471 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:48.031435    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:48.089296 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:48.089267    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:48.125560 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:48.125535    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-66c663-74bd6d9bf6-m5k85"
Sep 18 18:24:48.161044 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:48.161021    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-66c663-f7dc8b74f-tvnq5"
Sep 18 18:24:48.240100 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:48.240067    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acf839df-86e6-47e0-9cd8-c2d155ae6d6a-maas-api-tls\") pod \"acf839df-86e6-47e0-9cd8-c2d155ae6d6a\" (UID: \"acf839df-86e6-47e0-9cd8-c2d155ae6d6a\") "
Sep 18 18:24:48.240303 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:48.240139    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c49vr\" (UniqueName: \"kubernetes.io/projected/acf839df-86e6-47e0-9cd8-c2d155ae6d6a-kube-api-access-c49vr\") pod \"acf839df-86e6-47e0-9cd8-c2d155ae6d6a\" (UID: \"acf839df-86e6-47e0-9cd8-c2d155ae6d6a\") "
Sep 18 18:24:48.240303 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:48.240207    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-klj24\" (UniqueName: \"kubernetes.io/projected/d517f3ac-da19-4bef-bf4d-23aef7d01c29-kube-api-access-klj24\") pod \"d517f3ac-da19-4bef-bf4d-23aef7d01c29\" (UID: \"d517f3ac-da19-4bef-bf4d-23aef7d01c29\") "
Sep 18 18:24:48.240303 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:48.240267    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acf839df-86e6-47e0-9cd8-c2d155ae6d6a-openshift-service-ca\") pod \"acf839df-86e6-47e0-9cd8-c2d155ae6d6a\" (UID: \"acf839df-86e6-47e0-9cd8-c2d155ae6d6a\") "
Sep 18 18:24:48.240478 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:48.240314    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d517f3ac-da19-4bef-bf4d-23aef7d01c29-plugins-config-volume\") pod \"d517f3ac-da19-4bef-bf4d-23aef7d01c29\" (UID: \"d517f3ac-da19-4bef-bf4d-23aef7d01c29\") "
Sep 18 18:24:48.240478 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:48.240362    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acf839df-86e6-47e0-9cd8-c2d155ae6d6a-metrics-certs\") pod \"acf839df-86e6-47e0-9cd8-c2d155ae6d6a\" (UID: \"acf839df-86e6-47e0-9cd8-c2d155ae6d6a\") "
Sep 18 18:24:48.240656 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:48.240621    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acf839df-86e6-47e0-9cd8-c2d155ae6d6a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "acf839df-86e6-47e0-9cd8-c2d155ae6d6a" (UID: "acf839df-86e6-47e0-9cd8-c2d155ae6d6a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:24:48.240729 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:48.240687    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d517f3ac-da19-4bef-bf4d-23aef7d01c29-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d517f3ac-da19-4bef-bf4d-23aef7d01c29" (UID: "d517f3ac-da19-4bef-bf4d-23aef7d01c29"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:24:48.242365 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:48.242343    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acf839df-86e6-47e0-9cd8-c2d155ae6d6a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "acf839df-86e6-47e0-9cd8-c2d155ae6d6a" (UID: "acf839df-86e6-47e0-9cd8-c2d155ae6d6a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:24:48.242365 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:48.242360    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acf839df-86e6-47e0-9cd8-c2d155ae6d6a-kube-api-access-c49vr" (OuterVolumeSpecName: "kube-api-access-c49vr") pod "acf839df-86e6-47e0-9cd8-c2d155ae6d6a" (UID: "acf839df-86e6-47e0-9cd8-c2d155ae6d6a"). InnerVolumeSpecName "kube-api-access-c49vr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:24:48.242510 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:48.242386    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acf839df-86e6-47e0-9cd8-c2d155ae6d6a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "acf839df-86e6-47e0-9cd8-c2d155ae6d6a" (UID: "acf839df-86e6-47e0-9cd8-c2d155ae6d6a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:24:48.242510 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:48.242395    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d517f3ac-da19-4bef-bf4d-23aef7d01c29-kube-api-access-klj24" (OuterVolumeSpecName: "kube-api-access-klj24") pod "d517f3ac-da19-4bef-bf4d-23aef7d01c29" (UID: "d517f3ac-da19-4bef-bf4d-23aef7d01c29"). InnerVolumeSpecName "kube-api-access-klj24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:24:48.341675 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:48.341586    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acf839df-86e6-47e0-9cd8-c2d155ae6d6a-openshift-service-ca\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:24:48.341675 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:48.341620    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d517f3ac-da19-4bef-bf4d-23aef7d01c29-plugins-config-volume\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:24:48.341675 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:48.341634    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acf839df-86e6-47e0-9cd8-c2d155ae6d6a-metrics-certs\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:24:48.341675 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:48.341647    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acf839df-86e6-47e0-9cd8-c2d155ae6d6a-maas-api-tls\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:24:48.341675 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:48.341658    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c49vr\" (UniqueName: \"kubernetes.io/projected/acf839df-86e6-47e0-9cd8-c2d155ae6d6a-kube-api-access-c49vr\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:24:48.341675 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:48.341669    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klj24\" (UniqueName: \"kubernetes.io/projected/d517f3ac-da19-4bef-bf4d-23aef7d01c29-kube-api-access-klj24\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:24:48.435267 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:48.435225    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:48.537450 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:48.537415    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:48.547313 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:48.547286    2561 generic.go:356] "Generic (PLEG): container finished" podID="d517f3ac-da19-4bef-bf4d-23aef7d01c29" containerID="c1751f4ac2a8ebf593e57cfd7d1d22b4c5adb3849901ed8d5468b2f980e4ed71" exitCode=0
Sep 18 18:24:48.547483 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:48.547328    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-66c663-f7dc8b74f-tvnq5"
Sep 18 18:24:48.547483 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:48.547368    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-66c663-f7dc8b74f-tvnq5" event={"ID":"d517f3ac-da19-4bef-bf4d-23aef7d01c29","Type":"ContainerDied","Data":"c1751f4ac2a8ebf593e57cfd7d1d22b4c5adb3849901ed8d5468b2f980e4ed71"}
Sep 18 18:24:48.547483 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:48.547424    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-66c663-f7dc8b74f-tvnq5" event={"ID":"d517f3ac-da19-4bef-bf4d-23aef7d01c29","Type":"ContainerDied","Data":"3558d3f8d1757eebdb472a5b78a5b5f21cd39618f5b6e5d47e5d557a5bcae355"}
Sep 18 18:24:48.547483 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:48.547458    2561 scope.go:117] "RemoveContainer" containerID="c1751f4ac2a8ebf593e57cfd7d1d22b4c5adb3849901ed8d5468b2f980e4ed71"
Sep 18 18:24:48.556852 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:48.556821    2561 generic.go:356] "Generic (PLEG): container finished" podID="acf839df-86e6-47e0-9cd8-c2d155ae6d6a" containerID="9fcce50571b6a5dd670e226fbc26b92ed7da1f29afb7355faf1f2f3e77e8d0a8" exitCode=0
Sep 18 18:24:48.556980 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:48.556880    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-66c663-74bd6d9bf6-m5k85" event={"ID":"acf839df-86e6-47e0-9cd8-c2d155ae6d6a","Type":"ContainerDied","Data":"9fcce50571b6a5dd670e226fbc26b92ed7da1f29afb7355faf1f2f3e77e8d0a8"}
Sep 18 18:24:48.556980 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:48.556896    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-66c663-74bd6d9bf6-m5k85"
Sep 18 18:24:48.556980 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:48.556958    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-66c663-74bd6d9bf6-m5k85" event={"ID":"acf839df-86e6-47e0-9cd8-c2d155ae6d6a","Type":"ContainerDied","Data":"cb31fe1c6053a4784dd3dcf9cd79c5097b71e9a760100bc41403caa4fcd63d9e"}
Sep 18 18:24:48.565547 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:48.565465    2561 scope.go:117] "RemoveContainer" containerID="c1751f4ac2a8ebf593e57cfd7d1d22b4c5adb3849901ed8d5468b2f980e4ed71"
Sep 18 18:24:48.567251 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:24:48.567222    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1751f4ac2a8ebf593e57cfd7d1d22b4c5adb3849901ed8d5468b2f980e4ed71\": container with ID starting with c1751f4ac2a8ebf593e57cfd7d1d22b4c5adb3849901ed8d5468b2f980e4ed71 not found: ID does not exist" containerID="c1751f4ac2a8ebf593e57cfd7d1d22b4c5adb3849901ed8d5468b2f980e4ed71"
Sep 18 18:24:48.567352 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:48.567262    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1751f4ac2a8ebf593e57cfd7d1d22b4c5adb3849901ed8d5468b2f980e4ed71"} err="failed to get container status \"c1751f4ac2a8ebf593e57cfd7d1d22b4c5adb3849901ed8d5468b2f980e4ed71\": rpc error: code = NotFound desc = could not find container \"c1751f4ac2a8ebf593e57cfd7d1d22b4c5adb3849901ed8d5468b2f980e4ed71\": container with ID starting with c1751f4ac2a8ebf593e57cfd7d1d22b4c5adb3849901ed8d5468b2f980e4ed71 not found: ID does not exist"
Sep 18 18:24:48.567352 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:48.567287    2561 scope.go:117] "RemoveContainer" containerID="9fcce50571b6a5dd670e226fbc26b92ed7da1f29afb7355faf1f2f3e77e8d0a8"
Sep 18 18:24:48.575864 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:48.575846    2561 scope.go:117] "RemoveContainer" containerID="9fcce50571b6a5dd670e226fbc26b92ed7da1f29afb7355faf1f2f3e77e8d0a8"
Sep 18 18:24:48.576174 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:24:48.576105    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fcce50571b6a5dd670e226fbc26b92ed7da1f29afb7355faf1f2f3e77e8d0a8\": container with ID starting with 9fcce50571b6a5dd670e226fbc26b92ed7da1f29afb7355faf1f2f3e77e8d0a8 not found: ID does not exist" containerID="9fcce50571b6a5dd670e226fbc26b92ed7da1f29afb7355faf1f2f3e77e8d0a8"
Sep 18 18:24:48.576240 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:48.576186    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fcce50571b6a5dd670e226fbc26b92ed7da1f29afb7355faf1f2f3e77e8d0a8"} err="failed to get container status \"9fcce50571b6a5dd670e226fbc26b92ed7da1f29afb7355faf1f2f3e77e8d0a8\": rpc error: code = NotFound desc = could not find container \"9fcce50571b6a5dd670e226fbc26b92ed7da1f29afb7355faf1f2f3e77e8d0a8\": container with ID starting with 9fcce50571b6a5dd670e226fbc26b92ed7da1f29afb7355faf1f2f3e77e8d0a8 not found: ID does not exist"
Sep 18 18:24:48.576727 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:48.576709    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-66c663-f7dc8b74f-tvnq5"]
Sep 18 18:24:48.580353 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:48.580327    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-66c663-f7dc8b74f-tvnq5"]
Sep 18 18:24:48.590406 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:48.590381    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-66c663-74bd6d9bf6-m5k85"]
Sep 18 18:24:48.596063 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:48.596008    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-66c663-74bd6d9bf6-m5k85"]
Sep 18 18:24:49.152930 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:49.152901    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:49.861000 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:49.860963    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="acf839df-86e6-47e0-9cd8-c2d155ae6d6a" path="/var/lib/kubelet/pods/acf839df-86e6-47e0-9cd8-c2d155ae6d6a/volumes"
Sep 18 18:24:49.861397 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:49.861382    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d517f3ac-da19-4bef-bf4d-23aef7d01c29" path="/var/lib/kubelet/pods/d517f3ac-da19-4bef-bf4d-23aef7d01c29/volumes"
Sep 18 18:24:49.931834 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:49.931802    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:50.576594 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:50.576561    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:53.422379 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:53.422348    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:24:55.970191 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:55.970165    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-5kjlz_75f2a1f7-95a7-462e-9ea7-23c51dd29599/console-operator/1.log"
Sep 18 18:24:55.976389 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:55.976361    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-5kjlz_75f2a1f7-95a7-462e-9ea7-23c51dd29599/console-operator/1.log"
Sep 18 18:24:55.977012 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:55.976997    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkcgk_c5d026b7-af6b-4899-85f4-6554c0f7cd52/ovn-acl-logging/0.log"
Sep 18 18:24:55.982951 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:24:55.982934    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkcgk_c5d026b7-af6b-4899-85f4-6554c0f7cd52/ovn-acl-logging/0.log"
Sep 18 18:25:30.292341 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:25:30.292312    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:30.437561 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:25:30.437520    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:35.988252 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:25:35.988215    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:36.062222 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:25:36.062189    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:36.287497 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:25:36.287421    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:36.944469 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:25:36.944419    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:37.795288 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:25:37.795252    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:38.371137 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:25:38.367243    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:39.059840 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:25:39.059803    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:39.897090 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:25:39.897059    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:42.122830 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:25:42.122798    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:46.788964 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:25:46.788904    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:47.350571 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:25:47.350535    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-949a32-5c4c4d9c45-zrj6t"]
Sep 18 18:25:47.350850 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:25:47.350816    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-949a32-5c4c4d9c45-zrj6t" podUID="0525c780-301e-4a8c-9896-f34a912e0f3b" containerName="maas-api" containerID="cri-o://7f8e56d9cdf0fdbbc97bbcc5c37a624a820f846c79d304e9a51a08375ea2bb0e" gracePeriod=30
Sep 18 18:25:47.420261 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:25:47.420227    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-949a32-6c86dfd9c5-8jcxc"]
Sep 18 18:25:47.420527 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:25:47.420500    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-949a32-6c86dfd9c5-8jcxc" podUID="fdfbb3be-537d-413c-869e-61e01cb0e86c" containerName="payload-processing" containerID="cri-o://34072393f9b6584222d90b70ddc67b9aac35c2f24c45f61de613143696abe302" gracePeriod=30
Sep 18 18:25:47.689200 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:25:47.689170    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-949a32-5c4c4d9c45-zrj6t"
Sep 18 18:25:47.716899 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:25:47.716870    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-949a32-6c86dfd9c5-8jcxc"
Sep 18 18:25:47.789062 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:25:47.789033    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0525c780-301e-4a8c-9896-f34a912e0f3b-metrics-certs\") pod \"0525c780-301e-4a8c-9896-f34a912e0f3b\" (UID: \"0525c780-301e-4a8c-9896-f34a912e0f3b\") "
Sep 18 18:25:47.789526 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:25:47.789141    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0525c780-301e-4a8c-9896-f34a912e0f3b-maas-api-tls\") pod \"0525c780-301e-4a8c-9896-f34a912e0f3b\" (UID: \"0525c780-301e-4a8c-9896-f34a912e0f3b\") "
Sep 18 18:25:47.789526 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:25:47.789162    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0525c780-301e-4a8c-9896-f34a912e0f3b-openshift-service-ca\") pod \"0525c780-301e-4a8c-9896-f34a912e0f3b\" (UID: \"0525c780-301e-4a8c-9896-f34a912e0f3b\") "
Sep 18 18:25:47.789526 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:25:47.789193    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6mmr8\" (UniqueName: \"kubernetes.io/projected/fdfbb3be-537d-413c-869e-61e01cb0e86c-kube-api-access-6mmr8\") pod \"fdfbb3be-537d-413c-869e-61e01cb0e86c\" (UID: \"fdfbb3be-537d-413c-869e-61e01cb0e86c\") "
Sep 18 18:25:47.789526 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:25:47.789212    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdfbb3be-537d-413c-869e-61e01cb0e86c-plugins-config-volume\") pod \"fdfbb3be-537d-413c-869e-61e01cb0e86c\" (UID: \"fdfbb3be-537d-413c-869e-61e01cb0e86c\") "
Sep 18 18:25:47.789526 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:25:47.789251    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6h6x2\" (UniqueName: \"kubernetes.io/projected/0525c780-301e-4a8c-9896-f34a912e0f3b-kube-api-access-6h6x2\") pod \"0525c780-301e-4a8c-9896-f34a912e0f3b\" (UID: \"0525c780-301e-4a8c-9896-f34a912e0f3b\") "
Sep 18 18:25:47.789813 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:25:47.789614    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0525c780-301e-4a8c-9896-f34a912e0f3b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0525c780-301e-4a8c-9896-f34a912e0f3b" (UID: "0525c780-301e-4a8c-9896-f34a912e0f3b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:25:47.789995 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:25:47.789932    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdfbb3be-537d-413c-869e-61e01cb0e86c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fdfbb3be-537d-413c-869e-61e01cb0e86c" (UID: "fdfbb3be-537d-413c-869e-61e01cb0e86c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:25:47.791416 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:25:47.791388    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0525c780-301e-4a8c-9896-f34a912e0f3b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0525c780-301e-4a8c-9896-f34a912e0f3b" (UID: "0525c780-301e-4a8c-9896-f34a912e0f3b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:25:47.791532 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:25:47.791460    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0525c780-301e-4a8c-9896-f34a912e0f3b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0525c780-301e-4a8c-9896-f34a912e0f3b" (UID: "0525c780-301e-4a8c-9896-f34a912e0f3b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:25:47.791586 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:25:47.791531    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0525c780-301e-4a8c-9896-f34a912e0f3b-kube-api-access-6h6x2" (OuterVolumeSpecName: "kube-api-access-6h6x2") pod "0525c780-301e-4a8c-9896-f34a912e0f3b" (UID: "0525c780-301e-4a8c-9896-f34a912e0f3b"). InnerVolumeSpecName "kube-api-access-6h6x2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:25:47.791624 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:25:47.791596    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdfbb3be-537d-413c-869e-61e01cb0e86c-kube-api-access-6mmr8" (OuterVolumeSpecName: "kube-api-access-6mmr8") pod "fdfbb3be-537d-413c-869e-61e01cb0e86c" (UID: "fdfbb3be-537d-413c-869e-61e01cb0e86c"). InnerVolumeSpecName "kube-api-access-6mmr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:25:47.858314 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:25:47.858273    2561 generic.go:356] "Generic (PLEG): container finished" podID="0525c780-301e-4a8c-9896-f34a912e0f3b" containerID="7f8e56d9cdf0fdbbc97bbcc5c37a624a820f846c79d304e9a51a08375ea2bb0e" exitCode=0
Sep 18 18:25:47.858504 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:25:47.858355    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-949a32-5c4c4d9c45-zrj6t"
Sep 18 18:25:47.859936 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:25:47.859910    2561 generic.go:356] "Generic (PLEG): container finished" podID="fdfbb3be-537d-413c-869e-61e01cb0e86c" containerID="34072393f9b6584222d90b70ddc67b9aac35c2f24c45f61de613143696abe302" exitCode=0
Sep 18 18:25:47.860063 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:25:47.859962    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-949a32-6c86dfd9c5-8jcxc"
Sep 18 18:25:47.863382 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:25:47.863354    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-949a32-5c4c4d9c45-zrj6t" event={"ID":"0525c780-301e-4a8c-9896-f34a912e0f3b","Type":"ContainerDied","Data":"7f8e56d9cdf0fdbbc97bbcc5c37a624a820f846c79d304e9a51a08375ea2bb0e"}
Sep 18 18:25:47.863523 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:25:47.863406    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-949a32-5c4c4d9c45-zrj6t" event={"ID":"0525c780-301e-4a8c-9896-f34a912e0f3b","Type":"ContainerDied","Data":"0c1e8dfba0ad2a11ab418aab60f016cf5fb2d45dcd9996e1659da4e56fd75cf5"}
Sep 18 18:25:47.863523 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:25:47.863431    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-949a32-6c86dfd9c5-8jcxc" event={"ID":"fdfbb3be-537d-413c-869e-61e01cb0e86c","Type":"ContainerDied","Data":"34072393f9b6584222d90b70ddc67b9aac35c2f24c45f61de613143696abe302"}
Sep 18 18:25:47.863523 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:25:47.863454    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-949a32-6c86dfd9c5-8jcxc" event={"ID":"fdfbb3be-537d-413c-869e-61e01cb0e86c","Type":"ContainerDied","Data":"1a6685044a8437c68399fab24a7a6419deb49b469b7d2f86c617d19d1be9e433"}
Sep 18 18:25:47.863523 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:25:47.863486    2561 scope.go:117] "RemoveContainer" containerID="7f8e56d9cdf0fdbbc97bbcc5c37a624a820f846c79d304e9a51a08375ea2bb0e"
Sep 18 18:25:47.874495 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:25:47.874477    2561 scope.go:117] "RemoveContainer" containerID="7f8e56d9cdf0fdbbc97bbcc5c37a624a820f846c79d304e9a51a08375ea2bb0e"
Sep 18 18:25:47.874837 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:25:47.874818    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f8e56d9cdf0fdbbc97bbcc5c37a624a820f846c79d304e9a51a08375ea2bb0e\": container with ID starting with 7f8e56d9cdf0fdbbc97bbcc5c37a624a820f846c79d304e9a51a08375ea2bb0e not found: ID does not exist" containerID="7f8e56d9cdf0fdbbc97bbcc5c37a624a820f846c79d304e9a51a08375ea2bb0e"
Sep 18 18:25:47.874904 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:25:47.874845    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f8e56d9cdf0fdbbc97bbcc5c37a624a820f846c79d304e9a51a08375ea2bb0e"} err="failed to get container status \"7f8e56d9cdf0fdbbc97bbcc5c37a624a820f846c79d304e9a51a08375ea2bb0e\": rpc error: code = NotFound desc = could not find container \"7f8e56d9cdf0fdbbc97bbcc5c37a624a820f846c79d304e9a51a08375ea2bb0e\": container with ID starting with 7f8e56d9cdf0fdbbc97bbcc5c37a624a820f846c79d304e9a51a08375ea2bb0e not found: ID does not exist"
Sep 18 18:25:47.874904 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:25:47.874865    2561 scope.go:117] "RemoveContainer" containerID="34072393f9b6584222d90b70ddc67b9aac35c2f24c45f61de613143696abe302"
Sep 18 18:25:47.883702 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:25:47.883681    2561 scope.go:117] "RemoveContainer" containerID="34072393f9b6584222d90b70ddc67b9aac35c2f24c45f61de613143696abe302"
Sep 18 18:25:47.884028 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:25:47.884005    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34072393f9b6584222d90b70ddc67b9aac35c2f24c45f61de613143696abe302\": container with ID starting with 34072393f9b6584222d90b70ddc67b9aac35c2f24c45f61de613143696abe302 not found: ID does not exist" containerID="34072393f9b6584222d90b70ddc67b9aac35c2f24c45f61de613143696abe302"
Sep 18 18:25:47.884133 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:25:47.884034    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34072393f9b6584222d90b70ddc67b9aac35c2f24c45f61de613143696abe302"} err="failed to get container status \"34072393f9b6584222d90b70ddc67b9aac35c2f24c45f61de613143696abe302\": rpc error: code = NotFound desc = could not find container \"34072393f9b6584222d90b70ddc67b9aac35c2f24c45f61de613143696abe302\": container with ID starting with 34072393f9b6584222d90b70ddc67b9aac35c2f24c45f61de613143696abe302 not found: ID does not exist"
Sep 18 18:25:47.890575 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:25:47.890518    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0525c780-301e-4a8c-9896-f34a912e0f3b-metrics-certs\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:25:47.890575 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:25:47.890547    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0525c780-301e-4a8c-9896-f34a912e0f3b-maas-api-tls\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:25:47.890575 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:25:47.890562    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0525c780-301e-4a8c-9896-f34a912e0f3b-openshift-service-ca\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:25:47.890725 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:25:47.890576    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mmr8\" (UniqueName: \"kubernetes.io/projected/fdfbb3be-537d-413c-869e-61e01cb0e86c-kube-api-access-6mmr8\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:25:47.890725 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:25:47.890589    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdfbb3be-537d-413c-869e-61e01cb0e86c-plugins-config-volume\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:25:47.890725 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:25:47.890602    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6h6x2\" (UniqueName: \"kubernetes.io/projected/0525c780-301e-4a8c-9896-f34a912e0f3b-kube-api-access-6h6x2\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:25:47.891632 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:25:47.891611    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-949a32-6c86dfd9c5-8jcxc"]
Sep 18 18:25:47.895129 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:25:47.895099    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-949a32-6c86dfd9c5-8jcxc"]
Sep 18 18:25:47.905689 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:25:47.905668    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-949a32-5c4c4d9c45-zrj6t"]
Sep 18 18:25:47.907629 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:25:47.907610    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-949a32-5c4c4d9c45-zrj6t"]
Sep 18 18:25:48.171902 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:25:48.171872    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:48.820090 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:25:48.820056    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:49.500746 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:25:49.500712    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:49.866674 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:25:49.866580    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0525c780-301e-4a8c-9896-f34a912e0f3b" path="/var/lib/kubelet/pods/0525c780-301e-4a8c-9896-f34a912e0f3b/volumes"
Sep 18 18:25:49.867164 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:25:49.867141    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fdfbb3be-537d-413c-869e-61e01cb0e86c" path="/var/lib/kubelet/pods/fdfbb3be-537d-413c-869e-61e01cb0e86c/volumes"
Sep 18 18:25:54.256507 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:25:54.256478    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:54.340436 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:25:54.340408    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:54.534477 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:25:54.534395    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:55.746057 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:25:55.746025    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:56.404263 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:25:56.404235    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:25:57.197963 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:25:57.197930    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:05.321824 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:05.321788    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-53cde3-6b864cc5fd-tvlhr"]
Sep 18 18:26:05.322408 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:05.322078    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-53cde3-6b864cc5fd-tvlhr" podUID="f46e6d38-157e-4272-84f9-bacc3b231773" containerName="maas-api" containerID="cri-o://f51d1b73cbb85c4b1a8762156286dd841d28fea13d4383c902e5277f93093484" gracePeriod=30
Sep 18 18:26:05.348278 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:05.348244    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:05.364462 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:05.364432    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-53cde3-7d857f96b9-p9gft"]
Sep 18 18:26:05.364710 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:05.364657    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-53cde3-7d857f96b9-p9gft" podUID="0b5fbe79-9f46-4567-90f9-52f7c4ba11a8" containerName="payload-processing" containerID="cri-o://e0043796f38845e0c0f65502cafe918b69f240fb54b76c02cf4999a2637c544c" gracePeriod=30
Sep 18 18:26:05.447297 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:05.447256    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:05.534658 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:05.534442    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:05.602021 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:05.601992    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:05.614096 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:05.614073    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-53cde3-6b864cc5fd-tvlhr"
Sep 18 18:26:05.643012 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:05.642990    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-53cde3-7d857f96b9-p9gft"
Sep 18 18:26:05.650947 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:05.650919    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2dhkh\" (UniqueName: \"kubernetes.io/projected/f46e6d38-157e-4272-84f9-bacc3b231773-kube-api-access-2dhkh\") pod \"f46e6d38-157e-4272-84f9-bacc3b231773\" (UID: \"f46e6d38-157e-4272-84f9-bacc3b231773\") "
Sep 18 18:26:05.651091 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:05.650967    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f46e6d38-157e-4272-84f9-bacc3b231773-maas-api-tls\") pod \"f46e6d38-157e-4272-84f9-bacc3b231773\" (UID: \"f46e6d38-157e-4272-84f9-bacc3b231773\") "
Sep 18 18:26:05.651091 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:05.650996    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f46e6d38-157e-4272-84f9-bacc3b231773-metrics-certs\") pod \"f46e6d38-157e-4272-84f9-bacc3b231773\" (UID: \"f46e6d38-157e-4272-84f9-bacc3b231773\") "
Sep 18 18:26:05.651091 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:05.651066    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b5fbe79-9f46-4567-90f9-52f7c4ba11a8-plugins-config-volume\") pod \"0b5fbe79-9f46-4567-90f9-52f7c4ba11a8\" (UID: \"0b5fbe79-9f46-4567-90f9-52f7c4ba11a8\") "
Sep 18 18:26:05.651390 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:05.651102    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f46e6d38-157e-4272-84f9-bacc3b231773-openshift-service-ca\") pod \"f46e6d38-157e-4272-84f9-bacc3b231773\" (UID: \"f46e6d38-157e-4272-84f9-bacc3b231773\") "
Sep 18 18:26:05.651802 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:05.651773    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x4r5q\" (UniqueName: \"kubernetes.io/projected/0b5fbe79-9f46-4567-90f9-52f7c4ba11a8-kube-api-access-x4r5q\") pod \"0b5fbe79-9f46-4567-90f9-52f7c4ba11a8\" (UID: \"0b5fbe79-9f46-4567-90f9-52f7c4ba11a8\") "
Sep 18 18:26:05.653042 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:05.653015    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f46e6d38-157e-4272-84f9-bacc3b231773-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f46e6d38-157e-4272-84f9-bacc3b231773" (UID: "f46e6d38-157e-4272-84f9-bacc3b231773"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:26:05.653495 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:05.653467    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b5fbe79-9f46-4567-90f9-52f7c4ba11a8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0b5fbe79-9f46-4567-90f9-52f7c4ba11a8" (UID: "0b5fbe79-9f46-4567-90f9-52f7c4ba11a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:26:05.654590 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:05.654563    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f46e6d38-157e-4272-84f9-bacc3b231773-kube-api-access-2dhkh" (OuterVolumeSpecName: "kube-api-access-2dhkh") pod "f46e6d38-157e-4272-84f9-bacc3b231773" (UID: "f46e6d38-157e-4272-84f9-bacc3b231773"). InnerVolumeSpecName "kube-api-access-2dhkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:26:05.654906 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:05.654881    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b5fbe79-9f46-4567-90f9-52f7c4ba11a8-kube-api-access-x4r5q" (OuterVolumeSpecName: "kube-api-access-x4r5q") pod "0b5fbe79-9f46-4567-90f9-52f7c4ba11a8" (UID: "0b5fbe79-9f46-4567-90f9-52f7c4ba11a8"). InnerVolumeSpecName "kube-api-access-x4r5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:26:05.655822 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:05.655796    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f46e6d38-157e-4272-84f9-bacc3b231773-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f46e6d38-157e-4272-84f9-bacc3b231773" (UID: "f46e6d38-157e-4272-84f9-bacc3b231773"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:26:05.656761 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:05.656742    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f46e6d38-157e-4272-84f9-bacc3b231773-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f46e6d38-157e-4272-84f9-bacc3b231773" (UID: "f46e6d38-157e-4272-84f9-bacc3b231773"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:26:05.753440 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:05.753393    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b5fbe79-9f46-4567-90f9-52f7c4ba11a8-plugins-config-volume\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:26:05.753440 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:05.753436    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f46e6d38-157e-4272-84f9-bacc3b231773-openshift-service-ca\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:26:05.753440 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:05.753449    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4r5q\" (UniqueName: \"kubernetes.io/projected/0b5fbe79-9f46-4567-90f9-52f7c4ba11a8-kube-api-access-x4r5q\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:26:05.753674 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:05.753461    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dhkh\" (UniqueName: \"kubernetes.io/projected/f46e6d38-157e-4272-84f9-bacc3b231773-kube-api-access-2dhkh\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:26:05.753674 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:05.753473    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f46e6d38-157e-4272-84f9-bacc3b231773-maas-api-tls\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:26:05.753674 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:05.753485    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f46e6d38-157e-4272-84f9-bacc3b231773-metrics-certs\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:26:05.903729 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:05.903703    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:05.948813 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:05.948769    2561 generic.go:356] "Generic (PLEG): container finished" podID="f46e6d38-157e-4272-84f9-bacc3b231773" containerID="f51d1b73cbb85c4b1a8762156286dd841d28fea13d4383c902e5277f93093484" exitCode=0
Sep 18 18:26:05.948963 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:05.948906    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-53cde3-6b864cc5fd-tvlhr"
Sep 18 18:26:05.948963 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:05.948927    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-53cde3-6b864cc5fd-tvlhr" event={"ID":"f46e6d38-157e-4272-84f9-bacc3b231773","Type":"ContainerDied","Data":"f51d1b73cbb85c4b1a8762156286dd841d28fea13d4383c902e5277f93093484"}
Sep 18 18:26:05.949081 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:05.949013    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-53cde3-6b864cc5fd-tvlhr" event={"ID":"f46e6d38-157e-4272-84f9-bacc3b231773","Type":"ContainerDied","Data":"44202567d04cd2575cf4b8665261746720515f98f6011850511ae81d66c29f0d"}
Sep 18 18:26:05.949081 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:05.949048    2561 scope.go:117] "RemoveContainer" containerID="f51d1b73cbb85c4b1a8762156286dd841d28fea13d4383c902e5277f93093484"
Sep 18 18:26:05.950461 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:05.950438    2561 generic.go:356] "Generic (PLEG): container finished" podID="0b5fbe79-9f46-4567-90f9-52f7c4ba11a8" containerID="e0043796f38845e0c0f65502cafe918b69f240fb54b76c02cf4999a2637c544c" exitCode=0
Sep 18 18:26:05.950530 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:05.950477    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-53cde3-7d857f96b9-p9gft"
Sep 18 18:26:05.950530 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:05.950491    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-53cde3-7d857f96b9-p9gft" event={"ID":"0b5fbe79-9f46-4567-90f9-52f7c4ba11a8","Type":"ContainerDied","Data":"e0043796f38845e0c0f65502cafe918b69f240fb54b76c02cf4999a2637c544c"}
Sep 18 18:26:05.950658 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:05.950532    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-53cde3-7d857f96b9-p9gft" event={"ID":"0b5fbe79-9f46-4567-90f9-52f7c4ba11a8","Type":"ContainerDied","Data":"443bdd7446369c43dde6688cf984fb9e8c60544c1a907f80cd2cdbd5765519de"}
Sep 18 18:26:05.958753 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:05.958733    2561 scope.go:117] "RemoveContainer" containerID="f51d1b73cbb85c4b1a8762156286dd841d28fea13d4383c902e5277f93093484"
Sep 18 18:26:05.959023 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:26:05.959004    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f51d1b73cbb85c4b1a8762156286dd841d28fea13d4383c902e5277f93093484\": container with ID starting with f51d1b73cbb85c4b1a8762156286dd841d28fea13d4383c902e5277f93093484 not found: ID does not exist" containerID="f51d1b73cbb85c4b1a8762156286dd841d28fea13d4383c902e5277f93093484"
Sep 18 18:26:05.959066 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:05.959033    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f51d1b73cbb85c4b1a8762156286dd841d28fea13d4383c902e5277f93093484"} err="failed to get container status \"f51d1b73cbb85c4b1a8762156286dd841d28fea13d4383c902e5277f93093484\": rpc error: code = NotFound desc = could not find container \"f51d1b73cbb85c4b1a8762156286dd841d28fea13d4383c902e5277f93093484\": container with ID starting with f51d1b73cbb85c4b1a8762156286dd841d28fea13d4383c902e5277f93093484 not found: ID does not exist"
Sep 18 18:26:05.959066 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:05.959051    2561 scope.go:117] "RemoveContainer" containerID="e0043796f38845e0c0f65502cafe918b69f240fb54b76c02cf4999a2637c544c"
Sep 18 18:26:05.966927 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:05.966909    2561 scope.go:117] "RemoveContainer" containerID="e0043796f38845e0c0f65502cafe918b69f240fb54b76c02cf4999a2637c544c"
Sep 18 18:26:05.967263 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:26:05.967245    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0043796f38845e0c0f65502cafe918b69f240fb54b76c02cf4999a2637c544c\": container with ID starting with e0043796f38845e0c0f65502cafe918b69f240fb54b76c02cf4999a2637c544c not found: ID does not exist" containerID="e0043796f38845e0c0f65502cafe918b69f240fb54b76c02cf4999a2637c544c"
Sep 18 18:26:05.967324 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:05.967271    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0043796f38845e0c0f65502cafe918b69f240fb54b76c02cf4999a2637c544c"} err="failed to get container status \"e0043796f38845e0c0f65502cafe918b69f240fb54b76c02cf4999a2637c544c\": rpc error: code = NotFound desc = could not find container \"e0043796f38845e0c0f65502cafe918b69f240fb54b76c02cf4999a2637c544c\": container with ID starting with e0043796f38845e0c0f65502cafe918b69f240fb54b76c02cf4999a2637c544c not found: ID does not exist"
Sep 18 18:26:05.967755 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:05.967731    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-53cde3-7d857f96b9-p9gft"]
Sep 18 18:26:05.971285 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:05.971265    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:05.972730 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:05.972713    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-53cde3-7d857f96b9-p9gft"]
Sep 18 18:26:05.983705 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:05.983684    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-53cde3-6b864cc5fd-tvlhr"]
Sep 18 18:26:05.988619 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:05.988602    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-53cde3-6b864cc5fd-tvlhr"]
Sep 18 18:26:06.836987 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:06.836954    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:07.861253 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:07.861222    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b5fbe79-9f46-4567-90f9-52f7c4ba11a8" path="/var/lib/kubelet/pods/0b5fbe79-9f46-4567-90f9-52f7c4ba11a8/volumes"
Sep 18 18:26:07.861675 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:07.861659    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f46e6d38-157e-4272-84f9-bacc3b231773" path="/var/lib/kubelet/pods/f46e6d38-157e-4272-84f9-bacc3b231773/volumes"
Sep 18 18:26:11.845936 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:11.845888    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:11.947195 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:11.947164    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:12.100345 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:12.100262    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:13.129450 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:13.129417    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:13.945378 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:13.945338    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:22.864697 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:22.864655    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:22.928249 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:22.928215    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:22.995564 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:22.995534    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:23.053651 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:23.053621    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:23.417178 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:23.417148    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:23.482003 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:23.481972    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:29.203990 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:29.203957    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:29.304347 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:29.304316    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:29.457821 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:29.457729    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:33.863617 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:33.863579    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-655d8db75-qgn94"]
Sep 18 18:26:33.864071 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:33.863824    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-655d8db75-qgn94" podUID="25e183d8-1a1e-429e-81f6-ea656ad36be5" containerName="manager" containerID="cri-o://0680b8b2f3db1e1f1ec26af26bc89d36a231a0ac0fe7b214bb03d854ab8cd023" gracePeriod=30
Sep 18 18:26:34.087933 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:34.087904    2561 generic.go:356] "Generic (PLEG): container finished" podID="25e183d8-1a1e-429e-81f6-ea656ad36be5" containerID="0680b8b2f3db1e1f1ec26af26bc89d36a231a0ac0fe7b214bb03d854ab8cd023" exitCode=0
Sep 18 18:26:34.088089 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:34.087974    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-655d8db75-qgn94" event={"ID":"25e183d8-1a1e-429e-81f6-ea656ad36be5","Type":"ContainerDied","Data":"0680b8b2f3db1e1f1ec26af26bc89d36a231a0ac0fe7b214bb03d854ab8cd023"}
Sep 18 18:26:34.115398 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:34.115341    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-655d8db75-qgn94"
Sep 18 18:26:34.220415 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:34.220387    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/25e183d8-1a1e-429e-81f6-ea656ad36be5-webhook-cert\") pod \"25e183d8-1a1e-429e-81f6-ea656ad36be5\" (UID: \"25e183d8-1a1e-429e-81f6-ea656ad36be5\") "
Sep 18 18:26:34.220590 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:34.220445    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25e183d8-1a1e-429e-81f6-ea656ad36be5-metrics-certs\") pod \"25e183d8-1a1e-429e-81f6-ea656ad36be5\" (UID: \"25e183d8-1a1e-429e-81f6-ea656ad36be5\") "
Sep 18 18:26:34.220590 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:34.220515    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t72mq\" (UniqueName: \"kubernetes.io/projected/25e183d8-1a1e-429e-81f6-ea656ad36be5-kube-api-access-t72mq\") pod \"25e183d8-1a1e-429e-81f6-ea656ad36be5\" (UID: \"25e183d8-1a1e-429e-81f6-ea656ad36be5\") "
Sep 18 18:26:34.222525 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:34.222486    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25e183d8-1a1e-429e-81f6-ea656ad36be5-kube-api-access-t72mq" (OuterVolumeSpecName: "kube-api-access-t72mq") pod "25e183d8-1a1e-429e-81f6-ea656ad36be5" (UID: "25e183d8-1a1e-429e-81f6-ea656ad36be5"). InnerVolumeSpecName "kube-api-access-t72mq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:26:34.222648 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:34.222524    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25e183d8-1a1e-429e-81f6-ea656ad36be5-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "25e183d8-1a1e-429e-81f6-ea656ad36be5" (UID: "25e183d8-1a1e-429e-81f6-ea656ad36be5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:26:34.222648 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:34.222550    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25e183d8-1a1e-429e-81f6-ea656ad36be5-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "25e183d8-1a1e-429e-81f6-ea656ad36be5" (UID: "25e183d8-1a1e-429e-81f6-ea656ad36be5"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:26:34.321815 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:34.321783    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25e183d8-1a1e-429e-81f6-ea656ad36be5-metrics-certs\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:26:34.321815 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:34.321808    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t72mq\" (UniqueName: \"kubernetes.io/projected/25e183d8-1a1e-429e-81f6-ea656ad36be5-kube-api-access-t72mq\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:26:34.321815 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:34.321817    2561 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/25e183d8-1a1e-429e-81f6-ea656ad36be5-webhook-cert\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:26:35.094185 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:35.094150    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-655d8db75-qgn94" event={"ID":"25e183d8-1a1e-429e-81f6-ea656ad36be5","Type":"ContainerDied","Data":"c80e1211d57418c80f6576a05a11d84e2ba643e1fb6fa58e066bf0688d4b256f"}
Sep 18 18:26:35.094185 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:35.094182    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-655d8db75-qgn94"
Sep 18 18:26:35.094724 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:35.094218    2561 scope.go:117] "RemoveContainer" containerID="0680b8b2f3db1e1f1ec26af26bc89d36a231a0ac0fe7b214bb03d854ab8cd023"
Sep 18 18:26:35.115230 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:35.115204    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-655d8db75-qgn94"]
Sep 18 18:26:35.118876 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:35.118851    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-655d8db75-qgn94"]
Sep 18 18:26:35.861291 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:35.861256    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25e183d8-1a1e-429e-81f6-ea656ad36be5" path="/var/lib/kubelet/pods/25e183d8-1a1e-429e-81f6-ea656ad36be5/volumes"
Sep 18 18:26:41.519459 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:41.519428    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:41.619562 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:41.619533    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:26:46.519463 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:26:46.519425    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:27:35.910236 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:35.910204    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-655d8db75-998cr"]
Sep 18 18:27:35.910641 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:35.910550    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b5fbe79-9f46-4567-90f9-52f7c4ba11a8" containerName="payload-processing"
Sep 18 18:27:35.910641 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:35.910562    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b5fbe79-9f46-4567-90f9-52f7c4ba11a8" containerName="payload-processing"
Sep 18 18:27:35.910641 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:35.910577    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdfbb3be-537d-413c-869e-61e01cb0e86c" containerName="payload-processing"
Sep 18 18:27:35.910641 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:35.910582    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdfbb3be-537d-413c-869e-61e01cb0e86c" containerName="payload-processing"
Sep 18 18:27:35.910641 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:35.910590    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25e183d8-1a1e-429e-81f6-ea656ad36be5" containerName="manager"
Sep 18 18:27:35.910641 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:35.910595    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="25e183d8-1a1e-429e-81f6-ea656ad36be5" containerName="manager"
Sep 18 18:27:35.910641 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:35.910603    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d517f3ac-da19-4bef-bf4d-23aef7d01c29" containerName="payload-processing"
Sep 18 18:27:35.910641 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:35.910608    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d517f3ac-da19-4bef-bf4d-23aef7d01c29" containerName="payload-processing"
Sep 18 18:27:35.910641 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:35.910614    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4aaee9c-1d5e-4549-9f4e-119114e87e35" containerName="payload-processing"
Sep 18 18:27:35.910641 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:35.910619    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4aaee9c-1d5e-4549-9f4e-119114e87e35" containerName="payload-processing"
Sep 18 18:27:35.910641 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:35.910627    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acf839df-86e6-47e0-9cd8-c2d155ae6d6a" containerName="maas-api"
Sep 18 18:27:35.910641 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:35.910632    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="acf839df-86e6-47e0-9cd8-c2d155ae6d6a" containerName="maas-api"
Sep 18 18:27:35.910641 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:35.910638    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f46e6d38-157e-4272-84f9-bacc3b231773" containerName="maas-api"
Sep 18 18:27:35.910641 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:35.910643    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f46e6d38-157e-4272-84f9-bacc3b231773" containerName="maas-api"
Sep 18 18:27:35.911135 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:35.910654    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0525c780-301e-4a8c-9896-f34a912e0f3b" containerName="maas-api"
Sep 18 18:27:35.911135 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:35.910659    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0525c780-301e-4a8c-9896-f34a912e0f3b" containerName="maas-api"
Sep 18 18:27:35.911135 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:35.910726    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdfbb3be-537d-413c-869e-61e01cb0e86c" containerName="payload-processing"
Sep 18 18:27:35.911135 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:35.910735    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0525c780-301e-4a8c-9896-f34a912e0f3b" containerName="maas-api"
Sep 18 18:27:35.911135 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:35.910740    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="acf839df-86e6-47e0-9cd8-c2d155ae6d6a" containerName="maas-api"
Sep 18 18:27:35.911135 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:35.910746    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="25e183d8-1a1e-429e-81f6-ea656ad36be5" containerName="manager"
Sep 18 18:27:35.911135 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:35.910752    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b5fbe79-9f46-4567-90f9-52f7c4ba11a8" containerName="payload-processing"
Sep 18 18:27:35.911135 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:35.910756    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f46e6d38-157e-4272-84f9-bacc3b231773" containerName="maas-api"
Sep 18 18:27:35.911135 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:35.910762    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4aaee9c-1d5e-4549-9f4e-119114e87e35" containerName="payload-processing"
Sep 18 18:27:35.911135 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:35.910770    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d517f3ac-da19-4bef-bf4d-23aef7d01c29" containerName="payload-processing"
Sep 18 18:27:35.913785 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:35.913770    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-655d8db75-998cr"
Sep 18 18:27:35.917371 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:35.917345    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 18 18:27:35.917505 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:35.917345    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 18 18:27:35.917505 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:35.917380    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 18 18:27:35.917505 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:35.917345    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-f744x\""
Sep 18 18:27:35.922959 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:35.922934    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-655d8db75-998cr"]
Sep 18 18:27:35.973728 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:35.973695    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qjnp\" (UniqueName: \"kubernetes.io/projected/c223ea22-9414-4132-b8a5-4e8370c0feba-kube-api-access-8qjnp\") pod \"maas-controller-655d8db75-998cr\" (UID: \"c223ea22-9414-4132-b8a5-4e8370c0feba\") " pod="opendatahub/maas-controller-655d8db75-998cr"
Sep 18 18:27:35.973880 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:35.973753    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c223ea22-9414-4132-b8a5-4e8370c0feba-metrics-certs\") pod \"maas-controller-655d8db75-998cr\" (UID: \"c223ea22-9414-4132-b8a5-4e8370c0feba\") " pod="opendatahub/maas-controller-655d8db75-998cr"
Sep 18 18:27:35.973880 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:35.973864    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c223ea22-9414-4132-b8a5-4e8370c0feba-webhook-cert\") pod \"maas-controller-655d8db75-998cr\" (UID: \"c223ea22-9414-4132-b8a5-4e8370c0feba\") " pod="opendatahub/maas-controller-655d8db75-998cr"
Sep 18 18:27:36.074601 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:36.074573    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c223ea22-9414-4132-b8a5-4e8370c0feba-webhook-cert\") pod \"maas-controller-655d8db75-998cr\" (UID: \"c223ea22-9414-4132-b8a5-4e8370c0feba\") " pod="opendatahub/maas-controller-655d8db75-998cr"
Sep 18 18:27:36.074784 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:36.074627    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qjnp\" (UniqueName: \"kubernetes.io/projected/c223ea22-9414-4132-b8a5-4e8370c0feba-kube-api-access-8qjnp\") pod \"maas-controller-655d8db75-998cr\" (UID: \"c223ea22-9414-4132-b8a5-4e8370c0feba\") " pod="opendatahub/maas-controller-655d8db75-998cr"
Sep 18 18:27:36.074784 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:36.074657    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c223ea22-9414-4132-b8a5-4e8370c0feba-metrics-certs\") pod \"maas-controller-655d8db75-998cr\" (UID: \"c223ea22-9414-4132-b8a5-4e8370c0feba\") " pod="opendatahub/maas-controller-655d8db75-998cr"
Sep 18 18:27:36.077142 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:36.077101    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c223ea22-9414-4132-b8a5-4e8370c0feba-webhook-cert\") pod \"maas-controller-655d8db75-998cr\" (UID: \"c223ea22-9414-4132-b8a5-4e8370c0feba\") " pod="opendatahub/maas-controller-655d8db75-998cr"
Sep 18 18:27:36.077240 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:36.077172    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c223ea22-9414-4132-b8a5-4e8370c0feba-metrics-certs\") pod \"maas-controller-655d8db75-998cr\" (UID: \"c223ea22-9414-4132-b8a5-4e8370c0feba\") " pod="opendatahub/maas-controller-655d8db75-998cr"
Sep 18 18:27:36.082394 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:36.082375    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qjnp\" (UniqueName: \"kubernetes.io/projected/c223ea22-9414-4132-b8a5-4e8370c0feba-kube-api-access-8qjnp\") pod \"maas-controller-655d8db75-998cr\" (UID: \"c223ea22-9414-4132-b8a5-4e8370c0feba\") " pod="opendatahub/maas-controller-655d8db75-998cr"
Sep 18 18:27:36.224181 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:36.224079    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-655d8db75-998cr"
Sep 18 18:27:36.496511 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:36.496483    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-655d8db75-998cr"]
Sep 18 18:27:36.498882 ip-10-0-140-54 kubenswrapper[2561]: W0918 18:27:36.498849    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc223ea22_9414_4132_b8a5_4e8370c0feba.slice/crio-11725f8bb516d2d2cff4a839660fc915815ad7a56ad6e54a68ec82923d24be89 WatchSource:0}: Error finding container 11725f8bb516d2d2cff4a839660fc915815ad7a56ad6e54a68ec82923d24be89: Status 404 returned error can't find the container with id 11725f8bb516d2d2cff4a839660fc915815ad7a56ad6e54a68ec82923d24be89
Sep 18 18:27:36.500674 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:36.500654    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 18:27:37.381782 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:37.381697    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-655d8db75-998cr" event={"ID":"c223ea22-9414-4132-b8a5-4e8370c0feba","Type":"ContainerStarted","Data":"09083b71d020ea2bd18ddf62f1a04f2bcd061fae2c263fff98e8b0992aded275"}
Sep 18 18:27:37.381782 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:37.381750    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-655d8db75-998cr" event={"ID":"c223ea22-9414-4132-b8a5-4e8370c0feba","Type":"ContainerStarted","Data":"11725f8bb516d2d2cff4a839660fc915815ad7a56ad6e54a68ec82923d24be89"}
Sep 18 18:27:37.381782 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:37.381780    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-655d8db75-998cr"
Sep 18 18:27:37.403770 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:37.403722    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-655d8db75-998cr" podStartSLOduration=1.970921208 podStartE2EDuration="2.403707815s" totalImagesPullingTime="432.786607ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:27:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:27:37.400802215 +0000 UTC m=+2262.027933835" watchObservedRunningTime="2026-09-18 18:27:37.403707815 +0000 UTC m=+2262.030839435"
Sep 18 18:27:48.387904 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:48.387872    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-655d8db75-998cr"
Sep 18 18:27:55.453143 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:55.453100    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:27:55.549430 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:55.549399    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-655d8db75-998cr"]
Sep 18 18:27:55.549655 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:55.549627    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-655d8db75-998cr" podUID="c223ea22-9414-4132-b8a5-4e8370c0feba" containerName="manager" containerID="cri-o://09083b71d020ea2bd18ddf62f1a04f2bcd061fae2c263fff98e8b0992aded275" gracePeriod=30
Sep 18 18:27:55.789543 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:55.789519    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-655d8db75-998cr"
Sep 18 18:27:55.951625 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:55.951593    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c223ea22-9414-4132-b8a5-4e8370c0feba-metrics-certs\") pod \"c223ea22-9414-4132-b8a5-4e8370c0feba\" (UID: \"c223ea22-9414-4132-b8a5-4e8370c0feba\") "
Sep 18 18:27:55.951789 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:55.951634    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8qjnp\" (UniqueName: \"kubernetes.io/projected/c223ea22-9414-4132-b8a5-4e8370c0feba-kube-api-access-8qjnp\") pod \"c223ea22-9414-4132-b8a5-4e8370c0feba\" (UID: \"c223ea22-9414-4132-b8a5-4e8370c0feba\") "
Sep 18 18:27:55.951789 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:55.951677    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c223ea22-9414-4132-b8a5-4e8370c0feba-webhook-cert\") pod \"c223ea22-9414-4132-b8a5-4e8370c0feba\" (UID: \"c223ea22-9414-4132-b8a5-4e8370c0feba\") "
Sep 18 18:27:55.953693 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:55.953659    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c223ea22-9414-4132-b8a5-4e8370c0feba-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c223ea22-9414-4132-b8a5-4e8370c0feba" (UID: "c223ea22-9414-4132-b8a5-4e8370c0feba"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:27:55.953786 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:55.953725    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c223ea22-9414-4132-b8a5-4e8370c0feba-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "c223ea22-9414-4132-b8a5-4e8370c0feba" (UID: "c223ea22-9414-4132-b8a5-4e8370c0feba"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:27:55.953786 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:55.953740    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c223ea22-9414-4132-b8a5-4e8370c0feba-kube-api-access-8qjnp" (OuterVolumeSpecName: "kube-api-access-8qjnp") pod "c223ea22-9414-4132-b8a5-4e8370c0feba" (UID: "c223ea22-9414-4132-b8a5-4e8370c0feba"). InnerVolumeSpecName "kube-api-access-8qjnp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:27:56.053365 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:56.053299    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c223ea22-9414-4132-b8a5-4e8370c0feba-metrics-certs\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:27:56.053365 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:56.053322    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qjnp\" (UniqueName: \"kubernetes.io/projected/c223ea22-9414-4132-b8a5-4e8370c0feba-kube-api-access-8qjnp\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:27:56.053365 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:56.053331    2561 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c223ea22-9414-4132-b8a5-4e8370c0feba-webhook-cert\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:27:56.480615 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:56.480577    2561 generic.go:356] "Generic (PLEG): container finished" podID="c223ea22-9414-4132-b8a5-4e8370c0feba" containerID="09083b71d020ea2bd18ddf62f1a04f2bcd061fae2c263fff98e8b0992aded275" exitCode=0
Sep 18 18:27:56.481047 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:56.480619    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-655d8db75-998cr"
Sep 18 18:27:56.481047 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:56.480666    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-655d8db75-998cr" event={"ID":"c223ea22-9414-4132-b8a5-4e8370c0feba","Type":"ContainerDied","Data":"09083b71d020ea2bd18ddf62f1a04f2bcd061fae2c263fff98e8b0992aded275"}
Sep 18 18:27:56.481047 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:56.480721    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-655d8db75-998cr" event={"ID":"c223ea22-9414-4132-b8a5-4e8370c0feba","Type":"ContainerDied","Data":"11725f8bb516d2d2cff4a839660fc915815ad7a56ad6e54a68ec82923d24be89"}
Sep 18 18:27:56.481047 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:56.480743    2561 scope.go:117] "RemoveContainer" containerID="09083b71d020ea2bd18ddf62f1a04f2bcd061fae2c263fff98e8b0992aded275"
Sep 18 18:27:56.490316 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:56.490295    2561 scope.go:117] "RemoveContainer" containerID="09083b71d020ea2bd18ddf62f1a04f2bcd061fae2c263fff98e8b0992aded275"
Sep 18 18:27:56.490605 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:27:56.490583    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09083b71d020ea2bd18ddf62f1a04f2bcd061fae2c263fff98e8b0992aded275\": container with ID starting with 09083b71d020ea2bd18ddf62f1a04f2bcd061fae2c263fff98e8b0992aded275 not found: ID does not exist" containerID="09083b71d020ea2bd18ddf62f1a04f2bcd061fae2c263fff98e8b0992aded275"
Sep 18 18:27:56.490658 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:56.490613    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09083b71d020ea2bd18ddf62f1a04f2bcd061fae2c263fff98e8b0992aded275"} err="failed to get container status \"09083b71d020ea2bd18ddf62f1a04f2bcd061fae2c263fff98e8b0992aded275\": rpc error: code = NotFound desc = could not find container \"09083b71d020ea2bd18ddf62f1a04f2bcd061fae2c263fff98e8b0992aded275\": container with ID starting with 09083b71d020ea2bd18ddf62f1a04f2bcd061fae2c263fff98e8b0992aded275 not found: ID does not exist"
Sep 18 18:27:56.502553 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:56.502532    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-655d8db75-998cr"]
Sep 18 18:27:56.504928 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:56.504906    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-655d8db75-998cr"]
Sep 18 18:27:57.132970 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:57.132934    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:27:57.235489 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:57.235457    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-655d8db75-r9k8b"]
Sep 18 18:27:57.235838 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:57.235826    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c223ea22-9414-4132-b8a5-4e8370c0feba" containerName="manager"
Sep 18 18:27:57.235892 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:57.235839    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c223ea22-9414-4132-b8a5-4e8370c0feba" containerName="manager"
Sep 18 18:27:57.235933 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:57.235913    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c223ea22-9414-4132-b8a5-4e8370c0feba" containerName="manager"
Sep 18 18:27:57.240362 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:57.240343    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-655d8db75-r9k8b"
Sep 18 18:27:57.243285 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:57.243261    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 18 18:27:57.243409 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:57.243290    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 18 18:27:57.243409 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:57.243299    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-f744x\""
Sep 18 18:27:57.243409 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:57.243299    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 18 18:27:57.246854 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:57.246828    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-655d8db75-r9k8b"]
Sep 18 18:27:57.366429 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:57.366396    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c7d3e70a-05e7-4b50-a1e5-ee24f88fcbf6-webhook-cert\") pod \"maas-controller-655d8db75-r9k8b\" (UID: \"c7d3e70a-05e7-4b50-a1e5-ee24f88fcbf6\") " pod="opendatahub/maas-controller-655d8db75-r9k8b"
Sep 18 18:27:57.366572 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:57.366437    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lwjr\" (UniqueName: \"kubernetes.io/projected/c7d3e70a-05e7-4b50-a1e5-ee24f88fcbf6-kube-api-access-6lwjr\") pod \"maas-controller-655d8db75-r9k8b\" (UID: \"c7d3e70a-05e7-4b50-a1e5-ee24f88fcbf6\") " pod="opendatahub/maas-controller-655d8db75-r9k8b"
Sep 18 18:27:57.366572 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:57.366454    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7d3e70a-05e7-4b50-a1e5-ee24f88fcbf6-metrics-certs\") pod \"maas-controller-655d8db75-r9k8b\" (UID: \"c7d3e70a-05e7-4b50-a1e5-ee24f88fcbf6\") " pod="opendatahub/maas-controller-655d8db75-r9k8b"
Sep 18 18:27:57.467824 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:57.467790    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c7d3e70a-05e7-4b50-a1e5-ee24f88fcbf6-webhook-cert\") pod \"maas-controller-655d8db75-r9k8b\" (UID: \"c7d3e70a-05e7-4b50-a1e5-ee24f88fcbf6\") " pod="opendatahub/maas-controller-655d8db75-r9k8b"
Sep 18 18:27:57.468028 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:57.467834    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lwjr\" (UniqueName: \"kubernetes.io/projected/c7d3e70a-05e7-4b50-a1e5-ee24f88fcbf6-kube-api-access-6lwjr\") pod \"maas-controller-655d8db75-r9k8b\" (UID: \"c7d3e70a-05e7-4b50-a1e5-ee24f88fcbf6\") " pod="opendatahub/maas-controller-655d8db75-r9k8b"
Sep 18 18:27:57.468028 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:57.467851    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7d3e70a-05e7-4b50-a1e5-ee24f88fcbf6-metrics-certs\") pod \"maas-controller-655d8db75-r9k8b\" (UID: \"c7d3e70a-05e7-4b50-a1e5-ee24f88fcbf6\") " pod="opendatahub/maas-controller-655d8db75-r9k8b"
Sep 18 18:27:57.470417 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:57.470394    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7d3e70a-05e7-4b50-a1e5-ee24f88fcbf6-metrics-certs\") pod \"maas-controller-655d8db75-r9k8b\" (UID: \"c7d3e70a-05e7-4b50-a1e5-ee24f88fcbf6\") " pod="opendatahub/maas-controller-655d8db75-r9k8b"
Sep 18 18:27:57.470417 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:57.470409    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c7d3e70a-05e7-4b50-a1e5-ee24f88fcbf6-webhook-cert\") pod \"maas-controller-655d8db75-r9k8b\" (UID: \"c7d3e70a-05e7-4b50-a1e5-ee24f88fcbf6\") " pod="opendatahub/maas-controller-655d8db75-r9k8b"
Sep 18 18:27:57.476073 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:57.476051    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lwjr\" (UniqueName: \"kubernetes.io/projected/c7d3e70a-05e7-4b50-a1e5-ee24f88fcbf6-kube-api-access-6lwjr\") pod \"maas-controller-655d8db75-r9k8b\" (UID: \"c7d3e70a-05e7-4b50-a1e5-ee24f88fcbf6\") " pod="opendatahub/maas-controller-655d8db75-r9k8b"
Sep 18 18:27:57.552510 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:57.552475    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-655d8db75-r9k8b"
Sep 18 18:27:57.823357 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:57.823279    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-655d8db75-r9k8b"]
Sep 18 18:27:57.826382 ip-10-0-140-54 kubenswrapper[2561]: W0918 18:27:57.826345    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7d3e70a_05e7_4b50_a1e5_ee24f88fcbf6.slice/crio-764cf1892cf92aa0969debc83e3c994ba214c42ed65e96410040a1915703b516 WatchSource:0}: Error finding container 764cf1892cf92aa0969debc83e3c994ba214c42ed65e96410040a1915703b516: Status 404 returned error can't find the container with id 764cf1892cf92aa0969debc83e3c994ba214c42ed65e96410040a1915703b516
Sep 18 18:27:57.868077 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:57.868044    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c223ea22-9414-4132-b8a5-4e8370c0feba" path="/var/lib/kubelet/pods/c223ea22-9414-4132-b8a5-4e8370c0feba/volumes"
Sep 18 18:27:58.492822 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:58.492742    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-655d8db75-r9k8b" event={"ID":"c7d3e70a-05e7-4b50-a1e5-ee24f88fcbf6","Type":"ContainerStarted","Data":"296f6fb7acbf429529db629899a8afcb9040c9010c40f5bb7f85f80fb7341269"}
Sep 18 18:27:58.492822 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:58.492795    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-655d8db75-r9k8b" event={"ID":"c7d3e70a-05e7-4b50-a1e5-ee24f88fcbf6","Type":"ContainerStarted","Data":"764cf1892cf92aa0969debc83e3c994ba214c42ed65e96410040a1915703b516"}
Sep 18 18:27:58.493003 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:58.492863    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-655d8db75-r9k8b"
Sep 18 18:27:58.513895 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:27:58.513845    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-655d8db75-r9k8b" podStartSLOduration=1.105642589 podStartE2EDuration="1.513823566s" totalImagesPullingTime="408.180977ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:27:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:27:58.512444697 +0000 UTC m=+2283.139576318" watchObservedRunningTime="2026-09-18 18:27:58.513823566 +0000 UTC m=+2283.140955185"
Sep 18 18:28:09.500245 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:28:09.500214    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-655d8db75-r9k8b"
Sep 18 18:28:54.593383 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:28:54.593300    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:29:00.775729 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:29:00.775697    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:29:00.836834 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:29:00.836803    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:29:56.000153 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:29:56.000110    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-5kjlz_75f2a1f7-95a7-462e-9ea7-23c51dd29599/console-operator/1.log"
Sep 18 18:29:56.007238 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:29:56.007216    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkcgk_c5d026b7-af6b-4899-85f4-6554c0f7cd52/ovn-acl-logging/0.log"
Sep 18 18:29:56.007646 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:29:56.007624    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-5kjlz_75f2a1f7-95a7-462e-9ea7-23c51dd29599/console-operator/1.log"
Sep 18 18:29:56.014236 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:29:56.014219    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkcgk_c5d026b7-af6b-4899-85f4-6554c0f7cd52/ovn-acl-logging/0.log"
Sep 18 18:31:03.893918 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:31:03.893888    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:31:16.426310 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:31:16.426273    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-1ebdcfb2-kserve-c9648c57f-hnrj2"]
Sep 18 18:31:16.429683 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:31:16.429664    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-1ebdcfb2-kserve-c9648c57f-hnrj2"
Sep 18 18:31:16.431844 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:31:16.431825    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-1ebdcfb2-kserve-self-signed-certs\""
Sep 18 18:31:16.439559 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:31:16.439538    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-1ebdcfb2-kserve-c9648c57f-hnrj2"]
Sep 18 18:31:16.452834 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:31:16.452793    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0eb2545-3277-4cc7-ad06-d8c521cfe52b-tls-certs\") pod \"e2e-same-id-a-1ebdcfb2-kserve-c9648c57f-hnrj2\" (UID: \"e0eb2545-3277-4cc7-ad06-d8c521cfe52b\") " pod="llm/e2e-same-id-a-1ebdcfb2-kserve-c9648c57f-hnrj2"
Sep 18 18:31:16.452946 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:31:16.452895    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0eb2545-3277-4cc7-ad06-d8c521cfe52b-model-cache\") pod \"e2e-same-id-a-1ebdcfb2-kserve-c9648c57f-hnrj2\" (UID: \"e0eb2545-3277-4cc7-ad06-d8c521cfe52b\") " pod="llm/e2e-same-id-a-1ebdcfb2-kserve-c9648c57f-hnrj2"
Sep 18 18:31:16.453027 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:31:16.452959    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0eb2545-3277-4cc7-ad06-d8c521cfe52b-home\") pod \"e2e-same-id-a-1ebdcfb2-kserve-c9648c57f-hnrj2\" (UID: \"e0eb2545-3277-4cc7-ad06-d8c521cfe52b\") " pod="llm/e2e-same-id-a-1ebdcfb2-kserve-c9648c57f-hnrj2"
Sep 18 18:31:16.453027 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:31:16.452984    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0eb2545-3277-4cc7-ad06-d8c521cfe52b-tmp-dir\") pod \"e2e-same-id-a-1ebdcfb2-kserve-c9648c57f-hnrj2\" (UID: \"e0eb2545-3277-4cc7-ad06-d8c521cfe52b\") " pod="llm/e2e-same-id-a-1ebdcfb2-kserve-c9648c57f-hnrj2"
Sep 18 18:31:16.453161 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:31:16.453039    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjxhc\" (UniqueName: \"kubernetes.io/projected/e0eb2545-3277-4cc7-ad06-d8c521cfe52b-kube-api-access-mjxhc\") pod \"e2e-same-id-a-1ebdcfb2-kserve-c9648c57f-hnrj2\" (UID: \"e0eb2545-3277-4cc7-ad06-d8c521cfe52b\") " pod="llm/e2e-same-id-a-1ebdcfb2-kserve-c9648c57f-hnrj2"
Sep 18 18:31:16.453286 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:31:16.453256    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0eb2545-3277-4cc7-ad06-d8c521cfe52b-dshm\") pod \"e2e-same-id-a-1ebdcfb2-kserve-c9648c57f-hnrj2\" (UID: \"e0eb2545-3277-4cc7-ad06-d8c521cfe52b\") " pod="llm/e2e-same-id-a-1ebdcfb2-kserve-c9648c57f-hnrj2"
Sep 18 18:31:16.462550 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:31:16.462531    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:31:16.554216 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:31:16.554186    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0eb2545-3277-4cc7-ad06-d8c521cfe52b-tls-certs\") pod \"e2e-same-id-a-1ebdcfb2-kserve-c9648c57f-hnrj2\" (UID: \"e0eb2545-3277-4cc7-ad06-d8c521cfe52b\") " pod="llm/e2e-same-id-a-1ebdcfb2-kserve-c9648c57f-hnrj2"
Sep 18 18:31:16.554364 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:31:16.554244    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0eb2545-3277-4cc7-ad06-d8c521cfe52b-model-cache\") pod \"e2e-same-id-a-1ebdcfb2-kserve-c9648c57f-hnrj2\" (UID: \"e0eb2545-3277-4cc7-ad06-d8c521cfe52b\") " pod="llm/e2e-same-id-a-1ebdcfb2-kserve-c9648c57f-hnrj2"
Sep 18 18:31:16.554364 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:31:16.554275    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0eb2545-3277-4cc7-ad06-d8c521cfe52b-home\") pod \"e2e-same-id-a-1ebdcfb2-kserve-c9648c57f-hnrj2\" (UID: \"e0eb2545-3277-4cc7-ad06-d8c521cfe52b\") " pod="llm/e2e-same-id-a-1ebdcfb2-kserve-c9648c57f-hnrj2"
Sep 18 18:31:16.554364 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:31:16.554292    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0eb2545-3277-4cc7-ad06-d8c521cfe52b-tmp-dir\") pod \"e2e-same-id-a-1ebdcfb2-kserve-c9648c57f-hnrj2\" (UID: \"e0eb2545-3277-4cc7-ad06-d8c521cfe52b\") " pod="llm/e2e-same-id-a-1ebdcfb2-kserve-c9648c57f-hnrj2"
Sep 18 18:31:16.554364 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:31:16.554335    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjxhc\" (UniqueName: \"kubernetes.io/projected/e0eb2545-3277-4cc7-ad06-d8c521cfe52b-kube-api-access-mjxhc\") pod \"e2e-same-id-a-1ebdcfb2-kserve-c9648c57f-hnrj2\" (UID: \"e0eb2545-3277-4cc7-ad06-d8c521cfe52b\") " pod="llm/e2e-same-id-a-1ebdcfb2-kserve-c9648c57f-hnrj2"
Sep 18 18:31:16.554364 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:31:16.554363    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0eb2545-3277-4cc7-ad06-d8c521cfe52b-dshm\") pod \"e2e-same-id-a-1ebdcfb2-kserve-c9648c57f-hnrj2\" (UID: \"e0eb2545-3277-4cc7-ad06-d8c521cfe52b\") " pod="llm/e2e-same-id-a-1ebdcfb2-kserve-c9648c57f-hnrj2"
Sep 18 18:31:16.554791 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:31:16.554763    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0eb2545-3277-4cc7-ad06-d8c521cfe52b-model-cache\") pod \"e2e-same-id-a-1ebdcfb2-kserve-c9648c57f-hnrj2\" (UID: \"e0eb2545-3277-4cc7-ad06-d8c521cfe52b\") " pod="llm/e2e-same-id-a-1ebdcfb2-kserve-c9648c57f-hnrj2"
Sep 18 18:31:16.554890 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:31:16.554799    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0eb2545-3277-4cc7-ad06-d8c521cfe52b-home\") pod \"e2e-same-id-a-1ebdcfb2-kserve-c9648c57f-hnrj2\" (UID: \"e0eb2545-3277-4cc7-ad06-d8c521cfe52b\") " pod="llm/e2e-same-id-a-1ebdcfb2-kserve-c9648c57f-hnrj2"
Sep 18 18:31:16.554890 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:31:16.554827    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0eb2545-3277-4cc7-ad06-d8c521cfe52b-tmp-dir\") pod \"e2e-same-id-a-1ebdcfb2-kserve-c9648c57f-hnrj2\" (UID: \"e0eb2545-3277-4cc7-ad06-d8c521cfe52b\") " pod="llm/e2e-same-id-a-1ebdcfb2-kserve-c9648c57f-hnrj2"
Sep 18 18:31:16.556569 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:31:16.556540    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0eb2545-3277-4cc7-ad06-d8c521cfe52b-dshm\") pod \"e2e-same-id-a-1ebdcfb2-kserve-c9648c57f-hnrj2\" (UID: \"e0eb2545-3277-4cc7-ad06-d8c521cfe52b\") " pod="llm/e2e-same-id-a-1ebdcfb2-kserve-c9648c57f-hnrj2"
Sep 18 18:31:16.556668 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:31:16.556652    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0eb2545-3277-4cc7-ad06-d8c521cfe52b-tls-certs\") pod \"e2e-same-id-a-1ebdcfb2-kserve-c9648c57f-hnrj2\" (UID: \"e0eb2545-3277-4cc7-ad06-d8c521cfe52b\") " pod="llm/e2e-same-id-a-1ebdcfb2-kserve-c9648c57f-hnrj2"
Sep 18 18:31:16.562311 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:31:16.562289    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjxhc\" (UniqueName: \"kubernetes.io/projected/e0eb2545-3277-4cc7-ad06-d8c521cfe52b-kube-api-access-mjxhc\") pod \"e2e-same-id-a-1ebdcfb2-kserve-c9648c57f-hnrj2\" (UID: \"e0eb2545-3277-4cc7-ad06-d8c521cfe52b\") " pod="llm/e2e-same-id-a-1ebdcfb2-kserve-c9648c57f-hnrj2"
Sep 18 18:31:16.740047 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:31:16.739970    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-1ebdcfb2-kserve-c9648c57f-hnrj2"
Sep 18 18:31:17.014402 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:31:17.014370    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-1ebdcfb2-kserve-c9648c57f-hnrj2"]
Sep 18 18:31:17.016616 ip-10-0-140-54 kubenswrapper[2561]: W0918 18:31:17.016591    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode0eb2545_3277_4cc7_ad06_d8c521cfe52b.slice/crio-80b2830edef20958ff3d63723f989da4fa5082d8606f0b0b50e87fa3bdf88abf WatchSource:0}: Error finding container 80b2830edef20958ff3d63723f989da4fa5082d8606f0b0b50e87fa3bdf88abf: Status 404 returned error can't find the container with id 80b2830edef20958ff3d63723f989da4fa5082d8606f0b0b50e87fa3bdf88abf
Sep 18 18:31:17.425099 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:31:17.425064    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-1ebdcfb2-kserve-c9648c57f-hnrj2" event={"ID":"e0eb2545-3277-4cc7-ad06-d8c521cfe52b","Type":"ContainerStarted","Data":"cb24e2b54379cf82df47dfa602d4e91a3ad301129896c2b5e512ee06cdf71e3a"}
Sep 18 18:31:17.425304 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:31:17.425111    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-1ebdcfb2-kserve-c9648c57f-hnrj2" event={"ID":"e0eb2545-3277-4cc7-ad06-d8c521cfe52b","Type":"ContainerStarted","Data":"80b2830edef20958ff3d63723f989da4fa5082d8606f0b0b50e87fa3bdf88abf"}
Sep 18 18:31:17.442104 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:31:17.442047    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-1ebdcfb2-kserve-c9648c57f-hnrj2" podStartSLOduration=1.442026953 podStartE2EDuration="1.442026953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:31:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:31:17.441182754 +0000 UTC m=+2482.068314373" watchObservedRunningTime="2026-09-18 18:31:17.442026953 +0000 UTC m=+2482.069158575"
Sep 18 18:31:26.740220 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:31:26.740187    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-1ebdcfb2-kserve-c9648c57f-hnrj2"
Sep 18 18:31:26.740709 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:31:26.740463    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-1ebdcfb2-kserve-c9648c57f-hnrj2"
Sep 18 18:31:26.752802 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:31:26.752778    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-1ebdcfb2-kserve-c9648c57f-hnrj2"
Sep 18 18:31:27.485807 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:31:27.485781    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-1ebdcfb2-kserve-c9648c57f-hnrj2"
Sep 18 18:31:27.813547 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:31:27.813474    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-1ebdcfb2-kserve-6b97446575-sj6jg"]
Sep 18 18:31:27.817385 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:31:27.817361    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-1ebdcfb2-kserve-6b97446575-sj6jg"
Sep 18 18:31:27.819781 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:31:27.819756    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-1ebdcfb2-kserve-self-signed-certs\""
Sep 18 18:31:27.825608 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:31:27.825587    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-1ebdcfb2-kserve-6b97446575-sj6jg"]
Sep 18 18:31:27.844038 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:31:27.844016    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:31:27.853668 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:31:27.853646    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61e4e0d6-d88b-44af-a2d2-7038e41ca5c7-model-cache\") pod \"e2e-same-id-b-1ebdcfb2-kserve-6b97446575-sj6jg\" (UID: \"61e4e0d6-d88b-44af-a2d2-7038e41ca5c7\") " pod="llm/e2e-same-id-b-1ebdcfb2-kserve-6b97446575-sj6jg"
Sep 18 18:31:27.853775 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:31:27.853693    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61e4e0d6-d88b-44af-a2d2-7038e41ca5c7-tmp-dir\") pod \"e2e-same-id-b-1ebdcfb2-kserve-6b97446575-sj6jg\" (UID: \"61e4e0d6-d88b-44af-a2d2-7038e41ca5c7\") " pod="llm/e2e-same-id-b-1ebdcfb2-kserve-6b97446575-sj6jg"
Sep 18 18:31:27.853775 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:31:27.853720    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61e4e0d6-d88b-44af-a2d2-7038e41ca5c7-home\") pod \"e2e-same-id-b-1ebdcfb2-kserve-6b97446575-sj6jg\" (UID: \"61e4e0d6-d88b-44af-a2d2-7038e41ca5c7\") " pod="llm/e2e-same-id-b-1ebdcfb2-kserve-6b97446575-sj6jg"
Sep 18 18:31:27.853904 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:31:27.853814    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61e4e0d6-d88b-44af-a2d2-7038e41ca5c7-dshm\") pod \"e2e-same-id-b-1ebdcfb2-kserve-6b97446575-sj6jg\" (UID: \"61e4e0d6-d88b-44af-a2d2-7038e41ca5c7\") " pod="llm/e2e-same-id-b-1ebdcfb2-kserve-6b97446575-sj6jg"
Sep 18 18:31:27.853904 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:31:27.853869    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6xbg\" (UniqueName: \"kubernetes.io/projected/61e4e0d6-d88b-44af-a2d2-7038e41ca5c7-kube-api-access-p6xbg\") pod \"e2e-same-id-b-1ebdcfb2-kserve-6b97446575-sj6jg\" (UID: \"61e4e0d6-d88b-44af-a2d2-7038e41ca5c7\") " pod="llm/e2e-same-id-b-1ebdcfb2-kserve-6b97446575-sj6jg"
Sep 18 18:31:27.853998 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:31:27.853903    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61e4e0d6-d88b-44af-a2d2-7038e41ca5c7-tls-certs\") pod \"e2e-same-id-b-1ebdcfb2-kserve-6b97446575-sj6jg\" (UID: \"61e4e0d6-d88b-44af-a2d2-7038e41ca5c7\") " pod="llm/e2e-same-id-b-1ebdcfb2-kserve-6b97446575-sj6jg"
Sep 18 18:31:27.954497 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:31:27.954468    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61e4e0d6-d88b-44af-a2d2-7038e41ca5c7-model-cache\") pod \"e2e-same-id-b-1ebdcfb2-kserve-6b97446575-sj6jg\" (UID: \"61e4e0d6-d88b-44af-a2d2-7038e41ca5c7\") " pod="llm/e2e-same-id-b-1ebdcfb2-kserve-6b97446575-sj6jg"
Sep 18 18:31:27.954675 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:31:27.954519    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61e4e0d6-d88b-44af-a2d2-7038e41ca5c7-tmp-dir\") pod \"e2e-same-id-b-1ebdcfb2-kserve-6b97446575-sj6jg\" (UID: \"61e4e0d6-d88b-44af-a2d2-7038e41ca5c7\") " pod="llm/e2e-same-id-b-1ebdcfb2-kserve-6b97446575-sj6jg"
Sep 18 18:31:27.954675 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:31:27.954540    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61e4e0d6-d88b-44af-a2d2-7038e41ca5c7-home\") pod \"e2e-same-id-b-1ebdcfb2-kserve-6b97446575-sj6jg\" (UID: \"61e4e0d6-d88b-44af-a2d2-7038e41ca5c7\") " pod="llm/e2e-same-id-b-1ebdcfb2-kserve-6b97446575-sj6jg"
Sep 18 18:31:27.954675 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:31:27.954579    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61e4e0d6-d88b-44af-a2d2-7038e41ca5c7-dshm\") pod \"e2e-same-id-b-1ebdcfb2-kserve-6b97446575-sj6jg\" (UID: \"61e4e0d6-d88b-44af-a2d2-7038e41ca5c7\") " pod="llm/e2e-same-id-b-1ebdcfb2-kserve-6b97446575-sj6jg"
Sep 18 18:31:27.954675 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:31:27.954600    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6xbg\" (UniqueName: \"kubernetes.io/projected/61e4e0d6-d88b-44af-a2d2-7038e41ca5c7-kube-api-access-p6xbg\") pod \"e2e-same-id-b-1ebdcfb2-kserve-6b97446575-sj6jg\" (UID: \"61e4e0d6-d88b-44af-a2d2-7038e41ca5c7\") " pod="llm/e2e-same-id-b-1ebdcfb2-kserve-6b97446575-sj6jg"
Sep 18 18:31:27.954675 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:31:27.954622    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61e4e0d6-d88b-44af-a2d2-7038e41ca5c7-tls-certs\") pod \"e2e-same-id-b-1ebdcfb2-kserve-6b97446575-sj6jg\" (UID: \"61e4e0d6-d88b-44af-a2d2-7038e41ca5c7\") " pod="llm/e2e-same-id-b-1ebdcfb2-kserve-6b97446575-sj6jg"
Sep 18 18:31:27.954944 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:31:27.954899    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61e4e0d6-d88b-44af-a2d2-7038e41ca5c7-model-cache\") pod \"e2e-same-id-b-1ebdcfb2-kserve-6b97446575-sj6jg\" (UID: \"61e4e0d6-d88b-44af-a2d2-7038e41ca5c7\") " pod="llm/e2e-same-id-b-1ebdcfb2-kserve-6b97446575-sj6jg"
Sep 18 18:31:27.955002 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:31:27.954956    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61e4e0d6-d88b-44af-a2d2-7038e41ca5c7-home\") pod \"e2e-same-id-b-1ebdcfb2-kserve-6b97446575-sj6jg\" (UID: \"61e4e0d6-d88b-44af-a2d2-7038e41ca5c7\") " pod="llm/e2e-same-id-b-1ebdcfb2-kserve-6b97446575-sj6jg"
Sep 18 18:31:27.955042 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:31:27.954999    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61e4e0d6-d88b-44af-a2d2-7038e41ca5c7-tmp-dir\") pod \"e2e-same-id-b-1ebdcfb2-kserve-6b97446575-sj6jg\" (UID: \"61e4e0d6-d88b-44af-a2d2-7038e41ca5c7\") " pod="llm/e2e-same-id-b-1ebdcfb2-kserve-6b97446575-sj6jg"
Sep 18 18:31:27.956844 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:31:27.956822    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61e4e0d6-d88b-44af-a2d2-7038e41ca5c7-dshm\") pod \"e2e-same-id-b-1ebdcfb2-kserve-6b97446575-sj6jg\" (UID: \"61e4e0d6-d88b-44af-a2d2-7038e41ca5c7\") " pod="llm/e2e-same-id-b-1ebdcfb2-kserve-6b97446575-sj6jg"
Sep 18 18:31:27.957108 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:31:27.957091    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61e4e0d6-d88b-44af-a2d2-7038e41ca5c7-tls-certs\") pod \"e2e-same-id-b-1ebdcfb2-kserve-6b97446575-sj6jg\" (UID: \"61e4e0d6-d88b-44af-a2d2-7038e41ca5c7\") " pod="llm/e2e-same-id-b-1ebdcfb2-kserve-6b97446575-sj6jg"
Sep 18 18:31:27.962183 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:31:27.962160    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6xbg\" (UniqueName: \"kubernetes.io/projected/61e4e0d6-d88b-44af-a2d2-7038e41ca5c7-kube-api-access-p6xbg\") pod \"e2e-same-id-b-1ebdcfb2-kserve-6b97446575-sj6jg\" (UID: \"61e4e0d6-d88b-44af-a2d2-7038e41ca5c7\") " pod="llm/e2e-same-id-b-1ebdcfb2-kserve-6b97446575-sj6jg"
Sep 18 18:31:28.128621 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:31:28.128542    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-1ebdcfb2-kserve-6b97446575-sj6jg"
Sep 18 18:31:28.400094 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:31:28.400066    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-1ebdcfb2-kserve-6b97446575-sj6jg"]
Sep 18 18:31:28.403506 ip-10-0-140-54 kubenswrapper[2561]: W0918 18:31:28.403481    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod61e4e0d6_d88b_44af_a2d2_7038e41ca5c7.slice/crio-f59349f09369282d4ed1b1245c7b19f8ca8d751be3d6262cebcaddd1a63591c7 WatchSource:0}: Error finding container f59349f09369282d4ed1b1245c7b19f8ca8d751be3d6262cebcaddd1a63591c7: Status 404 returned error can't find the container with id f59349f09369282d4ed1b1245c7b19f8ca8d751be3d6262cebcaddd1a63591c7
Sep 18 18:31:28.481129 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:31:28.481080    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-1ebdcfb2-kserve-6b97446575-sj6jg" event={"ID":"61e4e0d6-d88b-44af-a2d2-7038e41ca5c7","Type":"ContainerStarted","Data":"7eeef4558e8ed7780d1b95cf45d55fb6c708e0062a9d0f841e76cdd3355698e4"}
Sep 18 18:31:28.481264 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:31:28.481147    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-1ebdcfb2-kserve-6b97446575-sj6jg" event={"ID":"61e4e0d6-d88b-44af-a2d2-7038e41ca5c7","Type":"ContainerStarted","Data":"f59349f09369282d4ed1b1245c7b19f8ca8d751be3d6262cebcaddd1a63591c7"}
Sep 18 18:31:28.500423 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:31:28.500378    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-1ebdcfb2-kserve-6b97446575-sj6jg" podStartSLOduration=1.5003629649999999 podStartE2EDuration="1.500362965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:31:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:31:28.498389497 +0000 UTC m=+2493.125521117" watchObservedRunningTime="2026-09-18 18:31:28.500362965 +0000 UTC m=+2493.127494585"
Sep 18 18:31:36.289498 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:31:36.289461    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:31:36.391516 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:31:36.391485    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:31:38.129447 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:31:38.129415    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-1ebdcfb2-kserve-6b97446575-sj6jg"
Sep 18 18:31:38.129830 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:31:38.129479    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-1ebdcfb2-kserve-6b97446575-sj6jg"
Sep 18 18:31:38.142201 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:31:38.142177    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-1ebdcfb2-kserve-6b97446575-sj6jg"
Sep 18 18:31:38.538886 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:31:38.538859    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-1ebdcfb2-kserve-6b97446575-sj6jg"
Sep 18 18:31:48.659437 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:31:48.658838    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-1ebdcfb2-kserve-c9648c57f-hnrj2"]
Sep 18 18:31:48.682011 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:31:48.681981    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:31:50.329599 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:31:50.329563    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-1ebdcfb2-kserve-6b97446575-sj6jg"]
Sep 18 18:31:50.349722 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:31:50.349693    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:32:03.680575 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:03.680533    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-1ebdcfb2-kserve-c9648c57f-hnrj2" podUID="e0eb2545-3277-4cc7-ad06-d8c521cfe52b" containerName="main" containerID="cri-o://cb24e2b54379cf82df47dfa602d4e91a3ad301129896c2b5e512ee06cdf71e3a" gracePeriod=45
Sep 18 18:32:03.927415 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:03.927392    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-1ebdcfb2-kserve-c9648c57f-hnrj2"
Sep 18 18:32:03.980076 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:03.980000    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0eb2545-3277-4cc7-ad06-d8c521cfe52b-dshm\") pod \"e0eb2545-3277-4cc7-ad06-d8c521cfe52b\" (UID: \"e0eb2545-3277-4cc7-ad06-d8c521cfe52b\") "
Sep 18 18:32:03.980076 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:03.980053    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0eb2545-3277-4cc7-ad06-d8c521cfe52b-tmp-dir\") pod \"e0eb2545-3277-4cc7-ad06-d8c521cfe52b\" (UID: \"e0eb2545-3277-4cc7-ad06-d8c521cfe52b\") "
Sep 18 18:32:03.980281 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:03.980101    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0eb2545-3277-4cc7-ad06-d8c521cfe52b-home\") pod \"e0eb2545-3277-4cc7-ad06-d8c521cfe52b\" (UID: \"e0eb2545-3277-4cc7-ad06-d8c521cfe52b\") "
Sep 18 18:32:03.980281 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:03.980164    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0eb2545-3277-4cc7-ad06-d8c521cfe52b-tls-certs\") pod \"e0eb2545-3277-4cc7-ad06-d8c521cfe52b\" (UID: \"e0eb2545-3277-4cc7-ad06-d8c521cfe52b\") "
Sep 18 18:32:03.980281 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:03.980191    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0eb2545-3277-4cc7-ad06-d8c521cfe52b-model-cache\") pod \"e0eb2545-3277-4cc7-ad06-d8c521cfe52b\" (UID: \"e0eb2545-3277-4cc7-ad06-d8c521cfe52b\") "
Sep 18 18:32:03.980738 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:03.980314    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mjxhc\" (UniqueName: \"kubernetes.io/projected/e0eb2545-3277-4cc7-ad06-d8c521cfe52b-kube-api-access-mjxhc\") pod \"e0eb2545-3277-4cc7-ad06-d8c521cfe52b\" (UID: \"e0eb2545-3277-4cc7-ad06-d8c521cfe52b\") "
Sep 18 18:32:03.980738 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:03.980376    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0eb2545-3277-4cc7-ad06-d8c521cfe52b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e0eb2545-3277-4cc7-ad06-d8c521cfe52b" (UID: "e0eb2545-3277-4cc7-ad06-d8c521cfe52b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:32:03.980738 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:03.980418    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0eb2545-3277-4cc7-ad06-d8c521cfe52b-home" (OuterVolumeSpecName: "home") pod "e0eb2545-3277-4cc7-ad06-d8c521cfe52b" (UID: "e0eb2545-3277-4cc7-ad06-d8c521cfe52b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:32:03.980738 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:03.980513    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0eb2545-3277-4cc7-ad06-d8c521cfe52b-model-cache" (OuterVolumeSpecName: "model-cache") pod "e0eb2545-3277-4cc7-ad06-d8c521cfe52b" (UID: "e0eb2545-3277-4cc7-ad06-d8c521cfe52b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:32:03.980738 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:03.980670    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0eb2545-3277-4cc7-ad06-d8c521cfe52b-tmp-dir\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:32:03.980738 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:03.980690    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0eb2545-3277-4cc7-ad06-d8c521cfe52b-home\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:32:03.980738 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:03.980708    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0eb2545-3277-4cc7-ad06-d8c521cfe52b-model-cache\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:32:03.982388 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:03.982360    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0eb2545-3277-4cc7-ad06-d8c521cfe52b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e0eb2545-3277-4cc7-ad06-d8c521cfe52b" (UID: "e0eb2545-3277-4cc7-ad06-d8c521cfe52b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:32:03.982388 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:03.982377    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0eb2545-3277-4cc7-ad06-d8c521cfe52b-kube-api-access-mjxhc" (OuterVolumeSpecName: "kube-api-access-mjxhc") pod "e0eb2545-3277-4cc7-ad06-d8c521cfe52b" (UID: "e0eb2545-3277-4cc7-ad06-d8c521cfe52b"). InnerVolumeSpecName "kube-api-access-mjxhc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:32:03.982388 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:03.982368    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0eb2545-3277-4cc7-ad06-d8c521cfe52b-dshm" (OuterVolumeSpecName: "dshm") pod "e0eb2545-3277-4cc7-ad06-d8c521cfe52b" (UID: "e0eb2545-3277-4cc7-ad06-d8c521cfe52b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:32:04.081195 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:04.081165    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0eb2545-3277-4cc7-ad06-d8c521cfe52b-tls-certs\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:32:04.081195 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:04.081196    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjxhc\" (UniqueName: \"kubernetes.io/projected/e0eb2545-3277-4cc7-ad06-d8c521cfe52b-kube-api-access-mjxhc\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:32:04.081373 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:04.081209    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0eb2545-3277-4cc7-ad06-d8c521cfe52b-dshm\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:32:04.662861 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:04.662826    2561 generic.go:356] "Generic (PLEG): container finished" podID="e0eb2545-3277-4cc7-ad06-d8c521cfe52b" containerID="cb24e2b54379cf82df47dfa602d4e91a3ad301129896c2b5e512ee06cdf71e3a" exitCode=0
Sep 18 18:32:04.663042 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:04.662873    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-1ebdcfb2-kserve-c9648c57f-hnrj2"
Sep 18 18:32:04.663042 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:04.662915    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-1ebdcfb2-kserve-c9648c57f-hnrj2" event={"ID":"e0eb2545-3277-4cc7-ad06-d8c521cfe52b","Type":"ContainerDied","Data":"cb24e2b54379cf82df47dfa602d4e91a3ad301129896c2b5e512ee06cdf71e3a"}
Sep 18 18:32:04.663042 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:04.662961    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-1ebdcfb2-kserve-c9648c57f-hnrj2" event={"ID":"e0eb2545-3277-4cc7-ad06-d8c521cfe52b","Type":"ContainerDied","Data":"80b2830edef20958ff3d63723f989da4fa5082d8606f0b0b50e87fa3bdf88abf"}
Sep 18 18:32:04.663042 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:04.662978    2561 scope.go:117] "RemoveContainer" containerID="cb24e2b54379cf82df47dfa602d4e91a3ad301129896c2b5e512ee06cdf71e3a"
Sep 18 18:32:04.671973 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:04.671953    2561 scope.go:117] "RemoveContainer" containerID="cb24e2b54379cf82df47dfa602d4e91a3ad301129896c2b5e512ee06cdf71e3a"
Sep 18 18:32:04.672265 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:32:04.672241    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb24e2b54379cf82df47dfa602d4e91a3ad301129896c2b5e512ee06cdf71e3a\": container with ID starting with cb24e2b54379cf82df47dfa602d4e91a3ad301129896c2b5e512ee06cdf71e3a not found: ID does not exist" containerID="cb24e2b54379cf82df47dfa602d4e91a3ad301129896c2b5e512ee06cdf71e3a"
Sep 18 18:32:04.672347 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:04.672268    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb24e2b54379cf82df47dfa602d4e91a3ad301129896c2b5e512ee06cdf71e3a"} err="failed to get container status \"cb24e2b54379cf82df47dfa602d4e91a3ad301129896c2b5e512ee06cdf71e3a\": rpc error: code = NotFound desc = could not find container \"cb24e2b54379cf82df47dfa602d4e91a3ad301129896c2b5e512ee06cdf71e3a\": container with ID starting with cb24e2b54379cf82df47dfa602d4e91a3ad301129896c2b5e512ee06cdf71e3a not found: ID does not exist"
Sep 18 18:32:04.684112 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:04.684089    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-1ebdcfb2-kserve-c9648c57f-hnrj2"]
Sep 18 18:32:04.687587 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:04.687567    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-1ebdcfb2-kserve-c9648c57f-hnrj2"]
Sep 18 18:32:05.352716 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:05.352652    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-1ebdcfb2-kserve-6b97446575-sj6jg" podUID="61e4e0d6-d88b-44af-a2d2-7038e41ca5c7" containerName="main" containerID="cri-o://7eeef4558e8ed7780d1b95cf45d55fb6c708e0062a9d0f841e76cdd3355698e4" gracePeriod=45
Sep 18 18:32:05.602762 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:05.602707    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-1ebdcfb2-kserve-6b97446575-sj6jg"
Sep 18 18:32:05.670403 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:05.670372    2561 generic.go:356] "Generic (PLEG): container finished" podID="61e4e0d6-d88b-44af-a2d2-7038e41ca5c7" containerID="7eeef4558e8ed7780d1b95cf45d55fb6c708e0062a9d0f841e76cdd3355698e4" exitCode=0
Sep 18 18:32:05.670536 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:05.670409    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-1ebdcfb2-kserve-6b97446575-sj6jg"
Sep 18 18:32:05.670536 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:05.670432    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-1ebdcfb2-kserve-6b97446575-sj6jg" event={"ID":"61e4e0d6-d88b-44af-a2d2-7038e41ca5c7","Type":"ContainerDied","Data":"7eeef4558e8ed7780d1b95cf45d55fb6c708e0062a9d0f841e76cdd3355698e4"}
Sep 18 18:32:05.670536 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:05.670463    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-1ebdcfb2-kserve-6b97446575-sj6jg" event={"ID":"61e4e0d6-d88b-44af-a2d2-7038e41ca5c7","Type":"ContainerDied","Data":"f59349f09369282d4ed1b1245c7b19f8ca8d751be3d6262cebcaddd1a63591c7"}
Sep 18 18:32:05.670536 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:05.670485    2561 scope.go:117] "RemoveContainer" containerID="7eeef4558e8ed7780d1b95cf45d55fb6c708e0062a9d0f841e76cdd3355698e4"
Sep 18 18:32:05.678773 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:05.678751    2561 scope.go:117] "RemoveContainer" containerID="7eeef4558e8ed7780d1b95cf45d55fb6c708e0062a9d0f841e76cdd3355698e4"
Sep 18 18:32:05.679033 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:32:05.679012    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7eeef4558e8ed7780d1b95cf45d55fb6c708e0062a9d0f841e76cdd3355698e4\": container with ID starting with 7eeef4558e8ed7780d1b95cf45d55fb6c708e0062a9d0f841e76cdd3355698e4 not found: ID does not exist" containerID="7eeef4558e8ed7780d1b95cf45d55fb6c708e0062a9d0f841e76cdd3355698e4"
Sep 18 18:32:05.679086 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:05.679044    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7eeef4558e8ed7780d1b95cf45d55fb6c708e0062a9d0f841e76cdd3355698e4"} err="failed to get container status \"7eeef4558e8ed7780d1b95cf45d55fb6c708e0062a9d0f841e76cdd3355698e4\": rpc error: code = NotFound desc = could not find container \"7eeef4558e8ed7780d1b95cf45d55fb6c708e0062a9d0f841e76cdd3355698e4\": container with ID starting with 7eeef4558e8ed7780d1b95cf45d55fb6c708e0062a9d0f841e76cdd3355698e4 not found: ID does not exist"
Sep 18 18:32:05.697628 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:05.697593    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p6xbg\" (UniqueName: \"kubernetes.io/projected/61e4e0d6-d88b-44af-a2d2-7038e41ca5c7-kube-api-access-p6xbg\") pod \"61e4e0d6-d88b-44af-a2d2-7038e41ca5c7\" (UID: \"61e4e0d6-d88b-44af-a2d2-7038e41ca5c7\") "
Sep 18 18:32:05.697980 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:05.697646    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61e4e0d6-d88b-44af-a2d2-7038e41ca5c7-dshm\") pod \"61e4e0d6-d88b-44af-a2d2-7038e41ca5c7\" (UID: \"61e4e0d6-d88b-44af-a2d2-7038e41ca5c7\") "
Sep 18 18:32:05.697980 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:05.697676    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61e4e0d6-d88b-44af-a2d2-7038e41ca5c7-home\") pod \"61e4e0d6-d88b-44af-a2d2-7038e41ca5c7\" (UID: \"61e4e0d6-d88b-44af-a2d2-7038e41ca5c7\") "
Sep 18 18:32:05.697980 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:05.697707    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61e4e0d6-d88b-44af-a2d2-7038e41ca5c7-tmp-dir\") pod \"61e4e0d6-d88b-44af-a2d2-7038e41ca5c7\" (UID: \"61e4e0d6-d88b-44af-a2d2-7038e41ca5c7\") "
Sep 18 18:32:05.697980 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:05.697733    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61e4e0d6-d88b-44af-a2d2-7038e41ca5c7-tls-certs\") pod \"61e4e0d6-d88b-44af-a2d2-7038e41ca5c7\" (UID: \"61e4e0d6-d88b-44af-a2d2-7038e41ca5c7\") "
Sep 18 18:32:05.697980 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:05.697766    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61e4e0d6-d88b-44af-a2d2-7038e41ca5c7-model-cache\") pod \"61e4e0d6-d88b-44af-a2d2-7038e41ca5c7\" (UID: \"61e4e0d6-d88b-44af-a2d2-7038e41ca5c7\") "
Sep 18 18:32:05.698256 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:05.697960    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61e4e0d6-d88b-44af-a2d2-7038e41ca5c7-home" (OuterVolumeSpecName: "home") pod "61e4e0d6-d88b-44af-a2d2-7038e41ca5c7" (UID: "61e4e0d6-d88b-44af-a2d2-7038e41ca5c7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:32:05.698256 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:05.697999    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61e4e0d6-d88b-44af-a2d2-7038e41ca5c7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "61e4e0d6-d88b-44af-a2d2-7038e41ca5c7" (UID: "61e4e0d6-d88b-44af-a2d2-7038e41ca5c7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:32:05.698256 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:05.698074    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61e4e0d6-d88b-44af-a2d2-7038e41ca5c7-model-cache" (OuterVolumeSpecName: "model-cache") pod "61e4e0d6-d88b-44af-a2d2-7038e41ca5c7" (UID: "61e4e0d6-d88b-44af-a2d2-7038e41ca5c7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:32:05.698256 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:05.698110    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61e4e0d6-d88b-44af-a2d2-7038e41ca5c7-home\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:32:05.699723 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:05.699700    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61e4e0d6-d88b-44af-a2d2-7038e41ca5c7-dshm" (OuterVolumeSpecName: "dshm") pod "61e4e0d6-d88b-44af-a2d2-7038e41ca5c7" (UID: "61e4e0d6-d88b-44af-a2d2-7038e41ca5c7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:32:05.699814 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:05.699742    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61e4e0d6-d88b-44af-a2d2-7038e41ca5c7-kube-api-access-p6xbg" (OuterVolumeSpecName: "kube-api-access-p6xbg") pod "61e4e0d6-d88b-44af-a2d2-7038e41ca5c7" (UID: "61e4e0d6-d88b-44af-a2d2-7038e41ca5c7"). InnerVolumeSpecName "kube-api-access-p6xbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:32:05.699814 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:05.699740    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61e4e0d6-d88b-44af-a2d2-7038e41ca5c7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "61e4e0d6-d88b-44af-a2d2-7038e41ca5c7" (UID: "61e4e0d6-d88b-44af-a2d2-7038e41ca5c7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:32:05.799149 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:05.799101    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6xbg\" (UniqueName: \"kubernetes.io/projected/61e4e0d6-d88b-44af-a2d2-7038e41ca5c7-kube-api-access-p6xbg\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:32:05.799149 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:05.799146    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61e4e0d6-d88b-44af-a2d2-7038e41ca5c7-dshm\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:32:05.799323 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:05.799176    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61e4e0d6-d88b-44af-a2d2-7038e41ca5c7-tmp-dir\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:32:05.799323 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:05.799184    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61e4e0d6-d88b-44af-a2d2-7038e41ca5c7-tls-certs\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:32:05.799323 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:05.799193    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61e4e0d6-d88b-44af-a2d2-7038e41ca5c7-model-cache\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:32:05.862149 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:05.862072    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0eb2545-3277-4cc7-ad06-d8c521cfe52b" path="/var/lib/kubelet/pods/e0eb2545-3277-4cc7-ad06-d8c521cfe52b/volumes"
Sep 18 18:32:05.985754 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:05.985722    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-1ebdcfb2-kserve-6b97446575-sj6jg"]
Sep 18 18:32:05.989022 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:05.988994    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-1ebdcfb2-kserve-6b97446575-sj6jg"]
Sep 18 18:32:07.861247 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:07.861217    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61e4e0d6-d88b-44af-a2d2-7038e41ca5c7" path="/var/lib/kubelet/pods/61e4e0d6-d88b-44af-a2d2-7038e41ca5c7/volumes"
Sep 18 18:32:09.330449 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:09.330414    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-58fcc54d4-lhnlj"]
Sep 18 18:32:09.331032 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:09.331015    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61e4e0d6-d88b-44af-a2d2-7038e41ca5c7" containerName="main"
Sep 18 18:32:09.331083 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:09.331034    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="61e4e0d6-d88b-44af-a2d2-7038e41ca5c7" containerName="main"
Sep 18 18:32:09.331083 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:09.331048    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0eb2545-3277-4cc7-ad06-d8c521cfe52b" containerName="main"
Sep 18 18:32:09.331083 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:09.331054    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0eb2545-3277-4cc7-ad06-d8c521cfe52b" containerName="main"
Sep 18 18:32:09.331198 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:09.331145    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="61e4e0d6-d88b-44af-a2d2-7038e41ca5c7" containerName="main"
Sep 18 18:32:09.331198 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:09.331154    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0eb2545-3277-4cc7-ad06-d8c521cfe52b" containerName="main"
Sep 18 18:32:09.335434 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:09.335415    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-58fcc54d4-lhnlj"
Sep 18 18:32:09.343772 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:09.343740    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-58fcc54d4-lhnlj"]
Sep 18 18:32:09.435857 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:09.435816    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4x4l\" (UniqueName: \"kubernetes.io/projected/d47a9562-61d9-41ec-a6e7-e0547aa57406-kube-api-access-k4x4l\") pod \"maas-api-58fcc54d4-lhnlj\" (UID: \"d47a9562-61d9-41ec-a6e7-e0547aa57406\") " pod="odh-ai-gateway-infra/maas-api-58fcc54d4-lhnlj"
Sep 18 18:32:09.435857 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:09.435860    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d47a9562-61d9-41ec-a6e7-e0547aa57406-maas-api-tls\") pod \"maas-api-58fcc54d4-lhnlj\" (UID: \"d47a9562-61d9-41ec-a6e7-e0547aa57406\") " pod="odh-ai-gateway-infra/maas-api-58fcc54d4-lhnlj"
Sep 18 18:32:09.436061 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:09.435963    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d47a9562-61d9-41ec-a6e7-e0547aa57406-openshift-service-ca\") pod \"maas-api-58fcc54d4-lhnlj\" (UID: \"d47a9562-61d9-41ec-a6e7-e0547aa57406\") " pod="odh-ai-gateway-infra/maas-api-58fcc54d4-lhnlj"
Sep 18 18:32:09.436061 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:09.435994    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d47a9562-61d9-41ec-a6e7-e0547aa57406-metrics-certs\") pod \"maas-api-58fcc54d4-lhnlj\" (UID: \"d47a9562-61d9-41ec-a6e7-e0547aa57406\") " pod="odh-ai-gateway-infra/maas-api-58fcc54d4-lhnlj"
Sep 18 18:32:09.537503 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:09.537469    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4x4l\" (UniqueName: \"kubernetes.io/projected/d47a9562-61d9-41ec-a6e7-e0547aa57406-kube-api-access-k4x4l\") pod \"maas-api-58fcc54d4-lhnlj\" (UID: \"d47a9562-61d9-41ec-a6e7-e0547aa57406\") " pod="odh-ai-gateway-infra/maas-api-58fcc54d4-lhnlj"
Sep 18 18:32:09.537503 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:09.537510    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d47a9562-61d9-41ec-a6e7-e0547aa57406-maas-api-tls\") pod \"maas-api-58fcc54d4-lhnlj\" (UID: \"d47a9562-61d9-41ec-a6e7-e0547aa57406\") " pod="odh-ai-gateway-infra/maas-api-58fcc54d4-lhnlj"
Sep 18 18:32:09.537755 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:09.537554    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d47a9562-61d9-41ec-a6e7-e0547aa57406-openshift-service-ca\") pod \"maas-api-58fcc54d4-lhnlj\" (UID: \"d47a9562-61d9-41ec-a6e7-e0547aa57406\") " pod="odh-ai-gateway-infra/maas-api-58fcc54d4-lhnlj"
Sep 18 18:32:09.537755 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:09.537571    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d47a9562-61d9-41ec-a6e7-e0547aa57406-metrics-certs\") pod \"maas-api-58fcc54d4-lhnlj\" (UID: \"d47a9562-61d9-41ec-a6e7-e0547aa57406\") " pod="odh-ai-gateway-infra/maas-api-58fcc54d4-lhnlj"
Sep 18 18:32:09.538373 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:09.538351    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d47a9562-61d9-41ec-a6e7-e0547aa57406-openshift-service-ca\") pod \"maas-api-58fcc54d4-lhnlj\" (UID: \"d47a9562-61d9-41ec-a6e7-e0547aa57406\") " pod="odh-ai-gateway-infra/maas-api-58fcc54d4-lhnlj"
Sep 18 18:32:09.539952 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:09.539933    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d47a9562-61d9-41ec-a6e7-e0547aa57406-metrics-certs\") pod \"maas-api-58fcc54d4-lhnlj\" (UID: \"d47a9562-61d9-41ec-a6e7-e0547aa57406\") " pod="odh-ai-gateway-infra/maas-api-58fcc54d4-lhnlj"
Sep 18 18:32:09.540070 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:09.540052    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d47a9562-61d9-41ec-a6e7-e0547aa57406-maas-api-tls\") pod \"maas-api-58fcc54d4-lhnlj\" (UID: \"d47a9562-61d9-41ec-a6e7-e0547aa57406\") " pod="odh-ai-gateway-infra/maas-api-58fcc54d4-lhnlj"
Sep 18 18:32:09.545618 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:09.545597    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4x4l\" (UniqueName: \"kubernetes.io/projected/d47a9562-61d9-41ec-a6e7-e0547aa57406-kube-api-access-k4x4l\") pod \"maas-api-58fcc54d4-lhnlj\" (UID: \"d47a9562-61d9-41ec-a6e7-e0547aa57406\") " pod="odh-ai-gateway-infra/maas-api-58fcc54d4-lhnlj"
Sep 18 18:32:09.649012 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:09.648969    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-58fcc54d4-lhnlj"
Sep 18 18:32:09.928791 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:09.928766    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-58fcc54d4-lhnlj"]
Sep 18 18:32:09.931178 ip-10-0-140-54 kubenswrapper[2561]: W0918 18:32:09.931151    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd47a9562_61d9_41ec_a6e7_e0547aa57406.slice/crio-77bd9ee92557d5e717ce47e6f3a8676295b38af8f47b3b4d9ac08c34d5522bd0 WatchSource:0}: Error finding container 77bd9ee92557d5e717ce47e6f3a8676295b38af8f47b3b4d9ac08c34d5522bd0: Status 404 returned error can't find the container with id 77bd9ee92557d5e717ce47e6f3a8676295b38af8f47b3b4d9ac08c34d5522bd0
Sep 18 18:32:10.702143 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:10.702095    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-58fcc54d4-lhnlj" event={"ID":"d47a9562-61d9-41ec-a6e7-e0547aa57406","Type":"ContainerStarted","Data":"6ef3aeacdc597c8c710e24338f7893e0f35f1c5e37f327860d41711cb2715226"}
Sep 18 18:32:10.702520 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:10.702175    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-58fcc54d4-lhnlj"
Sep 18 18:32:10.702520 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:10.702203    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-58fcc54d4-lhnlj" event={"ID":"d47a9562-61d9-41ec-a6e7-e0547aa57406","Type":"ContainerStarted","Data":"77bd9ee92557d5e717ce47e6f3a8676295b38af8f47b3b4d9ac08c34d5522bd0"}
Sep 18 18:32:10.719332 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:10.719281    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-58fcc54d4-lhnlj" podStartSLOduration=1.139134097 podStartE2EDuration="1.719261006s" totalImagesPullingTime="580.126909ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:32:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:32:10.718245344 +0000 UTC m=+2535.345376974" watchObservedRunningTime="2026-09-18 18:32:10.719261006 +0000 UTC m=+2535.346392626"
Sep 18 18:32:16.713136 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:16.713098    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-58fcc54d4-lhnlj"
Sep 18 18:32:24.137369 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:24.137330    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7dddcbdd5b-rvczn"]
Sep 18 18:32:24.137919 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:24.137542    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-7dddcbdd5b-rvczn" podUID="f4487ef5-ff26-41ba-9991-b52904cd0910" containerName="maas-api" containerID="cri-o://2a5a9f1eabd20fe67cd94dfac5ab2a0298fb6b6078128c2a004a4004f37efc55" gracePeriod=30
Sep 18 18:32:24.722357 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:24.722315    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-7dddcbdd5b-rvczn" podUID="f4487ef5-ff26-41ba-9991-b52904cd0910" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.38:8443/health\": dial tcp 10.132.0.38:8443: connect: connection refused"
Sep 18 18:32:25.387720 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:25.387696    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7dddcbdd5b-rvczn"
Sep 18 18:32:25.483226 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:25.483153    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5nt52\" (UniqueName: \"kubernetes.io/projected/f4487ef5-ff26-41ba-9991-b52904cd0910-kube-api-access-5nt52\") pod \"f4487ef5-ff26-41ba-9991-b52904cd0910\" (UID: \"f4487ef5-ff26-41ba-9991-b52904cd0910\") "
Sep 18 18:32:25.483360 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:25.483236    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4487ef5-ff26-41ba-9991-b52904cd0910-openshift-service-ca\") pod \"f4487ef5-ff26-41ba-9991-b52904cd0910\" (UID: \"f4487ef5-ff26-41ba-9991-b52904cd0910\") "
Sep 18 18:32:25.483360 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:25.483279    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4487ef5-ff26-41ba-9991-b52904cd0910-maas-api-tls\") pod \"f4487ef5-ff26-41ba-9991-b52904cd0910\" (UID: \"f4487ef5-ff26-41ba-9991-b52904cd0910\") "
Sep 18 18:32:25.483360 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:25.483316    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4487ef5-ff26-41ba-9991-b52904cd0910-metrics-certs\") pod \"f4487ef5-ff26-41ba-9991-b52904cd0910\" (UID: \"f4487ef5-ff26-41ba-9991-b52904cd0910\") "
Sep 18 18:32:25.483572 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:25.483542    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4487ef5-ff26-41ba-9991-b52904cd0910-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f4487ef5-ff26-41ba-9991-b52904cd0910" (UID: "f4487ef5-ff26-41ba-9991-b52904cd0910"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:32:25.485342 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:25.485316    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4487ef5-ff26-41ba-9991-b52904cd0910-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f4487ef5-ff26-41ba-9991-b52904cd0910" (UID: "f4487ef5-ff26-41ba-9991-b52904cd0910"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:32:25.485457 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:25.485339    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4487ef5-ff26-41ba-9991-b52904cd0910-kube-api-access-5nt52" (OuterVolumeSpecName: "kube-api-access-5nt52") pod "f4487ef5-ff26-41ba-9991-b52904cd0910" (UID: "f4487ef5-ff26-41ba-9991-b52904cd0910"). InnerVolumeSpecName "kube-api-access-5nt52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:32:25.485457 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:25.485368    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4487ef5-ff26-41ba-9991-b52904cd0910-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f4487ef5-ff26-41ba-9991-b52904cd0910" (UID: "f4487ef5-ff26-41ba-9991-b52904cd0910"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:32:25.583898 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:25.583867    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nt52\" (UniqueName: \"kubernetes.io/projected/f4487ef5-ff26-41ba-9991-b52904cd0910-kube-api-access-5nt52\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:32:25.583898 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:25.583893    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4487ef5-ff26-41ba-9991-b52904cd0910-openshift-service-ca\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:32:25.583898 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:25.583903    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4487ef5-ff26-41ba-9991-b52904cd0910-maas-api-tls\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:32:25.584136 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:25.583911    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4487ef5-ff26-41ba-9991-b52904cd0910-metrics-certs\") on node \"ip-10-0-140-54.ec2.internal\" DevicePath \"\""
Sep 18 18:32:25.770325 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:25.770253    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:32:25.777355 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:25.777326    2561 generic.go:356] "Generic (PLEG): container finished" podID="f4487ef5-ff26-41ba-9991-b52904cd0910" containerID="2a5a9f1eabd20fe67cd94dfac5ab2a0298fb6b6078128c2a004a4004f37efc55" exitCode=0
Sep 18 18:32:25.777476 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:25.777373    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7dddcbdd5b-rvczn"
Sep 18 18:32:25.777476 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:25.777412    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7dddcbdd5b-rvczn" event={"ID":"f4487ef5-ff26-41ba-9991-b52904cd0910","Type":"ContainerDied","Data":"2a5a9f1eabd20fe67cd94dfac5ab2a0298fb6b6078128c2a004a4004f37efc55"}
Sep 18 18:32:25.777553 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:25.777489    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7dddcbdd5b-rvczn" event={"ID":"f4487ef5-ff26-41ba-9991-b52904cd0910","Type":"ContainerDied","Data":"7333b45ea626b3f322ecd54a670af0b1ad15880686bf278dbaba0f8123ddc8c8"}
Sep 18 18:32:25.777553 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:25.777512    2561 scope.go:117] "RemoveContainer" containerID="2a5a9f1eabd20fe67cd94dfac5ab2a0298fb6b6078128c2a004a4004f37efc55"
Sep 18 18:32:25.788982 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:25.788966    2561 scope.go:117] "RemoveContainer" containerID="2a5a9f1eabd20fe67cd94dfac5ab2a0298fb6b6078128c2a004a4004f37efc55"
Sep 18 18:32:25.789259 ip-10-0-140-54 kubenswrapper[2561]: E0918 18:32:25.789238    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a5a9f1eabd20fe67cd94dfac5ab2a0298fb6b6078128c2a004a4004f37efc55\": container with ID starting with 2a5a9f1eabd20fe67cd94dfac5ab2a0298fb6b6078128c2a004a4004f37efc55 not found: ID does not exist" containerID="2a5a9f1eabd20fe67cd94dfac5ab2a0298fb6b6078128c2a004a4004f37efc55"
Sep 18 18:32:25.789311 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:25.789270    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a5a9f1eabd20fe67cd94dfac5ab2a0298fb6b6078128c2a004a4004f37efc55"} err="failed to get container status \"2a5a9f1eabd20fe67cd94dfac5ab2a0298fb6b6078128c2a004a4004f37efc55\": rpc error: code = NotFound desc = could not find container \"2a5a9f1eabd20fe67cd94dfac5ab2a0298fb6b6078128c2a004a4004f37efc55\": container with ID starting with 2a5a9f1eabd20fe67cd94dfac5ab2a0298fb6b6078128c2a004a4004f37efc55 not found: ID does not exist"
Sep 18 18:32:25.809606 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:25.809583    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7dddcbdd5b-rvczn"]
Sep 18 18:32:25.812347 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:25.812325    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-7dddcbdd5b-rvczn"]
Sep 18 18:32:25.861316 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:25.861295    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4487ef5-ff26-41ba-9991-b52904cd0910" path="/var/lib/kubelet/pods/f4487ef5-ff26-41ba-9991-b52904cd0910/volumes"
Sep 18 18:32:42.492406 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:42.492378    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:32:44.301870 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:44.301838    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-655d8db75-r9k8b_c7d3e70a-05e7-4b50-a1e5-ee24f88fcbf6/manager/0.log"
Sep 18 18:32:44.648031 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:44.648003    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-649dc7b766-qqtwg_89b6292a-2396-4d3e-88bb-f645a5db8832/copy-manifests/0.log"
Sep 18 18:32:44.655998 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:44.655976    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-649dc7b766-qqtwg_89b6292a-2396-4d3e-88bb-f645a5db8832/manager/0.log"
Sep 18 18:32:45.208568 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:45.208499    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-58fcc54d4-lhnlj_d47a9562-61d9-41ec-a6e7-e0547aa57406/maas-api/0.log"
Sep 18 18:32:45.642718 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:45.642634    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-646hr_d8ac5f0d-0c80-4a92-b41d-d49c6a66cbf5/postgres/0.log"
Sep 18 18:32:47.741327 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:47.741298    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6b7b7f7d54-gwfj9_626cdc85-2717-4abb-82e8-77da8fb69033/kube-auth-proxy/0.log"
Sep 18 18:32:47.964152 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:47.964127    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-fp2fb_5d9b729f-107b-4ab4-8516-42a1731679a5/payload-pre-processing/0.log"
Sep 18 18:32:48.075495 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:48.075422    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-j77pl_d1e487ec-d2f3-45a0-9b0c-0a14e89ea981/payload-processing/0.log"
Sep 18 18:32:48.179018 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:48.178990    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7dc96485b6-66p6f_20117f3e-e50b-4b03-aa48-604fc42d1737/router/0.log"
Sep 18 18:32:48.603026 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:48.602989    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-n26lb_b21f8cd0-20c7-4c0c-ae49-4bc752a38065/main/0.log"
Sep 18 18:32:49.131441 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:49.131415    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-j4hwc_d091ce3f-fbef-456b-b91c-217133bb3a09/main/0.log"
Sep 18 18:32:54.760439 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:54.760400    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tkl5n/must-gather-tmjpp"]
Sep 18 18:32:54.760821 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:54.760792    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4487ef5-ff26-41ba-9991-b52904cd0910" containerName="maas-api"
Sep 18 18:32:54.760821 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:54.760803    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4487ef5-ff26-41ba-9991-b52904cd0910" containerName="maas-api"
Sep 18 18:32:54.760902 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:54.760858    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4487ef5-ff26-41ba-9991-b52904cd0910" containerName="maas-api"
Sep 18 18:32:54.764221 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:54.764204    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tkl5n/must-gather-tmjpp"
Sep 18 18:32:54.767484 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:54.767458    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tkl5n\"/\"kube-root-ca.crt\""
Sep 18 18:32:54.767587 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:54.767501    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tkl5n\"/\"openshift-service-ca.crt\""
Sep 18 18:32:54.768249 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:54.768229    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-tkl5n\"/\"default-dockercfg-btmrw\""
Sep 18 18:32:54.777615 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:54.777595    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tkl5n/must-gather-tmjpp"]
Sep 18 18:32:54.806884 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:54.806861    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:32:54.851510 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:54.851485    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cwgh\" (UniqueName: \"kubernetes.io/projected/f50c7291-2195-4772-b4ee-f0a8d3ebb94f-kube-api-access-8cwgh\") pod \"must-gather-tmjpp\" (UID: \"f50c7291-2195-4772-b4ee-f0a8d3ebb94f\") " pod="openshift-must-gather-tkl5n/must-gather-tmjpp"
Sep 18 18:32:54.851634 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:54.851512    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f50c7291-2195-4772-b4ee-f0a8d3ebb94f-must-gather-output\") pod \"must-gather-tmjpp\" (UID: \"f50c7291-2195-4772-b4ee-f0a8d3ebb94f\") " pod="openshift-must-gather-tkl5n/must-gather-tmjpp"
Sep 18 18:32:54.903278 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:54.903252    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log" limit=1024
Sep 18 18:32:54.952833 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:54.952807    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cwgh\" (UniqueName: \"kubernetes.io/projected/f50c7291-2195-4772-b4ee-f0a8d3ebb94f-kube-api-access-8cwgh\") pod \"must-gather-tmjpp\" (UID: \"f50c7291-2195-4772-b4ee-f0a8d3ebb94f\") " pod="openshift-must-gather-tkl5n/must-gather-tmjpp"
Sep 18 18:32:54.952957 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:54.952836    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f50c7291-2195-4772-b4ee-f0a8d3ebb94f-must-gather-output\") pod \"must-gather-tmjpp\" (UID: \"f50c7291-2195-4772-b4ee-f0a8d3ebb94f\") " pod="openshift-must-gather-tkl5n/must-gather-tmjpp"
Sep 18 18:32:54.953216 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:54.953198    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f50c7291-2195-4772-b4ee-f0a8d3ebb94f-must-gather-output\") pod \"must-gather-tmjpp\" (UID: \"f50c7291-2195-4772-b4ee-f0a8d3ebb94f\") " pod="openshift-must-gather-tkl5n/must-gather-tmjpp"
Sep 18 18:32:54.961879 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:54.961858    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cwgh\" (UniqueName: \"kubernetes.io/projected/f50c7291-2195-4772-b4ee-f0a8d3ebb94f-kube-api-access-8cwgh\") pod \"must-gather-tmjpp\" (UID: \"f50c7291-2195-4772-b4ee-f0a8d3ebb94f\") " pod="openshift-must-gather-tkl5n/must-gather-tmjpp"
Sep 18 18:32:55.073342 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:55.073252    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tkl5n/must-gather-tmjpp"
Sep 18 18:32:55.351970 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:55.351946    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tkl5n/must-gather-tmjpp"]
Sep 18 18:32:55.354182 ip-10-0-140-54 kubenswrapper[2561]: W0918 18:32:55.354156    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf50c7291_2195_4772_b4ee_f0a8d3ebb94f.slice/crio-d0a2d95de6bfc03a71f5c15a6975fae01112eef40ef33f55c50dfcec2a7f3c75 WatchSource:0}: Error finding container d0a2d95de6bfc03a71f5c15a6975fae01112eef40ef33f55c50dfcec2a7f3c75: Status 404 returned error can't find the container with id d0a2d95de6bfc03a71f5c15a6975fae01112eef40ef33f55c50dfcec2a7f3c75
Sep 18 18:32:55.356000 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:55.355983    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 18:32:55.921302 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:55.921274    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tkl5n/must-gather-tmjpp" event={"ID":"f50c7291-2195-4772-b4ee-f0a8d3ebb94f","Type":"ContainerStarted","Data":"d0a2d95de6bfc03a71f5c15a6975fae01112eef40ef33f55c50dfcec2a7f3c75"}
Sep 18 18:32:56.930456 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:56.930412    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tkl5n/must-gather-tmjpp" event={"ID":"f50c7291-2195-4772-b4ee-f0a8d3ebb94f","Type":"ContainerStarted","Data":"d11446704e6d2b23dfa4f0d125c38d10cbf08096749c11499a37db9f0b7354c7"}
Sep 18 18:32:56.930942 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:56.930468    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tkl5n/must-gather-tmjpp" event={"ID":"f50c7291-2195-4772-b4ee-f0a8d3ebb94f","Type":"ContainerStarted","Data":"49d17eb0f3751f8feba27bc420202d8a5e7aacddb547550dee0614f6c7a80de0"}
Sep 18 18:32:56.948716 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:56.948664    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-tkl5n/must-gather-tmjpp" podStartSLOduration=2.169794521 podStartE2EDuration="2.948647289s" totalImagesPullingTime="778.852768ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:32:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:32:56.946544305 +0000 UTC m=+2581.573675926" watchObservedRunningTime="2026-09-18 18:32:56.948647289 +0000 UTC m=+2581.575778910"
Sep 18 18:32:57.792978 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:57.792941    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-mvtdm_7a3ae2d8-a30b-4c83-a53c-634fb07621d6/global-pull-secret-syncer/0.log"
Sep 18 18:32:57.856287 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:57.856256    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-k2xs7_313dfd8e-da3b-4eeb-9366-5e3a9ddcbe13/konnectivity-agent/0.log"
Sep 18 18:32:58.024395 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:32:58.024367    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-54.ec2.internal_bfcddef9dbb20737aafa5928064e8e0d/haproxy/0.log"
Sep 18 18:33:04.581693 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:33:04.581611    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-847b8bd8d7-xjzc9_a1e4f37f-b0ba-469f-8923-d0c9d38e619b/cluster-monitoring-operator/0.log"
Sep 18 18:33:04.611001 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:33:04.610975    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-knpq6_36b0382b-a6dd-4a2b-b5d3-77c6219666ec/kube-state-metrics/0.log"
Sep 18 18:33:04.687668 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:33:04.687636    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-knpq6_36b0382b-a6dd-4a2b-b5d3-77c6219666ec/kube-rbac-proxy-main/0.log"
Sep 18 18:33:04.760096 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:33:04.760063    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-knpq6_36b0382b-a6dd-4a2b-b5d3-77c6219666ec/kube-rbac-proxy-self/0.log"
Sep 18 18:33:04.840694 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:33:04.840620    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-74bb5f98ff-qbjk5_4eee484c-2603-426b-9e46-936a81e86906/metrics-server/0.log"
Sep 18 18:33:04.897694 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:33:04.897256    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5984459ff8-5mwwt_025ce4f7-ca1b-44fc-924a-fb32d1afca45/monitoring-plugin/0.log"
Sep 18 18:33:05.100891 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:33:05.100769    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-drc7s_ae03e141-49ed-4b4b-9ec3-1fd32d2dacff/node-exporter/0.log"
Sep 18 18:33:05.131480 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:33:05.131386    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-drc7s_ae03e141-49ed-4b4b-9ec3-1fd32d2dacff/kube-rbac-proxy/0.log"
Sep 18 18:33:05.160276 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:33:05.160107    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-drc7s_ae03e141-49ed-4b4b-9ec3-1fd32d2dacff/init-textfile/0.log"
Sep 18 18:33:05.271537 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:33:05.270838    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-kcr2n_fbb000f1-48a6-44b2-929e-0efba2b6dada/kube-rbac-proxy-main/0.log"
Sep 18 18:33:05.320005 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:33:05.319974    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-kcr2n_fbb000f1-48a6-44b2-929e-0efba2b6dada/kube-rbac-proxy-self/0.log"
Sep 18 18:33:05.345618 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:33:05.345586    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-kcr2n_fbb000f1-48a6-44b2-929e-0efba2b6dada/openshift-state-metrics/0.log"
Sep 18 18:33:05.383059 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:33:05.382965    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8b31feaf-fe2d-4a3e-b37d-7c979d716cf0/prometheus/0.log"
Sep 18 18:33:05.407758 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:33:05.407723    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8b31feaf-fe2d-4a3e-b37d-7c979d716cf0/config-reloader/0.log"
Sep 18 18:33:05.430937 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:33:05.430904    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8b31feaf-fe2d-4a3e-b37d-7c979d716cf0/thanos-sidecar/0.log"
Sep 18 18:33:05.456038 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:33:05.456010    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8b31feaf-fe2d-4a3e-b37d-7c979d716cf0/kube-rbac-proxy-web/0.log"
Sep 18 18:33:05.478980 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:33:05.478943    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8b31feaf-fe2d-4a3e-b37d-7c979d716cf0/kube-rbac-proxy/0.log"
Sep 18 18:33:05.508235 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:33:05.508209    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8b31feaf-fe2d-4a3e-b37d-7c979d716cf0/kube-rbac-proxy-thanos/0.log"
Sep 18 18:33:05.535038 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:33:05.535008    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8b31feaf-fe2d-4a3e-b37d-7c979d716cf0/init-config-reloader/0.log"
Sep 18 18:33:05.619378 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:33:05.619343    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5c6747dc7b-qwhfg_4bfe00a4-fcbf-4cd2-a1f2-44cbe3af7c13/prometheus-operator-admission-webhook/0.log"
Sep 18 18:33:07.773318 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:33:07.773290    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-5kjlz_75f2a1f7-95a7-462e-9ea7-23c51dd29599/console-operator/1.log"
Sep 18 18:33:07.780414 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:33:07.780365    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-5kjlz_75f2a1f7-95a7-462e-9ea7-23c51dd29599/console-operator/2.log"
Sep 18 18:33:07.826040 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:33:07.826000    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tkl5n/perf-node-gather-daemonset-zptl8"]
Sep 18 18:33:07.830491 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:33:07.830462    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tkl5n/perf-node-gather-daemonset-zptl8"
Sep 18 18:33:07.844224 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:33:07.843929    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tkl5n/perf-node-gather-daemonset-zptl8"]
Sep 18 18:33:07.897394 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:33:07.897143    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9tjr\" (UniqueName: \"kubernetes.io/projected/98ff89cd-c121-4e4a-8981-abed9c25a13b-kube-api-access-t9tjr\") pod \"perf-node-gather-daemonset-zptl8\" (UID: \"98ff89cd-c121-4e4a-8981-abed9c25a13b\") " pod="openshift-must-gather-tkl5n/perf-node-gather-daemonset-zptl8"
Sep 18 18:33:07.897394 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:33:07.897216    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/98ff89cd-c121-4e4a-8981-abed9c25a13b-lib-modules\") pod \"perf-node-gather-daemonset-zptl8\" (UID: \"98ff89cd-c121-4e4a-8981-abed9c25a13b\") " pod="openshift-must-gather-tkl5n/perf-node-gather-daemonset-zptl8"
Sep 18 18:33:07.897394 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:33:07.897269    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/98ff89cd-c121-4e4a-8981-abed9c25a13b-proc\") pod \"perf-node-gather-daemonset-zptl8\" (UID: \"98ff89cd-c121-4e4a-8981-abed9c25a13b\") " pod="openshift-must-gather-tkl5n/perf-node-gather-daemonset-zptl8"
Sep 18 18:33:07.897394 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:33:07.897302    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/98ff89cd-c121-4e4a-8981-abed9c25a13b-sys\") pod \"perf-node-gather-daemonset-zptl8\" (UID: \"98ff89cd-c121-4e4a-8981-abed9c25a13b\") " pod="openshift-must-gather-tkl5n/perf-node-gather-daemonset-zptl8"
Sep 18 18:33:07.897394 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:33:07.897332    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/98ff89cd-c121-4e4a-8981-abed9c25a13b-podres\") pod \"perf-node-gather-daemonset-zptl8\" (UID: \"98ff89cd-c121-4e4a-8981-abed9c25a13b\") " pod="openshift-must-gather-tkl5n/perf-node-gather-daemonset-zptl8"
Sep 18 18:33:07.998248 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:33:07.998206    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9tjr\" (UniqueName: \"kubernetes.io/projected/98ff89cd-c121-4e4a-8981-abed9c25a13b-kube-api-access-t9tjr\") pod \"perf-node-gather-daemonset-zptl8\" (UID: \"98ff89cd-c121-4e4a-8981-abed9c25a13b\") " pod="openshift-must-gather-tkl5n/perf-node-gather-daemonset-zptl8"
Sep 18 18:33:07.998441 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:33:07.998293    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/98ff89cd-c121-4e4a-8981-abed9c25a13b-lib-modules\") pod \"perf-node-gather-daemonset-zptl8\" (UID: \"98ff89cd-c121-4e4a-8981-abed9c25a13b\") " pod="openshift-must-gather-tkl5n/perf-node-gather-daemonset-zptl8"
Sep 18 18:33:07.998441 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:33:07.998379    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/98ff89cd-c121-4e4a-8981-abed9c25a13b-proc\") pod \"perf-node-gather-daemonset-zptl8\" (UID: \"98ff89cd-c121-4e4a-8981-abed9c25a13b\") " pod="openshift-must-gather-tkl5n/perf-node-gather-daemonset-zptl8"
Sep 18 18:33:07.998441 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:33:07.998410    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/98ff89cd-c121-4e4a-8981-abed9c25a13b-sys\") pod \"perf-node-gather-daemonset-zptl8\" (UID: \"98ff89cd-c121-4e4a-8981-abed9c25a13b\") " pod="openshift-must-gather-tkl5n/perf-node-gather-daemonset-zptl8"
Sep 18 18:33:07.998632 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:33:07.998445    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/98ff89cd-c121-4e4a-8981-abed9c25a13b-podres\") pod \"perf-node-gather-daemonset-zptl8\" (UID: \"98ff89cd-c121-4e4a-8981-abed9c25a13b\") " pod="openshift-must-gather-tkl5n/perf-node-gather-daemonset-zptl8"
Sep 18 18:33:07.998696 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:33:07.998650    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/98ff89cd-c121-4e4a-8981-abed9c25a13b-podres\") pod \"perf-node-gather-daemonset-zptl8\" (UID: \"98ff89cd-c121-4e4a-8981-abed9c25a13b\") " pod="openshift-must-gather-tkl5n/perf-node-gather-daemonset-zptl8"
Sep 18 18:33:07.999828 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:33:07.999803    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/98ff89cd-c121-4e4a-8981-abed9c25a13b-lib-modules\") pod \"perf-node-gather-daemonset-zptl8\" (UID: \"98ff89cd-c121-4e4a-8981-abed9c25a13b\") " pod="openshift-must-gather-tkl5n/perf-node-gather-daemonset-zptl8"
Sep 18 18:33:08.000090 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:33:07.999979    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/98ff89cd-c121-4e4a-8981-abed9c25a13b-proc\") pod \"perf-node-gather-daemonset-zptl8\" (UID: \"98ff89cd-c121-4e4a-8981-abed9c25a13b\") " pod="openshift-must-gather-tkl5n/perf-node-gather-daemonset-zptl8"
Sep 18 18:33:08.000256 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:33:08.000042    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/98ff89cd-c121-4e4a-8981-abed9c25a13b-sys\") pod \"perf-node-gather-daemonset-zptl8\" (UID: \"98ff89cd-c121-4e4a-8981-abed9c25a13b\") " pod="openshift-must-gather-tkl5n/perf-node-gather-daemonset-zptl8"
Sep 18 18:33:08.008785 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:33:08.008760    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9tjr\" (UniqueName: \"kubernetes.io/projected/98ff89cd-c121-4e4a-8981-abed9c25a13b-kube-api-access-t9tjr\") pod \"perf-node-gather-daemonset-zptl8\" (UID: \"98ff89cd-c121-4e4a-8981-abed9c25a13b\") " pod="openshift-must-gather-tkl5n/perf-node-gather-daemonset-zptl8"
Sep 18 18:33:08.145405 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:33:08.145329    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tkl5n/perf-node-gather-daemonset-zptl8"
Sep 18 18:33:08.442990 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:33:08.442948    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tkl5n/perf-node-gather-daemonset-zptl8"]
Sep 18 18:33:08.446098 ip-10-0-140-54 kubenswrapper[2561]: W0918 18:33:08.446053    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod98ff89cd_c121_4e4a_8981_abed9c25a13b.slice/crio-6cdfaf5ef5394147a4b8d553bb1ac5ff6ff1e2ab0a1d7d9cfa1eb5ad8a8532c5 WatchSource:0}: Error finding container 6cdfaf5ef5394147a4b8d553bb1ac5ff6ff1e2ab0a1d7d9cfa1eb5ad8a8532c5: Status 404 returned error can't find the container with id 6cdfaf5ef5394147a4b8d553bb1ac5ff6ff1e2ab0a1d7d9cfa1eb5ad8a8532c5
Sep 18 18:33:09.040157 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:33:09.040079    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tkl5n/perf-node-gather-daemonset-zptl8" event={"ID":"98ff89cd-c121-4e4a-8981-abed9c25a13b","Type":"ContainerStarted","Data":"6baadc3efee34c62fbd2fa318890e5bf4414157b0e00c4f2f3525fa499515bf3"}
Sep 18 18:33:09.040629 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:33:09.040167    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tkl5n/perf-node-gather-daemonset-zptl8" event={"ID":"98ff89cd-c121-4e4a-8981-abed9c25a13b","Type":"ContainerStarted","Data":"6cdfaf5ef5394147a4b8d553bb1ac5ff6ff1e2ab0a1d7d9cfa1eb5ad8a8532c5"}
Sep 18 18:33:09.040629 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:33:09.040212    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-tkl5n/perf-node-gather-daemonset-zptl8"
Sep 18 18:33:09.059668 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:33:09.059603    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-tkl5n/perf-node-gather-daemonset-zptl8" podStartSLOduration=2.059580388 podStartE2EDuration="2.059580388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 18:33:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:33:09.057035626 +0000 UTC m=+2593.684167239" watchObservedRunningTime="2026-09-18 18:33:09.059580388 +0000 UTC m=+2593.686712008"
Sep 18 18:33:09.125780 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:33:09.125740    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5c4db8bdf9-vzcw7_6d0c7ec9-9691-4341-afd6-ba415fdb1326/volume-data-source-validator/0.log"
Sep 18 18:33:10.348589 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:33:10.348560    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9hjjd_60a031b6-31f6-47a2-859c-bfd960f0b2c7/dns/0.log"
Sep 18 18:33:10.375793 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:33:10.375766    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9hjjd_60a031b6-31f6-47a2-859c-bfd960f0b2c7/kube-rbac-proxy/0.log"
Sep 18 18:33:10.576494 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:33:10.576459    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-q2btm_1907e944-be36-42b2-bc90-90938a227371/dns-node-resolver/0.log"
Sep 18 18:33:11.315008 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:33:11.314969    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zvmrv_4945651f-6aa7-4ce0-82e2-3f5f234b607f/node-ca/0.log"
Sep 18 18:33:13.166797 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:33:13.166769    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6b7b7f7d54-gwfj9_626cdc85-2717-4abb-82e8-77da8fb69033/kube-auth-proxy/0.log"
Sep 18 18:33:13.279039 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:33:13.278953    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-fp2fb_5d9b729f-107b-4ab4-8516-42a1731679a5/payload-pre-processing/0.log"
Sep 18 18:33:13.319897 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:33:13.319863    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-j77pl_d1e487ec-d2f3-45a0-9b0c-0a14e89ea981/payload-processing/0.log"
Sep 18 18:33:13.345632 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:33:13.345588    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7dc96485b6-66p6f_20117f3e-e50b-4b03-aa48-604fc42d1737/router/0.log"
Sep 18 18:33:14.334263 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:33:14.334235    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kw7df_ac028d85-f1a2-4c5a-91ad-fb7141335c07/serve-healthcheck-canary/0.log"
Sep 18 18:33:14.951759 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:33:14.951725    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-4hdg8_2e2c3ab1-da7b-40e3-8d33-4d7ec5980634/insights-operator/0.log"
Sep 18 18:33:14.952649 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:33:14.952617    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-4hdg8_2e2c3ab1-da7b-40e3-8d33-4d7ec5980634/insights-operator/1.log"
Sep 18 18:33:15.057895 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:33:15.057869    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-tkl5n/perf-node-gather-daemonset-zptl8"
Sep 18 18:33:15.168362 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:33:15.168328    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cjzln_6e4028c2-33ee-41f9-bb98-a4bbdffe8b24/kube-rbac-proxy/0.log"
Sep 18 18:33:15.234337 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:33:15.234266    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cjzln_6e4028c2-33ee-41f9-bb98-a4bbdffe8b24/exporter/0.log"
Sep 18 18:33:15.294865 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:33:15.294838    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cjzln_6e4028c2-33ee-41f9-bb98-a4bbdffe8b24/extractor/0.log"
Sep 18 18:33:18.628604 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:33:18.628518    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-655d8db75-r9k8b_c7d3e70a-05e7-4b50-a1e5-ee24f88fcbf6/manager/0.log"
Sep 18 18:33:19.013318 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:33:19.013288    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6b7b7f7d54-gwfj9_626cdc85-2717-4abb-82e8-77da8fb69033/kube-auth-proxy/0.log"
Sep 18 18:33:19.147196 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:33:19.147165    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-649dc7b766-qqtwg_89b6292a-2396-4d3e-88bb-f645a5db8832/manager/0.log"
Sep 18 18:33:19.160857 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:33:19.160822    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-fp2fb_5d9b729f-107b-4ab4-8516-42a1731679a5/payload-pre-processing/0.log"
Sep 18 18:33:19.193694 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:33:19.193671    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-j77pl_d1e487ec-d2f3-45a0-9b0c-0a14e89ea981/payload-processing/0.log"
Sep 18 18:33:19.198305 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:33:19.198284    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-649dc7b766-qqtwg_89b6292a-2396-4d3e-88bb-f645a5db8832/copy-manifests/0.log"
Sep 18 18:33:19.218615 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:33:19.218596    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7dc96485b6-66p6f_20117f3e-e50b-4b03-aa48-604fc42d1737/router/0.log"
Sep 18 18:33:20.926034 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:33:20.926005    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-786dfb84b9-bgs76_ab192091-eb80-4dd9-83bc-54ea383c7c09/manager/0.log"
Sep 18 18:33:21.627145 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:33:21.627103    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6b7b7f7d54-gwfj9_626cdc85-2717-4abb-82e8-77da8fb69033/kube-auth-proxy/0.log"
Sep 18 18:33:21.729145 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:33:21.729095    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-fp2fb_5d9b729f-107b-4ab4-8516-42a1731679a5/payload-pre-processing/0.log"
Sep 18 18:33:21.812780 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:33:21.812745    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-j77pl_d1e487ec-d2f3-45a0-9b0c-0a14e89ea981/payload-processing/0.log"
Sep 18 18:33:21.862013 ip-10-0-140-54 kubenswrapper[2561]: I0918 18:33:21.861986    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7dc96485b6-66p6f_20117f3e-e50b-4b03-aa48-604fc42d1737/router/0.log"