Sep 18 13:44:09.406238 ip-10-0-141-222 systemd[1]: Starting Kubernetes Kubelet...
Sep 18 13:44:12.911322 ip-10-0-141-222 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 13:44:12.911322 ip-10-0-141-222 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 18 13:44:12.911322 ip-10-0-141-222 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 13:44:12.911322 ip-10-0-141-222 kubenswrapper[2577]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 18 13:44:12.911322 ip-10-0-141-222 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 13:44:12.912379 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.911408    2577 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 18 13:44:12.918446 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918262    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 13:44:12.918446 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918440    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 13:44:12.918446 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918446    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 13:44:12.918446 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918450    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 13:44:12.918446 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918454    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 13:44:12.918767 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918458    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 13:44:12.918767 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918463    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 13:44:12.918767 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918469    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 13:44:12.918767 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918472    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 13:44:12.918767 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918476    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 13:44:12.918767 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918480    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 13:44:12.918767 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918484    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 13:44:12.918767 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918488    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 13:44:12.918767 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918492    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 13:44:12.918767 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918496    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 13:44:12.918767 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918499    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 13:44:12.918767 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918503    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 13:44:12.918767 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918508    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 13:44:12.918767 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918512    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 13:44:12.918767 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918516    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 13:44:12.918767 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918520    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 13:44:12.918767 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918524    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 13:44:12.918767 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918528    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 13:44:12.918767 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918532    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 13:44:12.918767 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918536    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 13:44:12.919662 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918540    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 13:44:12.919662 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918545    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 13:44:12.919662 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918550    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 13:44:12.919662 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918555    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 13:44:12.919662 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918559    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 13:44:12.919662 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918564    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 13:44:12.919662 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918568    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 13:44:12.919662 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918575    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 13:44:12.919662 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918581    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 13:44:12.919662 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918586    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 13:44:12.919662 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918590    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 13:44:12.919662 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918594    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 13:44:12.919662 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918599    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 13:44:12.919662 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918604    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 13:44:12.919662 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918610    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 13:44:12.919662 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918615    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 13:44:12.919662 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918622    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 13:44:12.919662 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918629    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 13:44:12.919662 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918634    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 13:44:12.920572 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918638    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 13:44:12.920572 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918643    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 13:44:12.920572 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918647    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 13:44:12.920572 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918652    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 13:44:12.920572 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918656    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 13:44:12.920572 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918661    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 13:44:12.920572 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918665    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 13:44:12.920572 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918670    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 13:44:12.920572 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918674    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 13:44:12.920572 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918678    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 13:44:12.920572 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918683    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 13:44:12.920572 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918687    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 13:44:12.920572 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918692    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 13:44:12.920572 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918697    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 13:44:12.920572 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918701    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 13:44:12.920572 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918706    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 13:44:12.920572 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918712    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 13:44:12.920572 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918716    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 13:44:12.920572 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918721    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 13:44:12.920572 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918725    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 13:44:12.921493 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918729    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 13:44:12.921493 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918734    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 13:44:12.921493 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918739    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 13:44:12.921493 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918743    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 13:44:12.921493 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918748    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 13:44:12.921493 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918753    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 13:44:12.921493 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918759    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 13:44:12.921493 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918763    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 13:44:12.921493 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918768    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 13:44:12.921493 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918772    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 13:44:12.921493 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918776    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 13:44:12.921493 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918781    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 13:44:12.921493 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918785    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 13:44:12.921493 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918789    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 13:44:12.921493 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918794    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 13:44:12.921493 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918799    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 13:44:12.921493 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918804    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 13:44:12.921493 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918808    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 13:44:12.921493 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918813    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 13:44:12.922402 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918819    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 13:44:12.922402 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918825    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 13:44:12.922402 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918830    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 13:44:12.922402 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918835    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 13:44:12.922402 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918840    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 13:44:12.922402 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918844    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 13:44:12.922402 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918851    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 13:44:12.922402 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918857    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 13:44:12.922402 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918862    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 13:44:12.922402 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918867    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 13:44:12.922402 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918872    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 13:44:12.922402 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918877    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 13:44:12.922402 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918881    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 13:44:12.922402 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918886    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 13:44:12.922402 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918890    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 13:44:12.922402 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.918912    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 13:44:12.922402 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920154    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 13:44:12.922402 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920166    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 13:44:12.922402 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920172    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 13:44:12.922402 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920177    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 13:44:12.923333 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920182    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 13:44:12.923333 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920187    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 13:44:12.923333 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920191    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 13:44:12.923333 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920196    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 13:44:12.923333 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920201    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 13:44:12.923333 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920205    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 13:44:12.923333 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920210    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 13:44:12.923333 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920214    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 13:44:12.923333 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920219    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 13:44:12.923333 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920223    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 13:44:12.923333 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920228    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 13:44:12.923333 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920232    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 13:44:12.923333 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920237    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 13:44:12.923333 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920242    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 13:44:12.923333 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920247    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 13:44:12.923333 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920251    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 13:44:12.923333 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920255    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 13:44:12.923333 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920261    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 13:44:12.923333 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920265    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 13:44:12.923333 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920270    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 13:44:12.924002 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920274    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 13:44:12.924002 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920279    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 13:44:12.924002 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920284    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 13:44:12.924002 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920288    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 13:44:12.924002 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920293    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 13:44:12.924002 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920298    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 13:44:12.924002 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920302    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 13:44:12.924002 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920307    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 13:44:12.924002 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920311    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 13:44:12.924002 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920315    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 13:44:12.924002 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920321    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 13:44:12.924002 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920326    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 13:44:12.924002 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920333    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 13:44:12.924002 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920341    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 13:44:12.924002 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920349    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 13:44:12.924002 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920354    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 13:44:12.924002 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920359    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 13:44:12.924002 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920363    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 13:44:12.924002 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920368    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 13:44:12.924610 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920372    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 13:44:12.924610 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920377    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 13:44:12.924610 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920381    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 13:44:12.924610 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920385    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 13:44:12.924610 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920390    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 13:44:12.924610 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920394    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 13:44:12.924610 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920399    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 13:44:12.924610 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920405    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 13:44:12.924610 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920409    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 13:44:12.924610 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920414    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 13:44:12.924610 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920418    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 13:44:12.924610 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920425    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 13:44:12.924610 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920430    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 13:44:12.924610 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920437    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 13:44:12.924610 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920442    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 13:44:12.924610 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920446    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 13:44:12.924610 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920452    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 13:44:12.924610 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920457    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 13:44:12.924610 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920462    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 13:44:12.925170 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920467    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 13:44:12.925170 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920472    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 13:44:12.925170 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920477    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 13:44:12.925170 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920482    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 13:44:12.925170 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920486    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 13:44:12.925170 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920492    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 13:44:12.925170 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920496    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 13:44:12.925170 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920501    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 13:44:12.925170 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920506    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 13:44:12.925170 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920510    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 13:44:12.925170 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920515    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 13:44:12.925170 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920520    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 13:44:12.925170 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920524    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 13:44:12.925170 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920529    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 13:44:12.925170 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920533    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 13:44:12.925170 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920538    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 13:44:12.925170 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920544    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 13:44:12.925170 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920549    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 13:44:12.925170 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920553    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 13:44:12.925170 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920558    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 13:44:12.925954 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920562    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 13:44:12.925954 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920568    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 13:44:12.925954 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920573    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 13:44:12.925954 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920577    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 13:44:12.925954 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920581    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 13:44:12.925954 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920585    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 13:44:12.925954 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920590    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 13:44:12.925954 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920595    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 13:44:12.925954 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920599    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 13:44:12.925954 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920604    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 13:44:12.925954 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920608    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 13:44:12.925954 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920612    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 13:44:12.925954 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920617    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 13:44:12.925954 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920622    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 13:44:12.925954 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920626    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 13:44:12.925954 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920631    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 13:44:12.925954 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.920636    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 13:44:12.925954 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921308    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 13:44:12.925954 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921318    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 13:44:12.925954 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921323    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 13:44:12.926540 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921328    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 13:44:12.926540 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921333    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 13:44:12.926540 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921338    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 13:44:12.926540 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921342    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 13:44:12.926540 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921347    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 13:44:12.926540 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921351    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 13:44:12.926540 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921355    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 13:44:12.926540 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921362    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 13:44:12.926540 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921368    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 13:44:12.926540 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921373    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 13:44:12.926540 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921377    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 13:44:12.926540 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921381    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 13:44:12.926540 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921386    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 13:44:12.926540 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921391    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 13:44:12.926540 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921396    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 13:44:12.926540 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921400    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 13:44:12.926540 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921405    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 13:44:12.926540 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921410    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 13:44:12.926540 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921414    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 13:44:12.926540 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921420    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 13:44:12.927230 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921424    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 13:44:12.927230 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921429    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 13:44:12.927230 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921433    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 13:44:12.927230 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921437    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 13:44:12.927230 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921442    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 13:44:12.927230 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921447    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 13:44:12.927230 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921452    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 13:44:12.927230 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921457    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 13:44:12.927230 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921462    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 13:44:12.927230 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921467    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 13:44:12.927230 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921471    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 13:44:12.927230 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921476    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 13:44:12.927230 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921481    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 13:44:12.927230 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921485    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 13:44:12.927230 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921489    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 13:44:12.927230 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921493    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 13:44:12.927230 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921497    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 13:44:12.927230 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921501    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 13:44:12.927230 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921505    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 13:44:12.927230 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921509    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 13:44:12.927768 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921514    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 13:44:12.927768 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921520    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 13:44:12.927768 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921525    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 13:44:12.927768 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921531    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 13:44:12.927768 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921535    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 13:44:12.927768 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921540    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 13:44:12.927768 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921545    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 13:44:12.927768 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921549    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 13:44:12.927768 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921554    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 13:44:12.927768 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921559    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 13:44:12.927768 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921563    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 13:44:12.927768 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921568    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 13:44:12.927768 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921572    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 13:44:12.927768 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921577    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 13:44:12.927768 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921581    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 13:44:12.927768 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921585    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 13:44:12.927768 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921590    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 13:44:12.927768 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921595    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 13:44:12.927768 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921599    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 13:44:12.927768 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921604    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 13:44:12.928355 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921611    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 13:44:12.928355 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921618    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 13:44:12.928355 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921624    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 13:44:12.928355 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921631    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 13:44:12.928355 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921636    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 13:44:12.928355 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921641    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 13:44:12.928355 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921645    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 13:44:12.928355 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921650    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 13:44:12.928355 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921654    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 13:44:12.928355 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921658    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 13:44:12.928355 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921663    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 13:44:12.928355 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921667    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 13:44:12.928355 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921671    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 13:44:12.928355 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921676    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 13:44:12.928355 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921680    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 13:44:12.928355 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921687    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 13:44:12.928355 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921693    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 13:44:12.928355 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921698    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 13:44:12.928355 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921703    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 13:44:12.928844 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921708    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 13:44:12.928844 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921714    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 13:44:12.928844 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921719    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 13:44:12.928844 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921723    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 13:44:12.928844 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921727    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 13:44:12.928844 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921732    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 13:44:12.928844 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921736    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 13:44:12.928844 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921741    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 13:44:12.928844 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921745    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 13:44:12.928844 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921750    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 13:44:12.928844 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921755    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 13:44:12.928844 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921760    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 13:44:12.928844 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921764    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 13:44:12.928844 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921769    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 13:44:12.928844 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921774    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 13:44:12.928844 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921778    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 13:44:12.928844 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.921783    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 13:44:12.928844 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.921921    2577 flags.go:64] FLAG: --address="0.0.0.0"
Sep 18 13:44:12.928844 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.921973    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 18 13:44:12.928844 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.921985    2577 flags.go:64] FLAG: --anonymous-auth="true"
Sep 18 13:44:12.928844 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.921992    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 18 13:44:12.929411 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.921999    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 18 13:44:12.929411 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922005    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 18 13:44:12.929411 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922019    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 18 13:44:12.929411 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922026    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 18 13:44:12.929411 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922032    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 18 13:44:12.929411 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922037    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 18 13:44:12.929411 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922043    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 18 13:44:12.929411 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922048    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 18 13:44:12.929411 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922054    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 18 13:44:12.929411 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922060    2577 flags.go:64] FLAG: --cgroup-root=""
Sep 18 13:44:12.929411 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922064    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 18 13:44:12.929411 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922069    2577 flags.go:64] FLAG: --client-ca-file=""
Sep 18 13:44:12.929411 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922076    2577 flags.go:64] FLAG: --cloud-provider="external"
Sep 18 13:44:12.929411 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922081    2577 flags.go:64] FLAG: --cluster-dns="[]"
Sep 18 13:44:12.929411 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922088    2577 flags.go:64] FLAG: --cluster-domain=""
Sep 18 13:44:12.929411 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922096    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 18 13:44:12.929411 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922102    2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 18 13:44:12.929411 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922107    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 18 13:44:12.929411 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922113    2577 flags.go:64] FLAG: --container-log-max-files="5"
Sep 18 13:44:12.929411 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922120    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 18 13:44:12.929411 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922125    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 18 13:44:12.929411 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922130    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 18 13:44:12.929411 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922137    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 18 13:44:12.930146 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922143    2577 flags.go:64] FLAG: --contention-profiling="false"
Sep 18 13:44:12.930146 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922148    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 18 13:44:12.930146 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922154    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 18 13:44:12.930146 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922159    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 18 13:44:12.930146 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922164    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 18 13:44:12.930146 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922171    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 18 13:44:12.930146 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922176    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 18 13:44:12.930146 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922181    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 18 13:44:12.930146 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922186    2577 flags.go:64] FLAG: --enable-load-reader="false"
Sep 18 13:44:12.930146 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922192    2577 flags.go:64] FLAG: --enable-server="true"
Sep 18 13:44:12.930146 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922197    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 18 13:44:12.930146 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922205    2577 flags.go:64] FLAG: --event-burst="100"
Sep 18 13:44:12.930146 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922210    2577 flags.go:64] FLAG: --event-qps="50"
Sep 18 13:44:12.930146 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922216    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 18 13:44:12.930146 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922221    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 18 13:44:12.930146 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922226    2577 flags.go:64] FLAG: --eviction-hard=""
Sep 18 13:44:12.930146 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922233    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 18 13:44:12.930146 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922238    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 18 13:44:12.930146 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922243    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 18 13:44:12.930146 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922248    2577 flags.go:64] FLAG: --eviction-soft=""
Sep 18 13:44:12.930146 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922253    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 18 13:44:12.930146 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922258    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 18 13:44:12.930146 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922265    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 18 13:44:12.930146 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922271    2577 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 18 13:44:12.930146 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922278    2577 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 18 13:44:12.930848 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922283    2577 flags.go:64] FLAG: --fail-swap-on="true"
Sep 18 13:44:12.930848 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922288    2577 flags.go:64] FLAG: --feature-gates=""
Sep 18 13:44:12.930848 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922294    2577 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 18 13:44:12.930848 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922300    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 18 13:44:12.930848 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922305    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 18 13:44:12.930848 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922310    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 18 13:44:12.930848 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922316    2577 flags.go:64] FLAG: --healthz-port="10248"
Sep 18 13:44:12.930848 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922321    2577 flags.go:64] FLAG: --help="false"
Sep 18 13:44:12.930848 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922327    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-141-222.ec2.internal"
Sep 18 13:44:12.930848 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922332    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 18 13:44:12.930848 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922337    2577 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 18 13:44:12.930848 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922342    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 18 13:44:12.930848 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922348    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 18 13:44:12.930848 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922354    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 18 13:44:12.930848 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922360    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 18 13:44:12.930848 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922365    2577 flags.go:64] FLAG: --image-service-endpoint=""
Sep 18 13:44:12.930848 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922370    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 18 13:44:12.930848 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922374    2577 flags.go:64] FLAG: --kube-api-burst="100"
Sep 18 13:44:12.930848 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922380    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 18 13:44:12.930848 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922385    2577 flags.go:64] FLAG: --kube-api-qps="50"
Sep 18 13:44:12.930848 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922390    2577 flags.go:64] FLAG: --kube-reserved=""
Sep 18 13:44:12.930848 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922395    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 18 13:44:12.930848 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922399    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 18 13:44:12.930848 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922404    2577 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 18 13:44:12.931511 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922408    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 18 13:44:12.931511 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922413    2577 flags.go:64] FLAG: --lock-file=""
Sep 18 13:44:12.931511 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922418    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 18 13:44:12.931511 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922422    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 18 13:44:12.931511 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922427    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 18 13:44:12.931511 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922437    2577 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 18 13:44:12.931511 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922443    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 18 13:44:12.931511 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922450    2577 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 18 13:44:12.931511 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922456    2577 flags.go:64] FLAG: --logging-format="text"
Sep 18 13:44:12.931511 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922460    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 18 13:44:12.931511 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922466    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 18 13:44:12.931511 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922473    2577 flags.go:64] FLAG: --manifest-url=""
Sep 18 13:44:12.931511 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922478    2577 flags.go:64] FLAG: --manifest-url-header=""
Sep 18 13:44:12.931511 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922486    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 18 13:44:12.931511 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922491    2577 flags.go:64] FLAG: --max-open-files="1000000"
Sep 18 13:44:12.931511 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922498    2577 flags.go:64] FLAG: --max-pods="110"
Sep 18 13:44:12.931511 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922504    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 18 13:44:12.931511 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922508    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 18 13:44:12.931511 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922513    2577 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 18 13:44:12.931511 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922519    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 18 13:44:12.931511 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922524    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 18 13:44:12.931511 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922529    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 18 13:44:12.931511 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922535    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 18 13:44:12.931511 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922547    2577 flags.go:64] FLAG: --node-status-max-images="50"
Sep 18 13:44:12.931511 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922552    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 18 13:44:12.932223 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922557    2577 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 18 13:44:12.932223 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922562    2577 flags.go:64] FLAG: --pod-cidr=""
Sep 18 13:44:12.932223 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922567    2577 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 18 13:44:12.932223 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922576    2577 flags.go:64] FLAG: --pod-manifest-path=""
Sep 18 13:44:12.932223 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922581    2577 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 18 13:44:12.932223 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922586    2577 flags.go:64] FLAG: --pods-per-core="0"
Sep 18 13:44:12.932223 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922591    2577 flags.go:64] FLAG: --port="10250"
Sep 18 13:44:12.932223 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922597    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 18 13:44:12.932223 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922602    2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-051eec48086464c28"
Sep 18 13:44:12.932223 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922607    2577 flags.go:64] FLAG: --qos-reserved=""
Sep 18 13:44:12.932223 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922613    2577 flags.go:64] FLAG: --read-only-port="10255"
Sep 18 13:44:12.932223 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922618    2577 flags.go:64] FLAG: --register-node="true"
Sep 18 13:44:12.932223 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922623    2577 flags.go:64] FLAG: --register-with-taints=""
Sep 18 13:44:12.932223 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922630    2577 flags.go:64] FLAG: --registry-burst="10"
Sep 18 13:44:12.932223 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922639    2577 flags.go:64] FLAG: --registry-qps="5"
Sep 18 13:44:12.932223 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922644    2577 flags.go:64] FLAG: --reserved-cpus=""
Sep 18 13:44:12.932223 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922649    2577 flags.go:64] FLAG: --reserved-memory=""
Sep 18 13:44:12.932223 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922655    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 18 13:44:12.932223 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922660    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 18 13:44:12.932223 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922667    2577 flags.go:64] FLAG: --rotate-certificates="false"
Sep 18 13:44:12.932223 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922672    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 18 13:44:12.932223 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922678    2577 flags.go:64] FLAG: --runonce="false"
Sep 18 13:44:12.932223 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922683    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 18 13:44:12.932223 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922688    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 18 13:44:12.932223 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922693    2577 flags.go:64] FLAG: --seccomp-default="false"
Sep 18 13:44:12.932879 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922698    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 18 13:44:12.932879 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922714    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 18 13:44:12.932879 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922720    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 18 13:44:12.932879 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922725    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 18 13:44:12.932879 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922730    2577 flags.go:64] FLAG: --storage-driver-password="root"
Sep 18 13:44:12.932879 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922735    2577 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 18 13:44:12.932879 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922740    2577 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 18 13:44:12.932879 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922745    2577 flags.go:64] FLAG: --storage-driver-user="root"
Sep 18 13:44:12.932879 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922750    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 18 13:44:12.932879 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922756    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 18 13:44:12.932879 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922761    2577 flags.go:64] FLAG: --system-cgroups=""
Sep 18 13:44:12.932879 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922766    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 18 13:44:12.932879 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922775    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 18 13:44:12.932879 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922780    2577 flags.go:64] FLAG: --tls-cert-file=""
Sep 18 13:44:12.932879 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922784    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 18 13:44:12.932879 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922791    2577 flags.go:64] FLAG: --tls-min-version=""
Sep 18 13:44:12.932879 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922795    2577 flags.go:64] FLAG: --tls-private-key-file=""
Sep 18 13:44:12.932879 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922801    2577 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 18 13:44:12.932879 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922806    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 18 13:44:12.932879 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922811    2577 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 18 13:44:12.932879 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922816    2577 flags.go:64] FLAG: --v="2"
Sep 18 13:44:12.932879 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922824    2577 flags.go:64] FLAG: --version="false"
Sep 18 13:44:12.932879 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922832    2577 flags.go:64] FLAG: --vmodule=""
Sep 18 13:44:12.932879 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922838    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 18 13:44:12.932879 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.922844    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 18 13:44:12.933568 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923048    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 13:44:12.933568 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923057    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 13:44:12.933568 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923062    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 13:44:12.933568 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923067    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 13:44:12.933568 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923071    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 13:44:12.933568 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923076    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 13:44:12.933568 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923081    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 13:44:12.933568 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923086    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 13:44:12.933568 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923091    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 13:44:12.933568 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923095    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 13:44:12.933568 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923100    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 13:44:12.933568 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923104    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 13:44:12.933568 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923108    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 13:44:12.933568 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923113    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 13:44:12.933568 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923118    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 13:44:12.933568 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923122    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 13:44:12.933568 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923127    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 13:44:12.933568 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923132    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 13:44:12.933568 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923137    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 13:44:12.933568 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923141    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 13:44:12.933568 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923146    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 13:44:12.934194 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923151    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 13:44:12.934194 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923155    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 13:44:12.934194 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923160    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 13:44:12.934194 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923164    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 13:44:12.934194 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923169    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 13:44:12.934194 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923174    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 13:44:12.934194 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923178    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 13:44:12.934194 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923184    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 13:44:12.934194 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923189    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 13:44:12.934194 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923194    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 13:44:12.934194 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923198    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 13:44:12.934194 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923206    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 13:44:12.934194 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923215    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 13:44:12.934194 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923220    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 13:44:12.934194 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923225    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 13:44:12.934194 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923229    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 13:44:12.934194 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923234    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 13:44:12.934194 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923238    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 13:44:12.934194 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923243    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 13:44:12.934194 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923247    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 13:44:12.934734 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923251    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 13:44:12.934734 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923256    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 13:44:12.934734 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923261    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 13:44:12.934734 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923265    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 13:44:12.934734 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923270    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 13:44:12.934734 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923274    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 13:44:12.934734 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923278    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 13:44:12.934734 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923283    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 13:44:12.934734 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923287    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 13:44:12.934734 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923291    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 13:44:12.934734 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923296    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 13:44:12.934734 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923300    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 13:44:12.934734 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923305    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 13:44:12.934734 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923311    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 13:44:12.934734 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923315    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 13:44:12.934734 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923319    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 13:44:12.934734 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923324    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 13:44:12.934734 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923328    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 13:44:12.934734 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923332    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 13:44:12.934734 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923338    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 13:44:12.935305 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923342    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 13:44:12.935305 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923347    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 13:44:12.935305 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923353    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 13:44:12.935305 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923357    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 13:44:12.935305 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923364    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 13:44:12.935305 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923368    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 13:44:12.935305 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923372    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 13:44:12.935305 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923377    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 13:44:12.935305 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923381    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 13:44:12.935305 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923386    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 13:44:12.935305 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923391    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 13:44:12.935305 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923395    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 13:44:12.935305 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923399    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 13:44:12.935305 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923404    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 13:44:12.935305 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923408    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 13:44:12.935305 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923413    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 13:44:12.935305 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923418    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 13:44:12.935305 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923425    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 13:44:12.935829 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923431    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 13:44:12.935829 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923436    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 13:44:12.935829 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923442    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 13:44:12.935829 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923447    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 13:44:12.935829 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923452    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 13:44:12.935829 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923456    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 13:44:12.935829 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923461    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 13:44:12.935829 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923465    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 13:44:12.935829 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923471    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 13:44:12.935829 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923475    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 13:44:12.935829 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923480    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 13:44:12.935829 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923484    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 13:44:12.935829 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923489    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 13:44:12.935829 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923497    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 13:44:12.935829 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923503    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 13:44:12.935829 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923507    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 13:44:12.935829 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923512    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 13:44:12.935829 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923517    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 13:44:12.935829 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923524    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 13:44:12.935829 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.923529    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 13:44:12.936377 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.924131    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 13:44:12.936377 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.931754    2577 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 18 13:44:12.936377 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.931774    2577 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 18 13:44:12.936377 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.931795    2577 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 18 13:44:12.936377 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.931802    2577 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 18 13:44:12.936377 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.931860    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 13:44:12.936377 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.931866    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 13:44:12.936377 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.931870    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 13:44:12.936377 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.931873    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 13:44:12.936377 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.931876    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 13:44:12.936377 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.931879    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 13:44:12.936377 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.931882    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 13:44:12.936377 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.931885    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 13:44:12.936377 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.931888    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 13:44:12.936768 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.931891    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 13:44:12.936768 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.931894    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 13:44:12.936768 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.931912    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 13:44:12.936768 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.931915    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 13:44:12.936768 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.931920    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 13:44:12.936768 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.931925    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 13:44:12.936768 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.931928    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 13:44:12.936768 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.931932    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 13:44:12.936768 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.931935    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 13:44:12.936768 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.931938    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 13:44:12.936768 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.931941    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 13:44:12.936768 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.931944    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 13:44:12.936768 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.931947    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 13:44:12.936768 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.931950    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 13:44:12.936768 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.931953    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 13:44:12.936768 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.931956    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 13:44:12.936768 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.931958    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 13:44:12.936768 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.931961    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 13:44:12.936768 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.931964    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 13:44:12.936768 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.931967    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 13:44:12.937321 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.931972    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 13:44:12.937321 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.931975    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 13:44:12.937321 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.931978    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 13:44:12.937321 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.931981    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 13:44:12.937321 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.931984    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 13:44:12.937321 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.931987    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 13:44:12.937321 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.931990    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 13:44:12.937321 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.931993    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 13:44:12.937321 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.931996    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 13:44:12.937321 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.931999    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 13:44:12.937321 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932002    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 13:44:12.937321 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932005    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 13:44:12.937321 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932008    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 13:44:12.937321 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932012    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 13:44:12.937321 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932016    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 13:44:12.937321 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932019    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 13:44:12.937321 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932023    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 13:44:12.937321 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932026    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 13:44:12.937321 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932029    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 13:44:12.937822 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932033    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 13:44:12.937822 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932036    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 13:44:12.937822 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932039    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 13:44:12.937822 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932042    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 13:44:12.937822 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932045    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 13:44:12.937822 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932049    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 13:44:12.937822 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932054    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 13:44:12.937822 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932057    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 13:44:12.937822 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932059    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 13:44:12.937822 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932062    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 13:44:12.937822 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932065    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 13:44:12.937822 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932068    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 13:44:12.937822 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932072    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 13:44:12.937822 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932075    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 13:44:12.937822 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932079    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 13:44:12.937822 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932082    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 13:44:12.937822 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932085    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 13:44:12.937822 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932088    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 13:44:12.937822 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932091    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 13:44:12.938383 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932094    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 13:44:12.938383 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932096    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 13:44:12.938383 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932099    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 13:44:12.938383 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932102    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 13:44:12.938383 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932105    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 13:44:12.938383 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932107    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 13:44:12.938383 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932110    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 13:44:12.938383 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932113    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 13:44:12.938383 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932116    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 13:44:12.938383 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932119    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 13:44:12.938383 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932121    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 13:44:12.938383 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932124    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 13:44:12.938383 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932127    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 13:44:12.938383 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932129    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 13:44:12.938383 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932132    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 13:44:12.938383 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932135    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 13:44:12.938383 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932138    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 13:44:12.938383 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932140    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 13:44:12.938383 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932143    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 13:44:12.938383 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932146    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 13:44:12.938918 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932149    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 13:44:12.938918 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932151    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 13:44:12.938918 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932154    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 13:44:12.938918 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932157    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 13:44:12.938918 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932160    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 13:44:12.938918 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932163    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 13:44:12.938918 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932167    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 13:44:12.938918 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932170    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 13:44:12.938918 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932174    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 13:44:12.938918 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932177    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 13:44:12.938918 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932180    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 13:44:12.938918 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932182    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 13:44:12.938918 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.932188    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 13:44:12.938918 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932313    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 13:44:12.938918 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932318    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 13:44:12.939337 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932321    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 13:44:12.939337 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932325    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 13:44:12.939337 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932328    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 13:44:12.939337 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932331    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 13:44:12.939337 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932334    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 13:44:12.939337 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932337    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 13:44:12.939337 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932340    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 13:44:12.939337 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932343    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 13:44:12.939337 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932346    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 13:44:12.939337 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932349    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 13:44:12.939337 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932352    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 13:44:12.939337 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932355    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 13:44:12.939337 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932358    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 13:44:12.939337 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932361    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 13:44:12.939337 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932364    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 13:44:12.939337 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932366    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 13:44:12.939337 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932369    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 13:44:12.939337 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932372    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 13:44:12.939337 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932374    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 13:44:12.939835 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932377    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 13:44:12.939835 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932381    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 13:44:12.939835 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932385    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 13:44:12.939835 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932389    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 13:44:12.939835 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932392    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 13:44:12.939835 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932395    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 13:44:12.939835 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932398    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 13:44:12.939835 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932402    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 13:44:12.939835 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932404    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 13:44:12.939835 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932407    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 13:44:12.939835 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932410    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 13:44:12.939835 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932413    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 13:44:12.939835 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932416    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 13:44:12.939835 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932418    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 13:44:12.939835 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932421    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 13:44:12.939835 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932424    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 13:44:12.939835 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932426    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 13:44:12.939835 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932429    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 13:44:12.939835 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932432    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 13:44:12.939835 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932435    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 13:44:12.940386 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932438    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 13:44:12.940386 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932441    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 13:44:12.940386 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932443    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 13:44:12.940386 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932446    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 13:44:12.940386 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932449    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 13:44:12.940386 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932452    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 13:44:12.940386 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932455    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 13:44:12.940386 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932457    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 13:44:12.940386 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932461    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 13:44:12.940386 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932465    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 13:44:12.940386 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932468    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 13:44:12.940386 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932472    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 13:44:12.940386 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932476    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 13:44:12.940386 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932479    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 13:44:12.940386 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932482    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 13:44:12.940386 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932486    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 13:44:12.940386 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932489    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 13:44:12.940386 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932492    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 13:44:12.940386 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932495    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 13:44:12.940882 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932498    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 13:44:12.940882 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932501    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 13:44:12.940882 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932504    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 13:44:12.940882 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932507    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 13:44:12.940882 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932509    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 13:44:12.940882 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932512    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 13:44:12.940882 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932515    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 13:44:12.940882 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932518    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 13:44:12.940882 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932520    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 13:44:12.940882 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932523    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 13:44:12.940882 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932525    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 13:44:12.940882 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932528    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 13:44:12.940882 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932531    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 13:44:12.940882 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932533    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 13:44:12.940882 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932536    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 13:44:12.940882 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932539    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 13:44:12.940882 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932542    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 13:44:12.940882 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932544    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 13:44:12.940882 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932547    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 13:44:12.940882 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932550    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 13:44:12.941571 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932552    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 13:44:12.941571 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932555    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 13:44:12.941571 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932558    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 13:44:12.941571 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932560    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 13:44:12.941571 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932563    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 13:44:12.941571 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932566    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 13:44:12.941571 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932569    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 13:44:12.941571 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932571    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 13:44:12.941571 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932574    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 13:44:12.941571 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932577    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 13:44:12.941571 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932580    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 13:44:12.941571 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932583    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 13:44:12.941571 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932586    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 13:44:12.941571 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932588    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 13:44:12.941571 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932591    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 13:44:12.941571 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932594    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 13:44:12.941571 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932597    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 13:44:12.941571 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932599    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 13:44:12.941571 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:12.932602    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 13:44:12.942439 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.932607    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 13:44:12.942439 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.932719    2577 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 18 13:44:12.942439 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.934814    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 18 13:44:12.942439 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.935841    2577 server.go:1019] "Starting client certificate rotation"
Sep 18 13:44:12.942439 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.935948    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 13:44:12.942439 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.935992    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 13:44:12.942439 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.938403    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 13:44:12.942439 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.939659    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 13:44:12.954534 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.954511    2577 log.go:25] "Validated CRI v1 runtime API"
Sep 18 13:44:12.958478 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.958461    2577 log.go:25] "Validated CRI v1 image API"
Sep 18 13:44:12.959451 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.959422    2577 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 18 13:44:12.962246 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.962226    2577 fs.go:135] Filesystem UUIDs: map[13453677-74ec-425b-ad10-a4d544ba7100:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 bcd31e8c-344e-4121-8ecd-bb21115f9bc6:/dev/nvme0n1p4]
Sep 18 13:44:12.962313 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.962246    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 18 13:44:12.967590 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.967473    2577 manager.go:216] Machine: {Timestamp:2026-09-18 13:44:12.966963102 +0000 UTC m=+2.019622423 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099749 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2157ef22312685b171fb54019485e5 SystemUUID:ec2157ef-2231-2685-b171-fb54019485e5 BootID:9e35b3b8-620a-4b07-96fb-995dff190050 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:de:fb:86:97 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:de:fb:86:97 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:f6:25:14:b4:ce:ba Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 18 13:44:12.967590 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.967585    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 18 13:44:12.967710 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.967698    2577 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 18 13:44:12.968274 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.968259    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 13:44:12.969400 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.969372    2577 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 18 13:44:12.969584 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.969404    2577 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-222.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 18 13:44:12.970961 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.970950    2577 topology_manager.go:138] "Creating topology manager with none policy"
Sep 18 13:44:12.971003 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.970965    2577 container_manager_linux.go:309] "Creating device plugin manager"
Sep 18 13:44:12.971003 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.970973    2577 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 13:44:12.971003 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.970987    2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 13:44:12.971091 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.971063    2577 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 18 13:44:12.975433 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.975421    2577 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 13:44:12.975560 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.975550    2577 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 18 13:44:12.975596 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.975585    2577 kubelet.go:486] "Attempting to sync node with API server"
Sep 18 13:44:12.975596 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.975595    2577 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 18 13:44:12.975665 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.975607    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 18 13:44:12.975665 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.975620    2577 kubelet.go:398] "Adding apiserver pod source"
Sep 18 13:44:12.975665 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.975632    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 18 13:44:12.976054 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.976040    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 13:44:12.976100 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.976060    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 13:44:12.982895 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.982873    2577 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 18 13:44:12.985486 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.985470    2577 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 18 13:44:12.985542 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.985505    2577 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 18 13:44:12.987004 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.986985    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 18 13:44:12.987050 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.987023    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 18 13:44:12.987050 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.987037    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 18 13:44:12.987109 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.987050    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 18 13:44:12.987109 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.987072    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 18 13:44:12.987109 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.987087    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 18 13:44:12.987192 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.987114    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 18 13:44:12.987192 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.987121    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 18 13:44:12.987192 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.987128    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 18 13:44:12.987192 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.987134    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 18 13:44:12.987192 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.987150    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 18 13:44:12.987192 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.987156    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 18 13:44:12.987192 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.987190    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 18 13:44:12.987389 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.987200    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 18 13:44:12.991010 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.990977    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-222.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 13:44:12.991449 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.991433    2577 server.go:1268] "Started kubelet"
Sep 18 13:44:12.991673 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:12.991632    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 18 13:44:12.991742 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:12.991632    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-222.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 18 13:44:12.991844 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.991673    2577 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 18 13:44:12.991969 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.991773    2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 18 13:44:12.992090 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.992003    2577 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 18 13:44:12.993358 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:12.993333    2577 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 18 13:44:12.993381 ip-10-0-141-222 systemd[1]: Started Kubernetes Kubelet.
Sep 18 13:44:13.024664 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.024645    2577 server.go:310] "Adding debug handlers to kubelet server"
Sep 18 13:44:13.025883 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:13.025828    2577 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 18 13:44:13.029879 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:13.029058    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-222.ec2.internal.18d66e2fae32293d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-222.ec2.internal,UID:ip-10-0-141-222.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-222.ec2.internal,},FirstTimestamp:2026-09-18 13:44:12.991392061 +0000 UTC m=+2.044051383,LastTimestamp:2026-09-18 13:44:12.991392061 +0000 UTC m=+2.044051383,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-222.ec2.internal,}"
Sep 18 13:44:13.030986 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.030966    2577 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 18 13:44:13.030986 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.030978    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 18 13:44:13.031149 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:13.031049    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-222.ec2.internal\" not found"
Sep 18 13:44:13.031760 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.031660    2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 18 13:44:13.031760 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.031662    2577 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 18 13:44:13.031760 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.031700    2577 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 18 13:44:13.031965 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.031773    2577 reconstruct.go:97] "Volume reconstruction finished"
Sep 18 13:44:13.031965 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.031782    2577 reconciler.go:29] "Reconciler: start to sync state"
Sep 18 13:44:13.031965 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.031902    2577 factory.go:55] Registering systemd factory
Sep 18 13:44:13.031965 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.031921    2577 factory.go:223] Registration of the systemd container factory successfully
Sep 18 13:44:13.032220 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.032192    2577 factory.go:153] Registering CRI-O factory
Sep 18 13:44:13.032220 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.032213    2577 factory.go:223] Registration of the crio container factory successfully
Sep 18 13:44:13.032321 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.032267    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 18 13:44:13.032321 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.032287    2577 factory.go:103] Registering Raw factory
Sep 18 13:44:13.032321 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.032300    2577 manager.go:1196] Started watching for new ooms in manager
Sep 18 13:44:13.032817 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.032804    2577 manager.go:349] Starting recovery of all containers
Sep 18 13:44:13.034580 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.034555    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2ddsn"
Sep 18 13:44:13.042030 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:13.041993    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 18 13:44:13.042139 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:13.042090    2577 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-222.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 18 13:44:13.042623 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.042605    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2ddsn"
Sep 18 13:44:13.044767 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.044749    2577 manager.go:354] Recovery completed
Sep 18 13:44:13.049794 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.049779    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 13:44:13.051917 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.051899    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-222.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 13:44:13.051979 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.051935    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-222.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 13:44:13.051979 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.051952    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-222.ec2.internal" event="NodeHasSufficientPID"
Sep 18 13:44:13.052495 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.052478    2577 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 18 13:44:13.052495 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.052494    2577 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 18 13:44:13.052597 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.052512    2577 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 13:44:13.052597 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.052497    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 13:44:13.053298 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.053283    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-222.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 13:44:13.053360 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.053310    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-222.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 13:44:13.053360 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.053320    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-222.ec2.internal" event="NodeHasSufficientPID"
Sep 18 13:44:13.054557 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:13.054483    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-222.ec2.internal.18d66e2fb1cdb4e5  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-222.ec2.internal,UID:ip-10-0-141-222.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-222.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-222.ec2.internal,},FirstTimestamp:2026-09-18 13:44:13.051917541 +0000 UTC m=+2.104576861,LastTimestamp:2026-09-18 13:44:13.051917541 +0000 UTC m=+2.104576861,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-222.ec2.internal,}"
Sep 18 13:44:13.054715 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.054702    2577 policy_none.go:49] "None policy: Start"
Sep 18 13:44:13.054757 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.054719    2577 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 18 13:44:13.054757 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.054730    2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 18 13:44:13.056844 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.056830    2577 policy_none.go:47] "Start"
Sep 18 13:44:13.093774 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.093750    2577 manager.go:337] "Starting Device Plugin manager"
Sep 18 13:44:13.105035 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:13.093819    2577 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 18 13:44:13.105035 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.093844    2577 server.go:87] "Starting device plugin registration server"
Sep 18 13:44:13.105035 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.094212    2577 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 18 13:44:13.105035 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.094237    2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 18 13:44:13.105035 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.094444    2577 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 18 13:44:13.105035 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.095138    2577 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 18 13:44:13.105035 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.095153    2577 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 18 13:44:13.105035 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:13.095591    2577 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 18 13:44:13.105035 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:13.095650    2577 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-222.ec2.internal\" not found"
Sep 18 13:44:13.105035 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.099576    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 18 13:44:13.105035 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.100752    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 18 13:44:13.105035 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.100798    2577 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 18 13:44:13.105035 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.100825    2577 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 18 13:44:13.105035 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:13.100862    2577 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 18 13:44:13.105035 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.104124    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 13:44:13.195301 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.195174    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 13:44:13.197106 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.197085    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-222.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 13:44:13.197217 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.197122    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-222.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 13:44:13.197217 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.197135    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-222.ec2.internal" event="NodeHasSufficientPID"
Sep 18 13:44:13.197217 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.197194    2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-222.ec2.internal"
Sep 18 13:44:13.201849 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.201814    2577 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-141-222.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-222.ec2.internal"]
Sep 18 13:44:13.201911 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.201900    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 13:44:13.202767 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.202751    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-222.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 13:44:13.202843 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.202779    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-222.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 13:44:13.202843 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.202789    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-222.ec2.internal" event="NodeHasSufficientPID"
Sep 18 13:44:13.205200 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.205187    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 13:44:13.205372 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.205359    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-222.ec2.internal"
Sep 18 13:44:13.205410 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.205394    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 13:44:13.207133 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.207108    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-222.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 13:44:13.207289 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.207145    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-222.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 13:44:13.207289 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.207162    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-222.ec2.internal" event="NodeHasSufficientPID"
Sep 18 13:44:13.207752 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.207468    2577 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-222.ec2.internal"
Sep 18 13:44:13.207752 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:13.207498    2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-222.ec2.internal\": node \"ip-10-0-141-222.ec2.internal\" not found"
Sep 18 13:44:13.207752 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.207504    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-222.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 13:44:13.207752 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.207523    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-222.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 13:44:13.207752 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.207599    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-222.ec2.internal" event="NodeHasSufficientPID"
Sep 18 13:44:13.210393 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.210376    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-222.ec2.internal"
Sep 18 13:44:13.210472 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.210408    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 13:44:13.211859 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.211843    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-222.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 13:44:13.211977 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.211871    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-222.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 13:44:13.211977 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.211886    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-222.ec2.internal" event="NodeHasSufficientPID"
Sep 18 13:44:13.230113 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:13.230092    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-222.ec2.internal\" not found"
Sep 18 13:44:13.236649 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:13.236634    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-222.ec2.internal\" not found" node="ip-10-0-141-222.ec2.internal"
Sep 18 13:44:13.241640 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:13.241625    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-222.ec2.internal\" not found" node="ip-10-0-141-222.ec2.internal"
Sep 18 13:44:13.330291 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:13.330257    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-222.ec2.internal\" not found"
Sep 18 13:44:13.332681 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.332661    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00bbb25ba7cee607198e2c37fb5d2b3f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-222.ec2.internal\" (UID: \"00bbb25ba7cee607198e2c37fb5d2b3f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-222.ec2.internal"
Sep 18 13:44:13.332782 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.332695    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4a4b0c9f3c3ac36b289317453f80d9d6-config\") pod \"kube-apiserver-proxy-ip-10-0-141-222.ec2.internal\" (UID: \"4a4b0c9f3c3ac36b289317453f80d9d6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-222.ec2.internal"
Sep 18 13:44:13.332782 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.332758    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/00bbb25ba7cee607198e2c37fb5d2b3f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-222.ec2.internal\" (UID: \"00bbb25ba7cee607198e2c37fb5d2b3f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-222.ec2.internal"
Sep 18 13:44:13.431786 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:13.431749    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-222.ec2.internal\" not found"
Sep 18 13:44:13.432988 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.432961    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4a4b0c9f3c3ac36b289317453f80d9d6-config\") pod \"kube-apiserver-proxy-ip-10-0-141-222.ec2.internal\" (UID: \"4a4b0c9f3c3ac36b289317453f80d9d6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-222.ec2.internal"
Sep 18 13:44:13.432988 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.432991    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/00bbb25ba7cee607198e2c37fb5d2b3f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-222.ec2.internal\" (UID: \"00bbb25ba7cee607198e2c37fb5d2b3f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-222.ec2.internal"
Sep 18 13:44:13.433197 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.433011    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00bbb25ba7cee607198e2c37fb5d2b3f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-222.ec2.internal\" (UID: \"00bbb25ba7cee607198e2c37fb5d2b3f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-222.ec2.internal"
Sep 18 13:44:13.433197 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.433061    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00bbb25ba7cee607198e2c37fb5d2b3f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-222.ec2.internal\" (UID: \"00bbb25ba7cee607198e2c37fb5d2b3f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-222.ec2.internal"
Sep 18 13:44:13.433197 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.433086    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/00bbb25ba7cee607198e2c37fb5d2b3f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-222.ec2.internal\" (UID: \"00bbb25ba7cee607198e2c37fb5d2b3f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-222.ec2.internal"
Sep 18 13:44:13.433197 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.433084    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4a4b0c9f3c3ac36b289317453f80d9d6-config\") pod \"kube-apiserver-proxy-ip-10-0-141-222.ec2.internal\" (UID: \"4a4b0c9f3c3ac36b289317453f80d9d6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-222.ec2.internal"
Sep 18 13:44:13.535489 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:13.535399    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-222.ec2.internal\" not found"
Sep 18 13:44:13.540999 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.540984    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-222.ec2.internal"
Sep 18 13:44:13.545437 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.545420    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-222.ec2.internal"
Sep 18 13:44:13.636378 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:13.636343    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-222.ec2.internal\" not found"
Sep 18 13:44:13.736880 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:13.736852    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-222.ec2.internal\" not found"
Sep 18 13:44:13.837957 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:13.837899    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-222.ec2.internal\" not found"
Sep 18 13:44:13.935298 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.935270    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 18 13:44:13.935726 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.935435    2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 13:44:13.938684 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:13.938659    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-222.ec2.internal\" not found"
Sep 18 13:44:13.950335 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.950316    2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 13:44:13.955874 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.955856    2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 13:44:13.976664 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.976632    2577 apiserver.go:52] "Watching apiserver"
Sep 18 13:44:13.983336 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.983314    2577 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 18 13:44:13.983695 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.983675    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-ztfrw","openshift-network-diagnostics/network-check-target-hj8lg","openshift-ovn-kubernetes/ovnkube-node-wbv96","kube-system/global-pull-secret-syncer-bdwmn","kube-system/konnectivity-agent-7cm48","openshift-dns/node-resolver-vjv4c","openshift-multus/network-metrics-daemon-nb9bg","openshift-network-operator/iptables-alerter-vc9jp","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nscch","openshift-cluster-node-tuning-operator/tuned-8k94g","openshift-image-registry/node-ca-wczf6","openshift-multus/multus-additional-cni-plugins-szg9k"]
Sep 18 13:44:13.986438 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.986420    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ztfrw"
Sep 18 13:44:13.988407 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.988378    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 18 13:44:13.988526 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.988380    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 18 13:44:13.988526 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.988379    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 18 13:44:13.988526 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.988456    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 18 13:44:13.988674 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.988619    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-jc4bs\""
Sep 18 13:44:13.988735 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.988720    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hj8lg"
Sep 18 13:44:13.988812 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:13.988794    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hj8lg" podUID="6f382898-0370-425b-8c01-6587a985e3f0"
Sep 18 13:44:13.991350 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.991332    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wbv96"
Sep 18 13:44:13.993080 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.993063    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 18 13:44:13.993473 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.993457    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-zzjhx\""
Sep 18 13:44:13.993615 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.993597    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 18 13:44:13.993700 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.993678    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 18 13:44:13.993760 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.993707    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 18 13:44:13.993760 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.993723    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 18 13:44:13.993760 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.993730    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 18 13:44:13.993987 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.993971    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdwmn"
Sep 18 13:44:13.994055 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:13.994035    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdwmn" podUID="802e0a84-ff96-44d6-9aa2-db34555f46eb"
Sep 18 13:44:13.996312 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.996293    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7cm48"
Sep 18 13:44:13.998340 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.998321    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 18 13:44:13.998408 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.998336    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-4wv82\""
Sep 18 13:44:13.998446 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.998414    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 18 13:44:13.998561 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:13.998548    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vjv4c"
Sep 18 13:44:14.001689 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.001665    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xl4df\""
Sep 18 13:44:14.001773 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.001687    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 18 13:44:14.001773 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.001733    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 18 13:44:14.001905 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.001872    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nb9bg"
Sep 18 13:44:14.001983 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:14.001956    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nb9bg" podUID="87eba103-cfec-4000-a10c-fd4fd512ac90"
Sep 18 13:44:14.004465 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.004446    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vc9jp"
Sep 18 13:44:14.006517 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.006501    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-flpdq\""
Sep 18 13:44:14.006638 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.006618    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 18 13:44:14.006683 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.006623    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 18 13:44:14.007050 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.007033    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nscch"
Sep 18 13:44:14.009009 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.008993    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 18 13:44:14.009102 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.009020    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 18 13:44:14.009102 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.009076    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 18 13:44:14.009286 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.009268    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 18 13:44:14.009354 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.009268    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-swhqc\""
Sep 18 13:44:14.009502 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.009486    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8k94g"
Sep 18 13:44:14.011439 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.011418    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 18 13:44:14.011550 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.011534    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-pj9t4\""
Sep 18 13:44:14.011786 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.011768    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 18 13:44:14.011856 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.011815    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wczf6"
Sep 18 13:44:14.014068 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.014041    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 18 13:44:14.014152 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.014116    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 18 13:44:14.014258 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.014244    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hwjhn\""
Sep 18 13:44:14.014294 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.014275    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 18 13:44:14.014395 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.014379    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-szg9k"
Sep 18 13:44:14.016222 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.016208    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 18 13:44:14.016587 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.016570    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 18 13:44:14.016661 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.016609    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hhjxb\""
Sep 18 13:44:14.031923 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.031898    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-222.ec2.internal"
Sep 18 13:44:14.032014 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.031934    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 18 13:44:14.032700 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.032686    2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 18 13:44:14.038797 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.038780    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b4829a7a-35e0-44e4-b738-4ba4bd403929-etc-kubernetes\") pod \"multus-ztfrw\" (UID: \"b4829a7a-35e0-44e4-b738-4ba4bd403929\") " pod="openshift-multus/multus-ztfrw"
Sep 18 13:44:14.038875 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.038806    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdv8r\" (UniqueName: \"kubernetes.io/projected/f5d1bf14-b322-49d0-9fc8-b3a9a0aca036-kube-api-access-kdv8r\") pod \"ovnkube-node-wbv96\" (UID: \"f5d1bf14-b322-49d0-9fc8-b3a9a0aca036\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbv96"
Sep 18 13:44:14.038875 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.038847    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e0b6f508-1744-4ab7-8e37-0f2f4df3a816-etc-systemd\") pod \"tuned-8k94g\" (UID: \"e0b6f508-1744-4ab7-8e37-0f2f4df3a816\") " pod="openshift-cluster-node-tuning-operator/tuned-8k94g"
Sep 18 13:44:14.039007 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.038872    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrxhq\" (UniqueName: \"kubernetes.io/projected/e0b6f508-1744-4ab7-8e37-0f2f4df3a816-kube-api-access-lrxhq\") pod \"tuned-8k94g\" (UID: \"e0b6f508-1744-4ab7-8e37-0f2f4df3a816\") " pod="openshift-cluster-node-tuning-operator/tuned-8k94g"
Sep 18 13:44:14.039007 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.038893    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbd84791-daa9-43f0-9f32-8f4415eceeca-tmp-dir\") pod \"node-resolver-vjv4c\" (UID: \"dbd84791-daa9-43f0-9f32-8f4415eceeca\") " pod="openshift-dns/node-resolver-vjv4c"
Sep 18 13:44:14.039007 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.038928    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b4829a7a-35e0-44e4-b738-4ba4bd403929-host-run-k8s-cni-cncf-io\") pod \"multus-ztfrw\" (UID: \"b4829a7a-35e0-44e4-b738-4ba4bd403929\") " pod="openshift-multus/multus-ztfrw"
Sep 18 13:44:14.039007 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.038950    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f5d1bf14-b322-49d0-9fc8-b3a9a0aca036-host-cni-bin\") pod \"ovnkube-node-wbv96\" (UID: \"f5d1bf14-b322-49d0-9fc8-b3a9a0aca036\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbv96"
Sep 18 13:44:14.039007 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.038985    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/66b62a87-76ae-41e9-8e0c-dd48ff323e5d-etc-selinux\") pod \"aws-ebs-csi-driver-node-nscch\" (UID: \"66b62a87-76ae-41e9-8e0c-dd48ff323e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nscch"
Sep 18 13:44:14.039218 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.039077    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ac788d27-88a8-4ed2-9363-fe4f0aa51013-serviceca\") pod \"node-ca-wczf6\" (UID: \"ac788d27-88a8-4ed2-9363-fe4f0aa51013\") " pod="openshift-image-registry/node-ca-wczf6"
Sep 18 13:44:14.039218 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.039121    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f5d1bf14-b322-49d0-9fc8-b3a9a0aca036-ovn-node-metrics-cert\") pod \"ovnkube-node-wbv96\" (UID: \"f5d1bf14-b322-49d0-9fc8-b3a9a0aca036\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbv96"
Sep 18 13:44:14.039218 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.039150    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8902efdf-cf1d-4513-bfbb-cdf2b9a2b922-konnectivity-ca\") pod \"konnectivity-agent-7cm48\" (UID: \"8902efdf-cf1d-4513-bfbb-cdf2b9a2b922\") " pod="kube-system/konnectivity-agent-7cm48"
Sep 18 13:44:14.039218 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.039175    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e0b6f508-1744-4ab7-8e37-0f2f4df3a816-run\") pod \"tuned-8k94g\" (UID: \"e0b6f508-1744-4ab7-8e37-0f2f4df3a816\") " pod="openshift-cluster-node-tuning-operator/tuned-8k94g"
Sep 18 13:44:14.039218 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.039200    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87eba103-cfec-4000-a10c-fd4fd512ac90-metrics-certs\") pod \"network-metrics-daemon-nb9bg\" (UID: \"87eba103-cfec-4000-a10c-fd4fd512ac90\") " pod="openshift-multus/network-metrics-daemon-nb9bg"
Sep 18 13:44:14.039451 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.039241    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnjtp\" (UniqueName: \"kubernetes.io/projected/87eba103-cfec-4000-a10c-fd4fd512ac90-kube-api-access-xnjtp\") pod \"network-metrics-daemon-nb9bg\" (UID: \"87eba103-cfec-4000-a10c-fd4fd512ac90\") " pod="openshift-multus/network-metrics-daemon-nb9bg"
Sep 18 13:44:14.039451 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.039268    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2e756a4b-a7f8-4688-9a3a-efecba8641d2-iptables-alerter-script\") pod \"iptables-alerter-vc9jp\" (UID: \"2e756a4b-a7f8-4688-9a3a-efecba8641d2\") " pod="openshift-network-operator/iptables-alerter-vc9jp"
Sep 18 13:44:14.039451 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.039294    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b4829a7a-35e0-44e4-b738-4ba4bd403929-host-var-lib-cni-multus\") pod \"multus-ztfrw\" (UID: \"b4829a7a-35e0-44e4-b738-4ba4bd403929\") " pod="openshift-multus/multus-ztfrw"
Sep 18 13:44:14.039451 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.039318    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f5d1bf14-b322-49d0-9fc8-b3a9a0aca036-host-kubelet\") pod \"ovnkube-node-wbv96\" (UID: \"f5d1bf14-b322-49d0-9fc8-b3a9a0aca036\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbv96"
Sep 18 13:44:14.039451 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.039344    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f5d1bf14-b322-49d0-9fc8-b3a9a0aca036-host-run-ovn-kubernetes\") pod \"ovnkube-node-wbv96\" (UID: \"f5d1bf14-b322-49d0-9fc8-b3a9a0aca036\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbv96"
Sep 18 13:44:14.039451 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.039391    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f5d1bf14-b322-49d0-9fc8-b3a9a0aca036-ovnkube-config\") pod \"ovnkube-node-wbv96\" (UID: \"f5d1bf14-b322-49d0-9fc8-b3a9a0aca036\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbv96"
Sep 18 13:44:14.039451 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.039427    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/802e0a84-ff96-44d6-9aa2-db34555f46eb-dbus\") pod \"global-pull-secret-syncer-bdwmn\" (UID: \"802e0a84-ff96-44d6-9aa2-db34555f46eb\") " pod="kube-system/global-pull-secret-syncer-bdwmn"
Sep 18 13:44:14.039451 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.039453    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b4829a7a-35e0-44e4-b738-4ba4bd403929-multus-cni-dir\") pod \"multus-ztfrw\" (UID: \"b4829a7a-35e0-44e4-b738-4ba4bd403929\") " pod="openshift-multus/multus-ztfrw"
Sep 18 13:44:14.039858 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.039482    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f5d1bf14-b322-49d0-9fc8-b3a9a0aca036-run-systemd\") pod \"ovnkube-node-wbv96\" (UID: \"f5d1bf14-b322-49d0-9fc8-b3a9a0aca036\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbv96"
Sep 18 13:44:14.039858 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.039508    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f5d1bf14-b322-49d0-9fc8-b3a9a0aca036-ovnkube-script-lib\") pod \"ovnkube-node-wbv96\" (UID: \"f5d1bf14-b322-49d0-9fc8-b3a9a0aca036\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbv96"
Sep 18 13:44:14.039858 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.039533    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/66b62a87-76ae-41e9-8e0c-dd48ff323e5d-device-dir\") pod \"aws-ebs-csi-driver-node-nscch\" (UID: \"66b62a87-76ae-41e9-8e0c-dd48ff323e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nscch"
Sep 18 13:44:14.039858 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.039560    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/66b62a87-76ae-41e9-8e0c-dd48ff323e5d-sys-fs\") pod \"aws-ebs-csi-driver-node-nscch\" (UID: \"66b62a87-76ae-41e9-8e0c-dd48ff323e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nscch"
Sep 18 13:44:14.039858 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.039585    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ac788d27-88a8-4ed2-9363-fe4f0aa51013-host\") pod \"node-ca-wczf6\" (UID: \"ac788d27-88a8-4ed2-9363-fe4f0aa51013\") " pod="openshift-image-registry/node-ca-wczf6"
Sep 18 13:44:14.039858 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.039609    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/47923c57-2f7a-46f5-b8c4-10efd987fa8b-system-cni-dir\") pod \"multus-additional-cni-plugins-szg9k\" (UID: \"47923c57-2f7a-46f5-b8c4-10efd987fa8b\") " pod="openshift-multus/multus-additional-cni-plugins-szg9k"
Sep 18 13:44:14.039858 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.039635    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/47923c57-2f7a-46f5-b8c4-10efd987fa8b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-szg9k\" (UID: \"47923c57-2f7a-46f5-b8c4-10efd987fa8b\") " pod="openshift-multus/multus-additional-cni-plugins-szg9k"
Sep 18 13:44:14.039858 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.039661    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f5d1bf14-b322-49d0-9fc8-b3a9a0aca036-var-lib-openvswitch\") pod \"ovnkube-node-wbv96\" (UID: \"f5d1bf14-b322-49d0-9fc8-b3a9a0aca036\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbv96"
Sep 18 13:44:14.039858 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.039687    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f5d1bf14-b322-49d0-9fc8-b3a9a0aca036-env-overrides\") pod \"ovnkube-node-wbv96\" (UID: \"f5d1bf14-b322-49d0-9fc8-b3a9a0aca036\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbv96"
Sep 18 13:44:14.039858 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.039734    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cm5q\" (UniqueName: \"kubernetes.io/projected/2e756a4b-a7f8-4688-9a3a-efecba8641d2-kube-api-access-6cm5q\") pod \"iptables-alerter-vc9jp\" (UID: \"2e756a4b-a7f8-4688-9a3a-efecba8641d2\") " pod="openshift-network-operator/iptables-alerter-vc9jp"
Sep 18 13:44:14.039858 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.039771    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/47923c57-2f7a-46f5-b8c4-10efd987fa8b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-szg9k\" (UID: \"47923c57-2f7a-46f5-b8c4-10efd987fa8b\") " pod="openshift-multus/multus-additional-cni-plugins-szg9k"
Sep 18 13:44:14.039858 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.039810    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b4829a7a-35e0-44e4-b738-4ba4bd403929-system-cni-dir\") pod \"multus-ztfrw\" (UID: \"b4829a7a-35e0-44e4-b738-4ba4bd403929\") " pod="openshift-multus/multus-ztfrw"
Sep 18 13:44:14.039858 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.039835    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b4829a7a-35e0-44e4-b738-4ba4bd403929-cnibin\") pod \"multus-ztfrw\" (UID: \"b4829a7a-35e0-44e4-b738-4ba4bd403929\") " pod="openshift-multus/multus-ztfrw"
Sep 18 13:44:14.040390 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.039892    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e0b6f508-1744-4ab7-8e37-0f2f4df3a816-etc-sysconfig\") pod \"tuned-8k94g\" (UID: \"e0b6f508-1744-4ab7-8e37-0f2f4df3a816\") " pod="openshift-cluster-node-tuning-operator/tuned-8k94g"
Sep 18 13:44:14.040390 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.039933    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/66b62a87-76ae-41e9-8e0c-dd48ff323e5d-socket-dir\") pod \"aws-ebs-csi-driver-node-nscch\" (UID: \"66b62a87-76ae-41e9-8e0c-dd48ff323e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nscch"
Sep 18 13:44:14.040390 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.039991    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b4zq\" (UniqueName: \"kubernetes.io/projected/ac788d27-88a8-4ed2-9363-fe4f0aa51013-kube-api-access-9b4zq\") pod \"node-ca-wczf6\" (UID: \"ac788d27-88a8-4ed2-9363-fe4f0aa51013\") " pod="openshift-image-registry/node-ca-wczf6"
Sep 18 13:44:14.040390 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.040033    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b4829a7a-35e0-44e4-b738-4ba4bd403929-cni-binary-copy\") pod \"multus-ztfrw\" (UID: \"b4829a7a-35e0-44e4-b738-4ba4bd403929\") " pod="openshift-multus/multus-ztfrw"
Sep 18 13:44:14.040390 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.040064    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b4829a7a-35e0-44e4-b738-4ba4bd403929-host-run-netns\") pod \"multus-ztfrw\" (UID: \"b4829a7a-35e0-44e4-b738-4ba4bd403929\") " pod="openshift-multus/multus-ztfrw"
Sep 18 13:44:14.040390 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.040089    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f5d1bf14-b322-49d0-9fc8-b3a9a0aca036-systemd-units\") pod \"ovnkube-node-wbv96\" (UID: \"f5d1bf14-b322-49d0-9fc8-b3a9a0aca036\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbv96"
Sep 18 13:44:14.040390 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.040123    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/802e0a84-ff96-44d6-9aa2-db34555f46eb-original-pull-secret\") pod \"global-pull-secret-syncer-bdwmn\" (UID: \"802e0a84-ff96-44d6-9aa2-db34555f46eb\") " pod="kube-system/global-pull-secret-syncer-bdwmn"
Sep 18 13:44:14.040390 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.040139    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0b6f508-1744-4ab7-8e37-0f2f4df3a816-etc-kubernetes\") pod \"tuned-8k94g\" (UID: \"e0b6f508-1744-4ab7-8e37-0f2f4df3a816\") " pod="openshift-cluster-node-tuning-operator/tuned-8k94g"
Sep 18 13:44:14.040390 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.040157    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e0b6f508-1744-4ab7-8e37-0f2f4df3a816-host\") pod \"tuned-8k94g\" (UID: \"e0b6f508-1744-4ab7-8e37-0f2f4df3a816\") " pod="openshift-cluster-node-tuning-operator/tuned-8k94g"
Sep 18 13:44:14.040390 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.040189    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/66b62a87-76ae-41e9-8e0c-dd48ff323e5d-registration-dir\") pod \"aws-ebs-csi-driver-node-nscch\" (UID: \"66b62a87-76ae-41e9-8e0c-dd48ff323e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nscch"
Sep 18 13:44:14.040390 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.040209    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b4829a7a-35e0-44e4-b738-4ba4bd403929-multus-daemon-config\") pod \"multus-ztfrw\" (UID: \"b4829a7a-35e0-44e4-b738-4ba4bd403929\") " pod="openshift-multus/multus-ztfrw"
Sep 18 13:44:14.040390 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.040222    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e0b6f508-1744-4ab7-8e37-0f2f4df3a816-etc-sysctl-d\") pod \"tuned-8k94g\" (UID: \"e0b6f508-1744-4ab7-8e37-0f2f4df3a816\") " pod="openshift-cluster-node-tuning-operator/tuned-8k94g"
Sep 18 13:44:14.040390 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.040236    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2e756a4b-a7f8-4688-9a3a-efecba8641d2-host-slash\") pod \"iptables-alerter-vc9jp\" (UID: \"2e756a4b-a7f8-4688-9a3a-efecba8641d2\") " pod="openshift-network-operator/iptables-alerter-vc9jp"
Sep 18 13:44:14.040390 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.040260    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wqns\" (UniqueName: \"kubernetes.io/projected/66b62a87-76ae-41e9-8e0c-dd48ff323e5d-kube-api-access-5wqns\") pod \"aws-ebs-csi-driver-node-nscch\" (UID: \"66b62a87-76ae-41e9-8e0c-dd48ff323e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nscch"
Sep 18 13:44:14.040390 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.040304    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/47923c57-2f7a-46f5-b8c4-10efd987fa8b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-szg9k\" (UID: \"47923c57-2f7a-46f5-b8c4-10efd987fa8b\") " pod="openshift-multus/multus-additional-cni-plugins-szg9k"
Sep 18 13:44:14.040390 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.040325    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtkdz\" (UniqueName: \"kubernetes.io/projected/47923c57-2f7a-46f5-b8c4-10efd987fa8b-kube-api-access-mtkdz\") pod \"multus-additional-cni-plugins-szg9k\" (UID: \"47923c57-2f7a-46f5-b8c4-10efd987fa8b\") " pod="openshift-multus/multus-additional-cni-plugins-szg9k"
Sep 18 13:44:14.041152 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.040341    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b4829a7a-35e0-44e4-b738-4ba4bd403929-host-var-lib-cni-bin\") pod \"multus-ztfrw\" (UID: \"b4829a7a-35e0-44e4-b738-4ba4bd403929\") " pod="openshift-multus/multus-ztfrw"
Sep 18 13:44:14.041152 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.040356    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f5d1bf14-b322-49d0-9fc8-b3a9a0aca036-run-ovn\") pod \"ovnkube-node-wbv96\" (UID: \"f5d1bf14-b322-49d0-9fc8-b3a9a0aca036\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbv96"
Sep 18 13:44:14.041152 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.040372    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f5d1bf14-b322-49d0-9fc8-b3a9a0aca036-host-cni-netd\") pod \"ovnkube-node-wbv96\" (UID: \"f5d1bf14-b322-49d0-9fc8-b3a9a0aca036\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbv96"
Sep 18 13:44:14.041152 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.040387    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f5d1bf14-b322-49d0-9fc8-b3a9a0aca036-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wbv96\" (UID: \"f5d1bf14-b322-49d0-9fc8-b3a9a0aca036\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbv96"
Sep 18 13:44:14.041152 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.040412    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e0b6f508-1744-4ab7-8e37-0f2f4df3a816-etc-modprobe-d\") pod \"tuned-8k94g\" (UID: \"e0b6f508-1744-4ab7-8e37-0f2f4df3a816\") " pod="openshift-cluster-node-tuning-operator/tuned-8k94g"
Sep 18 13:44:14.041152 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.040456    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b4829a7a-35e0-44e4-b738-4ba4bd403929-multus-conf-dir\") pod \"multus-ztfrw\" (UID: \"b4829a7a-35e0-44e4-b738-4ba4bd403929\") " pod="openshift-multus/multus-ztfrw"
Sep 18 13:44:14.041152 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.040515    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f5d1bf14-b322-49d0-9fc8-b3a9a0aca036-host-slash\") pod \"ovnkube-node-wbv96\" (UID: \"f5d1bf14-b322-49d0-9fc8-b3a9a0aca036\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbv96"
Sep 18 13:44:14.041152 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.040543    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f5d1bf14-b322-49d0-9fc8-b3a9a0aca036-etc-openvswitch\") pod \"ovnkube-node-wbv96\" (UID: \"f5d1bf14-b322-49d0-9fc8-b3a9a0aca036\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbv96"
Sep 18 13:44:14.041152 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.040570    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f5d1bf14-b322-49d0-9fc8-b3a9a0aca036-log-socket\") pod \"ovnkube-node-wbv96\" (UID: \"f5d1bf14-b322-49d0-9fc8-b3a9a0aca036\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbv96"
Sep 18 13:44:14.041152 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.040595    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e0b6f508-1744-4ab7-8e37-0f2f4df3a816-sys\") pod \"tuned-8k94g\" (UID: \"e0b6f508-1744-4ab7-8e37-0f2f4df3a816\") " pod="openshift-cluster-node-tuning-operator/tuned-8k94g"
Sep 18 13:44:14.041152 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.040626    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmqn6\" (UniqueName: \"kubernetes.io/projected/6f382898-0370-425b-8c01-6587a985e3f0-kube-api-access-dmqn6\") pod \"network-check-target-hj8lg\" (UID: \"6f382898-0370-425b-8c01-6587a985e3f0\") " pod="openshift-network-diagnostics/network-check-target-hj8lg"
Sep 18 13:44:14.041152 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.040667    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dbd84791-daa9-43f0-9f32-8f4415eceeca-hosts-file\") pod \"node-resolver-vjv4c\" (UID: \"dbd84791-daa9-43f0-9f32-8f4415eceeca\") " pod="openshift-dns/node-resolver-vjv4c"
Sep 18 13:44:14.041152 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.040702    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f5d1bf14-b322-49d0-9fc8-b3a9a0aca036-run-openvswitch\") pod \"ovnkube-node-wbv96\" (UID: \"f5d1bf14-b322-49d0-9fc8-b3a9a0aca036\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbv96"
Sep 18 13:44:14.041152 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.040728    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b4829a7a-35e0-44e4-b738-4ba4bd403929-host-run-multus-certs\") pod \"multus-ztfrw\" (UID: \"b4829a7a-35e0-44e4-b738-4ba4bd403929\") " pod="openshift-multus/multus-ztfrw"
Sep 18 13:44:14.041152 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.040753    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f5d1bf14-b322-49d0-9fc8-b3a9a0aca036-node-log\") pod \"ovnkube-node-wbv96\" (UID: \"f5d1bf14-b322-49d0-9fc8-b3a9a0aca036\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbv96"
Sep 18 13:44:14.041152 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.040778    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e0b6f508-1744-4ab7-8e37-0f2f4df3a816-lib-modules\") pod \"tuned-8k94g\" (UID: \"e0b6f508-1744-4ab7-8e37-0f2f4df3a816\") " pod="openshift-cluster-node-tuning-operator/tuned-8k94g"
Sep 18 13:44:14.041152 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.040802    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8902efdf-cf1d-4513-bfbb-cdf2b9a2b922-agent-certs\") pod \"konnectivity-agent-7cm48\" (UID: \"8902efdf-cf1d-4513-bfbb-cdf2b9a2b922\") " pod="kube-system/konnectivity-agent-7cm48"
Sep 18 13:44:14.041778 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.040845    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/66b62a87-76ae-41e9-8e0c-dd48ff323e5d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nscch\" (UID: \"66b62a87-76ae-41e9-8e0c-dd48ff323e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nscch"
Sep 18 13:44:14.041778 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.040866    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b4829a7a-35e0-44e4-b738-4ba4bd403929-host-var-lib-kubelet\") pod \"multus-ztfrw\" (UID: \"b4829a7a-35e0-44e4-b738-4ba4bd403929\") " pod="openshift-multus/multus-ztfrw"
Sep 18 13:44:14.041778 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.040881    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b4829a7a-35e0-44e4-b738-4ba4bd403929-hostroot\") pod \"multus-ztfrw\" (UID: \"b4829a7a-35e0-44e4-b738-4ba4bd403929\") " pod="openshift-multus/multus-ztfrw"
Sep 18 13:44:14.041778 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.040896    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/802e0a84-ff96-44d6-9aa2-db34555f46eb-kubelet-config\") pod \"global-pull-secret-syncer-bdwmn\" (UID: \"802e0a84-ff96-44d6-9aa2-db34555f46eb\") " pod="kube-system/global-pull-secret-syncer-bdwmn"
Sep 18 13:44:14.041778 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.040916    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e0b6f508-1744-4ab7-8e37-0f2f4df3a816-etc-sysctl-conf\") pod \"tuned-8k94g\" (UID: \"e0b6f508-1744-4ab7-8e37-0f2f4df3a816\") " pod="openshift-cluster-node-tuning-operator/tuned-8k94g"
Sep 18 13:44:14.041778 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.040938    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e0b6f508-1744-4ab7-8e37-0f2f4df3a816-etc-tuned\") pod \"tuned-8k94g\" (UID: \"e0b6f508-1744-4ab7-8e37-0f2f4df3a816\") " pod="openshift-cluster-node-tuning-operator/tuned-8k94g"
Sep 18 13:44:14.041778 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.040956    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/47923c57-2f7a-46f5-b8c4-10efd987fa8b-cnibin\") pod \"multus-additional-cni-plugins-szg9k\" (UID: \"47923c57-2f7a-46f5-b8c4-10efd987fa8b\") " pod="openshift-multus/multus-additional-cni-plugins-szg9k"
Sep 18 13:44:14.041778 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.040970    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b4829a7a-35e0-44e4-b738-4ba4bd403929-multus-socket-dir-parent\") pod \"multus-ztfrw\" (UID: \"b4829a7a-35e0-44e4-b738-4ba4bd403929\") " pod="openshift-multus/multus-ztfrw"
Sep 18 13:44:14.041778 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.040984    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9txp\" (UniqueName: \"kubernetes.io/projected/b4829a7a-35e0-44e4-b738-4ba4bd403929-kube-api-access-s9txp\") pod \"multus-ztfrw\" (UID: \"b4829a7a-35e0-44e4-b738-4ba4bd403929\") " pod="openshift-multus/multus-ztfrw"
Sep 18 13:44:14.041778 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.041002    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f5d1bf14-b322-49d0-9fc8-b3a9a0aca036-host-run-netns\") pod \"ovnkube-node-wbv96\" (UID: \"f5d1bf14-b322-49d0-9fc8-b3a9a0aca036\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbv96"
Sep 18 13:44:14.041778 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.041026    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0b6f508-1744-4ab7-8e37-0f2f4df3a816-var-lib-kubelet\") pod \"tuned-8k94g\" (UID: \"e0b6f508-1744-4ab7-8e37-0f2f4df3a816\") " pod="openshift-cluster-node-tuning-operator/tuned-8k94g"
Sep 18 13:44:14.041778 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.041048    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0b6f508-1744-4ab7-8e37-0f2f4df3a816-tmp\") pod \"tuned-8k94g\" (UID: \"e0b6f508-1744-4ab7-8e37-0f2f4df3a816\") " pod="openshift-cluster-node-tuning-operator/tuned-8k94g"
Sep 18 13:44:14.041778 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.041075    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h4d5\" (UniqueName: \"kubernetes.io/projected/dbd84791-daa9-43f0-9f32-8f4415eceeca-kube-api-access-7h4d5\") pod \"node-resolver-vjv4c\" (UID: \"dbd84791-daa9-43f0-9f32-8f4415eceeca\") " pod="openshift-dns/node-resolver-vjv4c"
Sep 18 13:44:14.041778 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.041090    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/47923c57-2f7a-46f5-b8c4-10efd987fa8b-cni-binary-copy\") pod \"multus-additional-cni-plugins-szg9k\" (UID: \"47923c57-2f7a-46f5-b8c4-10efd987fa8b\") " pod="openshift-multus/multus-additional-cni-plugins-szg9k"
Sep 18 13:44:14.047011 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.046994    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-141-222.ec2.internal"]
Sep 18 13:44:14.047691 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.047675    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 13:44:14.047784 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.047771    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-222.ec2.internal"
Sep 18 13:44:14.047962 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.047914    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-17 13:39:13 +0000 UTC" deadline="2028-06-20 04:41:54.49107646 +0000 UTC"
Sep 18 13:44:14.047962 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.047959    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15374h57m40.443119338s"
Sep 18 13:44:14.052956 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.052939    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 13:44:14.067226 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.067206    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 13:44:14.067332 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.067298    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-222.ec2.internal"]
Sep 18 13:44:14.138053 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.138033    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-t8cvj"
Sep 18 13:44:14.141330 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.141309    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/66b62a87-76ae-41e9-8e0c-dd48ff323e5d-device-dir\") pod \"aws-ebs-csi-driver-node-nscch\" (UID: \"66b62a87-76ae-41e9-8e0c-dd48ff323e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nscch"
Sep 18 13:44:14.141437 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.141336    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/66b62a87-76ae-41e9-8e0c-dd48ff323e5d-sys-fs\") pod \"aws-ebs-csi-driver-node-nscch\" (UID: \"66b62a87-76ae-41e9-8e0c-dd48ff323e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nscch"
Sep 18 13:44:14.141437 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.141354    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ac788d27-88a8-4ed2-9363-fe4f0aa51013-host\") pod \"node-ca-wczf6\" (UID: \"ac788d27-88a8-4ed2-9363-fe4f0aa51013\") " pod="openshift-image-registry/node-ca-wczf6"
Sep 18 13:44:14.141437 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.141377    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/47923c57-2f7a-46f5-b8c4-10efd987fa8b-system-cni-dir\") pod \"multus-additional-cni-plugins-szg9k\" (UID: \"47923c57-2f7a-46f5-b8c4-10efd987fa8b\") " pod="openshift-multus/multus-additional-cni-plugins-szg9k"
Sep 18 13:44:14.141437 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.141433    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ac788d27-88a8-4ed2-9363-fe4f0aa51013-host\") pod \"node-ca-wczf6\" (UID: \"ac788d27-88a8-4ed2-9363-fe4f0aa51013\") " pod="openshift-image-registry/node-ca-wczf6"
Sep 18 13:44:14.141656 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.141441    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/47923c57-2f7a-46f5-b8c4-10efd987fa8b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-szg9k\" (UID: \"47923c57-2f7a-46f5-b8c4-10efd987fa8b\") " pod="openshift-multus/multus-additional-cni-plugins-szg9k"
Sep 18 13:44:14.141656 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.141452    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/66b62a87-76ae-41e9-8e0c-dd48ff323e5d-device-dir\") pod \"aws-ebs-csi-driver-node-nscch\" (UID: \"66b62a87-76ae-41e9-8e0c-dd48ff323e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nscch"
Sep 18 13:44:14.141656 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.141478    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/47923c57-2f7a-46f5-b8c4-10efd987fa8b-system-cni-dir\") pod \"multus-additional-cni-plugins-szg9k\" (UID: \"47923c57-2f7a-46f5-b8c4-10efd987fa8b\") " pod="openshift-multus/multus-additional-cni-plugins-szg9k"
Sep 18 13:44:14.141656 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.141486    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f5d1bf14-b322-49d0-9fc8-b3a9a0aca036-var-lib-openvswitch\") pod \"ovnkube-node-wbv96\" (UID: \"f5d1bf14-b322-49d0-9fc8-b3a9a0aca036\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbv96"
Sep 18 13:44:14.141656 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.141449    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/66b62a87-76ae-41e9-8e0c-dd48ff323e5d-sys-fs\") pod \"aws-ebs-csi-driver-node-nscch\" (UID: \"66b62a87-76ae-41e9-8e0c-dd48ff323e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nscch"
Sep 18 13:44:14.141656 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.141517    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f5d1bf14-b322-49d0-9fc8-b3a9a0aca036-env-overrides\") pod \"ovnkube-node-wbv96\" (UID: \"f5d1bf14-b322-49d0-9fc8-b3a9a0aca036\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbv96"
Sep 18 13:44:14.141656 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.141537    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f5d1bf14-b322-49d0-9fc8-b3a9a0aca036-var-lib-openvswitch\") pod \"ovnkube-node-wbv96\" (UID: \"f5d1bf14-b322-49d0-9fc8-b3a9a0aca036\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbv96"
Sep 18 13:44:14.141656 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.141555    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cm5q\" (UniqueName: \"kubernetes.io/projected/2e756a4b-a7f8-4688-9a3a-efecba8641d2-kube-api-access-6cm5q\") pod \"iptables-alerter-vc9jp\" (UID: \"2e756a4b-a7f8-4688-9a3a-efecba8641d2\") " pod="openshift-network-operator/iptables-alerter-vc9jp"
Sep 18 13:44:14.141656 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.141589    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/47923c57-2f7a-46f5-b8c4-10efd987fa8b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-szg9k\" (UID: \"47923c57-2f7a-46f5-b8c4-10efd987fa8b\") " pod="openshift-multus/multus-additional-cni-plugins-szg9k"
Sep 18 13:44:14.141656 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.141613    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b4829a7a-35e0-44e4-b738-4ba4bd403929-system-cni-dir\") pod \"multus-ztfrw\" (UID: \"b4829a7a-35e0-44e4-b738-4ba4bd403929\") " pod="openshift-multus/multus-ztfrw"
Sep 18 13:44:14.141656 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.141635    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b4829a7a-35e0-44e4-b738-4ba4bd403929-cnibin\") pod \"multus-ztfrw\" (UID: \"b4829a7a-35e0-44e4-b738-4ba4bd403929\") " pod="openshift-multus/multus-ztfrw"
Sep 18 13:44:14.141656 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.141655    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e0b6f508-1744-4ab7-8e37-0f2f4df3a816-etc-sysconfig\") pod \"tuned-8k94g\" (UID: \"e0b6f508-1744-4ab7-8e37-0f2f4df3a816\") " pod="openshift-cluster-node-tuning-operator/tuned-8k94g"
Sep 18 13:44:14.142307 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.141673    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/66b62a87-76ae-41e9-8e0c-dd48ff323e5d-socket-dir\") pod \"aws-ebs-csi-driver-node-nscch\" (UID: \"66b62a87-76ae-41e9-8e0c-dd48ff323e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nscch"
Sep 18 13:44:14.142307 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.141691    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b4zq\" (UniqueName: \"kubernetes.io/projected/ac788d27-88a8-4ed2-9363-fe4f0aa51013-kube-api-access-9b4zq\") pod \"node-ca-wczf6\" (UID: \"ac788d27-88a8-4ed2-9363-fe4f0aa51013\") " pod="openshift-image-registry/node-ca-wczf6"
Sep 18 13:44:14.142307 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.141707    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b4829a7a-35e0-44e4-b738-4ba4bd403929-cni-binary-copy\") pod \"multus-ztfrw\" (UID: \"b4829a7a-35e0-44e4-b738-4ba4bd403929\") " pod="openshift-multus/multus-ztfrw"
Sep 18 13:44:14.142307 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.141730    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b4829a7a-35e0-44e4-b738-4ba4bd403929-host-run-netns\") pod \"multus-ztfrw\" (UID: \"b4829a7a-35e0-44e4-b738-4ba4bd403929\") " pod="openshift-multus/multus-ztfrw"
Sep 18 13:44:14.142307 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.141748    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f5d1bf14-b322-49d0-9fc8-b3a9a0aca036-systemd-units\") pod \"ovnkube-node-wbv96\" (UID: \"f5d1bf14-b322-49d0-9fc8-b3a9a0aca036\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbv96"
Sep 18 13:44:14.142307 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.141764    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/802e0a84-ff96-44d6-9aa2-db34555f46eb-original-pull-secret\") pod \"global-pull-secret-syncer-bdwmn\" (UID: \"802e0a84-ff96-44d6-9aa2-db34555f46eb\") " pod="kube-system/global-pull-secret-syncer-bdwmn"
Sep 18 13:44:14.142307 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.141780    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0b6f508-1744-4ab7-8e37-0f2f4df3a816-etc-kubernetes\") pod \"tuned-8k94g\" (UID: \"e0b6f508-1744-4ab7-8e37-0f2f4df3a816\") " pod="openshift-cluster-node-tuning-operator/tuned-8k94g"
Sep 18 13:44:14.142307 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.141781    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/47923c57-2f7a-46f5-b8c4-10efd987fa8b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-szg9k\" (UID: \"47923c57-2f7a-46f5-b8c4-10efd987fa8b\") " pod="openshift-multus/multus-additional-cni-plugins-szg9k"
Sep 18 13:44:14.142307 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.141794    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e0b6f508-1744-4ab7-8e37-0f2f4df3a816-host\") pod \"tuned-8k94g\" (UID: \"e0b6f508-1744-4ab7-8e37-0f2f4df3a816\") " pod="openshift-cluster-node-tuning-operator/tuned-8k94g"
Sep 18 13:44:14.142307 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.141818    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/66b62a87-76ae-41e9-8e0c-dd48ff323e5d-registration-dir\") pod \"aws-ebs-csi-driver-node-nscch\" (UID: \"66b62a87-76ae-41e9-8e0c-dd48ff323e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nscch"
Sep 18 13:44:14.142307 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.141836    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b4829a7a-35e0-44e4-b738-4ba4bd403929-multus-daemon-config\") pod \"multus-ztfrw\" (UID: \"b4829a7a-35e0-44e4-b738-4ba4bd403929\") " pod="openshift-multus/multus-ztfrw"
Sep 18 13:44:14.142307 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.141875    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b4829a7a-35e0-44e4-b738-4ba4bd403929-cnibin\") pod \"multus-ztfrw\" (UID: \"b4829a7a-35e0-44e4-b738-4ba4bd403929\") " pod="openshift-multus/multus-ztfrw"
Sep 18 13:44:14.142307 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.142018    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e0b6f508-1744-4ab7-8e37-0f2f4df3a816-etc-sysctl-d\") pod \"tuned-8k94g\" (UID: \"e0b6f508-1744-4ab7-8e37-0f2f4df3a816\") " pod="openshift-cluster-node-tuning-operator/tuned-8k94g"
Sep 18 13:44:14.142307 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.142049    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e0b6f508-1744-4ab7-8e37-0f2f4df3a816-etc-sysconfig\") pod \"tuned-8k94g\" (UID: \"e0b6f508-1744-4ab7-8e37-0f2f4df3a816\") " pod="openshift-cluster-node-tuning-operator/tuned-8k94g"
Sep 18 13:44:14.142307 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.142087    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f5d1bf14-b322-49d0-9fc8-b3a9a0aca036-env-overrides\") pod \"ovnkube-node-wbv96\" (UID: \"f5d1bf14-b322-49d0-9fc8-b3a9a0aca036\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbv96"
Sep 18 13:44:14.142307 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.142106    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b4829a7a-35e0-44e4-b738-4ba4bd403929-system-cni-dir\") pod \"multus-ztfrw\" (UID: \"b4829a7a-35e0-44e4-b738-4ba4bd403929\") " pod="openshift-multus/multus-ztfrw"
Sep 18 13:44:14.142307 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.141878    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e0b6f508-1744-4ab7-8e37-0f2f4df3a816-etc-sysctl-d\") pod \"tuned-8k94g\" (UID: \"e0b6f508-1744-4ab7-8e37-0f2f4df3a816\") " pod="openshift-cluster-node-tuning-operator/tuned-8k94g"
Sep 18 13:44:14.142307 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.142165    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0b6f508-1744-4ab7-8e37-0f2f4df3a816-etc-kubernetes\") pod \"tuned-8k94g\" (UID: \"e0b6f508-1744-4ab7-8e37-0f2f4df3a816\") " pod="openshift-cluster-node-tuning-operator/tuned-8k94g"
Sep 18 13:44:14.143277 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.142186    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2e756a4b-a7f8-4688-9a3a-efecba8641d2-host-slash\") pod \"iptables-alerter-vc9jp\" (UID: \"2e756a4b-a7f8-4688-9a3a-efecba8641d2\") " pod="openshift-network-operator/iptables-alerter-vc9jp"
Sep 18 13:44:14.143277 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.142193    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/66b62a87-76ae-41e9-8e0c-dd48ff323e5d-socket-dir\") pod \"aws-ebs-csi-driver-node-nscch\" (UID: \"66b62a87-76ae-41e9-8e0c-dd48ff323e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nscch"
Sep 18 13:44:14.143277 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.142205    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b4829a7a-35e0-44e4-b738-4ba4bd403929-host-run-netns\") pod \"multus-ztfrw\" (UID: \"b4829a7a-35e0-44e4-b738-4ba4bd403929\") " pod="openshift-multus/multus-ztfrw"
Sep 18 13:44:14.143277 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.142239    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2e756a4b-a7f8-4688-9a3a-efecba8641d2-host-slash\") pod \"iptables-alerter-vc9jp\" (UID: \"2e756a4b-a7f8-4688-9a3a-efecba8641d2\") " pod="openshift-network-operator/iptables-alerter-vc9jp"
Sep 18 13:44:14.143277 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.142241    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wqns\" (UniqueName: \"kubernetes.io/projected/66b62a87-76ae-41e9-8e0c-dd48ff323e5d-kube-api-access-5wqns\") pod \"aws-ebs-csi-driver-node-nscch\" (UID: \"66b62a87-76ae-41e9-8e0c-dd48ff323e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nscch"
Sep 18 13:44:14.143277 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.142279    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/47923c57-2f7a-46f5-b8c4-10efd987fa8b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-szg9k\" (UID: \"47923c57-2f7a-46f5-b8c4-10efd987fa8b\") " pod="openshift-multus/multus-additional-cni-plugins-szg9k"
Sep 18 13:44:14.143277 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.142292    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f5d1bf14-b322-49d0-9fc8-b3a9a0aca036-systemd-units\") pod \"ovnkube-node-wbv96\" (UID: \"f5d1bf14-b322-49d0-9fc8-b3a9a0aca036\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbv96"
Sep 18 13:44:14.143277 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.142310    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtkdz\" (UniqueName: \"kubernetes.io/projected/47923c57-2f7a-46f5-b8c4-10efd987fa8b-kube-api-access-mtkdz\") pod \"multus-additional-cni-plugins-szg9k\" (UID: \"47923c57-2f7a-46f5-b8c4-10efd987fa8b\") " pod="openshift-multus/multus-additional-cni-plugins-szg9k"
Sep 18 13:44:14.143277 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.142337    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b4829a7a-35e0-44e4-b738-4ba4bd403929-host-var-lib-cni-bin\") pod \"multus-ztfrw\" (UID: \"b4829a7a-35e0-44e4-b738-4ba4bd403929\") " pod="openshift-multus/multus-ztfrw"
Sep 18 13:44:14.143277 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:14.142365    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 13:44:14.143277 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.142385    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f5d1bf14-b322-49d0-9fc8-b3a9a0aca036-run-ovn\") pod \"ovnkube-node-wbv96\" (UID: \"f5d1bf14-b322-49d0-9fc8-b3a9a0aca036\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbv96"
Sep 18 13:44:14.143277 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.142390    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b4829a7a-35e0-44e4-b738-4ba4bd403929-cni-binary-copy\") pod \"multus-ztfrw\" (UID: \"b4829a7a-35e0-44e4-b738-4ba4bd403929\") " pod="openshift-multus/multus-ztfrw"
Sep 18 13:44:14.143277 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.142409    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f5d1bf14-b322-49d0-9fc8-b3a9a0aca036-host-cni-netd\") pod \"ovnkube-node-wbv96\" (UID: \"f5d1bf14-b322-49d0-9fc8-b3a9a0aca036\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbv96"
Sep 18 13:44:14.143277 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:14.142438    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/802e0a84-ff96-44d6-9aa2-db34555f46eb-original-pull-secret podName:802e0a84-ff96-44d6-9aa2-db34555f46eb nodeName:}" failed. No retries permitted until 2026-09-18 13:44:14.642416394 +0000 UTC m=+3.695075728 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/802e0a84-ff96-44d6-9aa2-db34555f46eb-original-pull-secret") pod "global-pull-secret-syncer-bdwmn" (UID: "802e0a84-ff96-44d6-9aa2-db34555f46eb") : object "kube-system"/"original-pull-secret" not registered
Sep 18 13:44:14.143277 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.142445    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f5d1bf14-b322-49d0-9fc8-b3a9a0aca036-host-cni-netd\") pod \"ovnkube-node-wbv96\" (UID: \"f5d1bf14-b322-49d0-9fc8-b3a9a0aca036\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbv96"
Sep 18 13:44:14.143277 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.142473    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f5d1bf14-b322-49d0-9fc8-b3a9a0aca036-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wbv96\" (UID: \"f5d1bf14-b322-49d0-9fc8-b3a9a0aca036\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbv96"
Sep 18 13:44:14.143277 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.142502    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e0b6f508-1744-4ab7-8e37-0f2f4df3a816-etc-modprobe-d\") pod \"tuned-8k94g\" (UID: \"e0b6f508-1744-4ab7-8e37-0f2f4df3a816\") " pod="openshift-cluster-node-tuning-operator/tuned-8k94g"
Sep 18 13:44:14.144135 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.142531    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b4829a7a-35e0-44e4-b738-4ba4bd403929-multus-conf-dir\") pod \"multus-ztfrw\" (UID: \"b4829a7a-35e0-44e4-b738-4ba4bd403929\") " pod="openshift-multus/multus-ztfrw"
Sep 18 13:44:14.144135 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.142537    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e0b6f508-1744-4ab7-8e37-0f2f4df3a816-host\") pod \"tuned-8k94g\" (UID: \"e0b6f508-1744-4ab7-8e37-0f2f4df3a816\") " pod="openshift-cluster-node-tuning-operator/tuned-8k94g"
Sep 18 13:44:14.144135 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.142501    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/66b62a87-76ae-41e9-8e0c-dd48ff323e5d-registration-dir\") pod \"aws-ebs-csi-driver-node-nscch\" (UID: \"66b62a87-76ae-41e9-8e0c-dd48ff323e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nscch"
Sep 18 13:44:14.144135 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.142590    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f5d1bf14-b322-49d0-9fc8-b3a9a0aca036-host-slash\") pod \"ovnkube-node-wbv96\" (UID: \"f5d1bf14-b322-49d0-9fc8-b3a9a0aca036\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbv96"
Sep 18 13:44:14.144135 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.142616    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f5d1bf14-b322-49d0-9fc8-b3a9a0aca036-etc-openvswitch\") pod \"ovnkube-node-wbv96\" (UID: \"f5d1bf14-b322-49d0-9fc8-b3a9a0aca036\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbv96"
Sep 18 13:44:14.144135 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.142641    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f5d1bf14-b322-49d0-9fc8-b3a9a0aca036-log-socket\") pod \"ovnkube-node-wbv96\" (UID: \"f5d1bf14-b322-49d0-9fc8-b3a9a0aca036\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbv96"
Sep 18 13:44:14.144135 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.142668    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e0b6f508-1744-4ab7-8e37-0f2f4df3a816-sys\") pod \"tuned-8k94g\" (UID: \"e0b6f508-1744-4ab7-8e37-0f2f4df3a816\") " pod="openshift-cluster-node-tuning-operator/tuned-8k94g"
Sep 18 13:44:14.144135 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.142697    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmqn6\" (UniqueName: \"kubernetes.io/projected/6f382898-0370-425b-8c01-6587a985e3f0-kube-api-access-dmqn6\") pod \"network-check-target-hj8lg\" (UID: \"6f382898-0370-425b-8c01-6587a985e3f0\") " pod="openshift-network-diagnostics/network-check-target-hj8lg"
Sep 18 13:44:14.144135 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.142713    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b4829a7a-35e0-44e4-b738-4ba4bd403929-host-var-lib-cni-bin\") pod \"multus-ztfrw\" (UID: \"b4829a7a-35e0-44e4-b738-4ba4bd403929\") " pod="openshift-multus/multus-ztfrw"
Sep 18 13:44:14.144135 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.142722    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b4829a7a-35e0-44e4-b738-4ba4bd403929-multus-daemon-config\") pod \"multus-ztfrw\" (UID: \"b4829a7a-35e0-44e4-b738-4ba4bd403929\") " pod="openshift-multus/multus-ztfrw"
Sep 18 13:44:14.144135 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.142770    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f5d1bf14-b322-49d0-9fc8-b3a9a0aca036-etc-openvswitch\") pod \"ovnkube-node-wbv96\" (UID: \"f5d1bf14-b322-49d0-9fc8-b3a9a0aca036\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbv96"
Sep 18 13:44:14.144135 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.142791    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f5d1bf14-b322-49d0-9fc8-b3a9a0aca036-run-ovn\") pod \"ovnkube-node-wbv96\" (UID: \"f5d1bf14-b322-49d0-9fc8-b3a9a0aca036\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbv96"
Sep 18 13:44:14.144135 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.142804    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dbd84791-daa9-43f0-9f32-8f4415eceeca-hosts-file\") pod \"node-resolver-vjv4c\" (UID: \"dbd84791-daa9-43f0-9f32-8f4415eceeca\") " pod="openshift-dns/node-resolver-vjv4c"
Sep 18 13:44:14.144135 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.142802    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e0b6f508-1744-4ab7-8e37-0f2f4df3a816-sys\") pod \"tuned-8k94g\" (UID: \"e0b6f508-1744-4ab7-8e37-0f2f4df3a816\") " pod="openshift-cluster-node-tuning-operator/tuned-8k94g"
Sep 18 13:44:14.144135 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.142818    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/47923c57-2f7a-46f5-b8c4-10efd987fa8b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-szg9k\" (UID: \"47923c57-2f7a-46f5-b8c4-10efd987fa8b\") " pod="openshift-multus/multus-additional-cni-plugins-szg9k"
Sep 18 13:44:14.144135 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.142838    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/47923c57-2f7a-46f5-b8c4-10efd987fa8b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-szg9k\" (UID: \"47923c57-2f7a-46f5-b8c4-10efd987fa8b\") " pod="openshift-multus/multus-additional-cni-plugins-szg9k"
Sep 18 13:44:14.144135 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.142848    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f5d1bf14-b322-49d0-9fc8-b3a9a0aca036-host-slash\") pod \"ovnkube-node-wbv96\" (UID: \"f5d1bf14-b322-49d0-9fc8-b3a9a0aca036\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbv96"
Sep 18 13:44:14.144135 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.142861    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dbd84791-daa9-43f0-9f32-8f4415eceeca-hosts-file\") pod \"node-resolver-vjv4c\" (UID: \"dbd84791-daa9-43f0-9f32-8f4415eceeca\") " pod="openshift-dns/node-resolver-vjv4c"
Sep 18 13:44:14.144871 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.142889    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f5d1bf14-b322-49d0-9fc8-b3a9a0aca036-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wbv96\" (UID: \"f5d1bf14-b322-49d0-9fc8-b3a9a0aca036\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbv96"
Sep 18 13:44:14.144871 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.142895    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e0b6f508-1744-4ab7-8e37-0f2f4df3a816-etc-modprobe-d\") pod \"tuned-8k94g\" (UID: \"e0b6f508-1744-4ab7-8e37-0f2f4df3a816\") " pod="openshift-cluster-node-tuning-operator/tuned-8k94g"
Sep 18 13:44:14.144871 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.142916    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f5d1bf14-b322-49d0-9fc8-b3a9a0aca036-run-openvswitch\") pod \"ovnkube-node-wbv96\" (UID: \"f5d1bf14-b322-49d0-9fc8-b3a9a0aca036\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbv96"
Sep 18 13:44:14.144871 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.142941    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f5d1bf14-b322-49d0-9fc8-b3a9a0aca036-log-socket\") pod \"ovnkube-node-wbv96\" (UID: \"f5d1bf14-b322-49d0-9fc8-b3a9a0aca036\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbv96"
Sep 18 13:44:14.144871 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.142960    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b4829a7a-35e0-44e4-b738-4ba4bd403929-host-run-multus-certs\") pod \"multus-ztfrw\" (UID: \"b4829a7a-35e0-44e4-b738-4ba4bd403929\") " pod="openshift-multus/multus-ztfrw"
Sep 18 13:44:14.144871 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.142964    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b4829a7a-35e0-44e4-b738-4ba4bd403929-multus-conf-dir\") pod \"multus-ztfrw\" (UID: \"b4829a7a-35e0-44e4-b738-4ba4bd403929\") " pod="openshift-multus/multus-ztfrw"
Sep 18 13:44:14.144871 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.142985    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f5d1bf14-b322-49d0-9fc8-b3a9a0aca036-run-openvswitch\") pod \"ovnkube-node-wbv96\" (UID: \"f5d1bf14-b322-49d0-9fc8-b3a9a0aca036\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbv96"
Sep 18 13:44:14.144871 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.143019    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f5d1bf14-b322-49d0-9fc8-b3a9a0aca036-node-log\") pod \"ovnkube-node-wbv96\" (UID: \"f5d1bf14-b322-49d0-9fc8-b3a9a0aca036\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbv96"
Sep 18 13:44:14.144871 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.143037    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b4829a7a-35e0-44e4-b738-4ba4bd403929-host-run-multus-certs\") pod \"multus-ztfrw\" (UID: \"b4829a7a-35e0-44e4-b738-4ba4bd403929\") " pod="openshift-multus/multus-ztfrw"
Sep 18 13:44:14.144871 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.143081    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e0b6f508-1744-4ab7-8e37-0f2f4df3a816-lib-modules\") pod \"tuned-8k94g\" (UID: \"e0b6f508-1744-4ab7-8e37-0f2f4df3a816\") " pod="openshift-cluster-node-tuning-operator/tuned-8k94g"
Sep 18 13:44:14.144871 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.143086    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f5d1bf14-b322-49d0-9fc8-b3a9a0aca036-node-log\") pod \"ovnkube-node-wbv96\" (UID: \"f5d1bf14-b322-49d0-9fc8-b3a9a0aca036\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbv96"
Sep 18 13:44:14.144871 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.143169    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8902efdf-cf1d-4513-bfbb-cdf2b9a2b922-agent-certs\") pod \"konnectivity-agent-7cm48\" (UID: \"8902efdf-cf1d-4513-bfbb-cdf2b9a2b922\") " pod="kube-system/konnectivity-agent-7cm48"
Sep 18 13:44:14.144871 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.143202    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/66b62a87-76ae-41e9-8e0c-dd48ff323e5d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nscch\" (UID: \"66b62a87-76ae-41e9-8e0c-dd48ff323e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nscch"
Sep 18 13:44:14.144871 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.143230    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b4829a7a-35e0-44e4-b738-4ba4bd403929-host-var-lib-kubelet\") pod \"multus-ztfrw\" (UID: \"b4829a7a-35e0-44e4-b738-4ba4bd403929\") " pod="openshift-multus/multus-ztfrw"
Sep 18 13:44:14.144871 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.143259    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b4829a7a-35e0-44e4-b738-4ba4bd403929-hostroot\") pod \"multus-ztfrw\" (UID: \"b4829a7a-35e0-44e4-b738-4ba4bd403929\") " pod="openshift-multus/multus-ztfrw"
Sep 18 13:44:14.144871 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.143283    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/802e0a84-ff96-44d6-9aa2-db34555f46eb-kubelet-config\") pod \"global-pull-secret-syncer-bdwmn\" (UID: \"802e0a84-ff96-44d6-9aa2-db34555f46eb\") " pod="kube-system/global-pull-secret-syncer-bdwmn"
Sep 18 13:44:14.144871 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.143292    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b4829a7a-35e0-44e4-b738-4ba4bd403929-host-var-lib-kubelet\") pod \"multus-ztfrw\" (UID: \"b4829a7a-35e0-44e4-b738-4ba4bd403929\") " pod="openshift-multus/multus-ztfrw"
Sep 18 13:44:14.144871 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.143294    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/66b62a87-76ae-41e9-8e0c-dd48ff323e5d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nscch\" (UID: \"66b62a87-76ae-41e9-8e0c-dd48ff323e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nscch"
Sep 18 13:44:14.145942 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.143307    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e0b6f508-1744-4ab7-8e37-0f2f4df3a816-etc-sysctl-conf\") pod \"tuned-8k94g\" (UID: \"e0b6f508-1744-4ab7-8e37-0f2f4df3a816\") " pod="openshift-cluster-node-tuning-operator/tuned-8k94g"
Sep 18 13:44:14.145942 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.143293    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e0b6f508-1744-4ab7-8e37-0f2f4df3a816-lib-modules\") pod \"tuned-8k94g\" (UID: \"e0b6f508-1744-4ab7-8e37-0f2f4df3a816\") " pod="openshift-cluster-node-tuning-operator/tuned-8k94g"
Sep 18 13:44:14.145942 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.143327    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b4829a7a-35e0-44e4-b738-4ba4bd403929-hostroot\") pod \"multus-ztfrw\" (UID: \"b4829a7a-35e0-44e4-b738-4ba4bd403929\") " pod="openshift-multus/multus-ztfrw"
Sep 18 13:44:14.145942 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.143334    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e0b6f508-1744-4ab7-8e37-0f2f4df3a816-etc-tuned\") pod \"tuned-8k94g\" (UID: \"e0b6f508-1744-4ab7-8e37-0f2f4df3a816\") " pod="openshift-cluster-node-tuning-operator/tuned-8k94g"
Sep 18 13:44:14.145942 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.143392    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/47923c57-2f7a-46f5-b8c4-10efd987fa8b-cnibin\") pod \"multus-additional-cni-plugins-szg9k\" (UID: \"47923c57-2f7a-46f5-b8c4-10efd987fa8b\") " pod="openshift-multus/multus-additional-cni-plugins-szg9k"
Sep 18 13:44:14.145942 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.143346    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/802e0a84-ff96-44d6-9aa2-db34555f46eb-kubelet-config\") pod \"global-pull-secret-syncer-bdwmn\" (UID: \"802e0a84-ff96-44d6-9aa2-db34555f46eb\") " pod="kube-system/global-pull-secret-syncer-bdwmn"
Sep 18 13:44:14.145942 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.143423    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b4829a7a-35e0-44e4-b738-4ba4bd403929-multus-socket-dir-parent\") pod \"multus-ztfrw\" (UID: \"b4829a7a-35e0-44e4-b738-4ba4bd403929\") " pod="openshift-multus/multus-ztfrw"
Sep 18 13:44:14.145942 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.143462    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/47923c57-2f7a-46f5-b8c4-10efd987fa8b-cnibin\") pod \"multus-additional-cni-plugins-szg9k\" (UID: \"47923c57-2f7a-46f5-b8c4-10efd987fa8b\") " pod="openshift-multus/multus-additional-cni-plugins-szg9k"
Sep 18 13:44:14.145942 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.143486    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b4829a7a-35e0-44e4-b738-4ba4bd403929-multus-socket-dir-parent\") pod \"multus-ztfrw\" (UID: \"b4829a7a-35e0-44e4-b738-4ba4bd403929\") " pod="openshift-multus/multus-ztfrw"
Sep 18 13:44:14.145942 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.143501    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e0b6f508-1744-4ab7-8e37-0f2f4df3a816-etc-sysctl-conf\") pod \"tuned-8k94g\" (UID: \"e0b6f508-1744-4ab7-8e37-0f2f4df3a816\") " pod="openshift-cluster-node-tuning-operator/tuned-8k94g"
Sep 18 13:44:14.145942 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.143492    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9txp\" (UniqueName: \"kubernetes.io/projected/b4829a7a-35e0-44e4-b738-4ba4bd403929-kube-api-access-s9txp\") pod \"multus-ztfrw\" (UID: \"b4829a7a-35e0-44e4-b738-4ba4bd403929\") " pod="openshift-multus/multus-ztfrw"
Sep 18 13:44:14.145942 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.143493    2577 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 18 13:44:14.145942 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.143554    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f5d1bf14-b322-49d0-9fc8-b3a9a0aca036-host-run-netns\") pod \"ovnkube-node-wbv96\" (UID: \"f5d1bf14-b322-49d0-9fc8-b3a9a0aca036\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbv96"
Sep 18 13:44:14.145942 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.143580    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0b6f508-1744-4ab7-8e37-0f2f4df3a816-var-lib-kubelet\") pod \"tuned-8k94g\" (UID: \"e0b6f508-1744-4ab7-8e37-0f2f4df3a816\") " pod="openshift-cluster-node-tuning-operator/tuned-8k94g"
Sep 18 13:44:14.145942 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.143604    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0b6f508-1744-4ab7-8e37-0f2f4df3a816-tmp\") pod \"tuned-8k94g\" (UID: \"e0b6f508-1744-4ab7-8e37-0f2f4df3a816\") " pod="openshift-cluster-node-tuning-operator/tuned-8k94g"
Sep 18 13:44:14.145942 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.143633    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h4d5\" (UniqueName: \"kubernetes.io/projected/dbd84791-daa9-43f0-9f32-8f4415eceeca-kube-api-access-7h4d5\") pod \"node-resolver-vjv4c\" (UID: \"dbd84791-daa9-43f0-9f32-8f4415eceeca\") " pod="openshift-dns/node-resolver-vjv4c"
Sep 18 13:44:14.145942 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.143639    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f5d1bf14-b322-49d0-9fc8-b3a9a0aca036-host-run-netns\") pod \"ovnkube-node-wbv96\" (UID: \"f5d1bf14-b322-49d0-9fc8-b3a9a0aca036\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbv96"
Sep 18 13:44:14.145942 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.143665    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/47923c57-2f7a-46f5-b8c4-10efd987fa8b-cni-binary-copy\") pod \"multus-additional-cni-plugins-szg9k\" (UID: \"47923c57-2f7a-46f5-b8c4-10efd987fa8b\") " pod="openshift-multus/multus-additional-cni-plugins-szg9k"
Sep 18 13:44:14.146891 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.143682    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0b6f508-1744-4ab7-8e37-0f2f4df3a816-var-lib-kubelet\") pod \"tuned-8k94g\" (UID: \"e0b6f508-1744-4ab7-8e37-0f2f4df3a816\") " pod="openshift-cluster-node-tuning-operator/tuned-8k94g"
Sep 18 13:44:14.146891 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.143806    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b4829a7a-35e0-44e4-b738-4ba4bd403929-etc-kubernetes\") pod \"multus-ztfrw\" (UID: \"b4829a7a-35e0-44e4-b738-4ba4bd403929\") " pod="openshift-multus/multus-ztfrw"
Sep 18 13:44:14.146891 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.143873    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdv8r\" (UniqueName: \"kubernetes.io/projected/f5d1bf14-b322-49d0-9fc8-b3a9a0aca036-kube-api-access-kdv8r\") pod \"ovnkube-node-wbv96\" (UID: \"f5d1bf14-b322-49d0-9fc8-b3a9a0aca036\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbv96"
Sep 18 13:44:14.146891 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.143900    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e0b6f508-1744-4ab7-8e37-0f2f4df3a816-etc-systemd\") pod \"tuned-8k94g\" (UID: \"e0b6f508-1744-4ab7-8e37-0f2f4df3a816\") " pod="openshift-cluster-node-tuning-operator/tuned-8k94g"
Sep 18 13:44:14.146891 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.143977    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrxhq\" (UniqueName: \"kubernetes.io/projected/e0b6f508-1744-4ab7-8e37-0f2f4df3a816-kube-api-access-lrxhq\") pod \"tuned-8k94g\" (UID: \"e0b6f508-1744-4ab7-8e37-0f2f4df3a816\") " pod="openshift-cluster-node-tuning-operator/tuned-8k94g"
Sep 18 13:44:14.146891 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.144003    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbd84791-daa9-43f0-9f32-8f4415eceeca-tmp-dir\") pod \"node-resolver-vjv4c\" (UID: \"dbd84791-daa9-43f0-9f32-8f4415eceeca\") " pod="openshift-dns/node-resolver-vjv4c"
Sep 18 13:44:14.146891 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.144047    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b4829a7a-35e0-44e4-b738-4ba4bd403929-host-run-k8s-cni-cncf-io\") pod \"multus-ztfrw\" (UID: \"b4829a7a-35e0-44e4-b738-4ba4bd403929\") " pod="openshift-multus/multus-ztfrw"
Sep 18 13:44:14.146891 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.144075    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f5d1bf14-b322-49d0-9fc8-b3a9a0aca036-host-cni-bin\") pod \"ovnkube-node-wbv96\" (UID: \"f5d1bf14-b322-49d0-9fc8-b3a9a0aca036\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbv96"
Sep 18 13:44:14.146891 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.144118    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/66b62a87-76ae-41e9-8e0c-dd48ff323e5d-etc-selinux\") pod \"aws-ebs-csi-driver-node-nscch\" (UID: \"66b62a87-76ae-41e9-8e0c-dd48ff323e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nscch"
Sep 18 13:44:14.146891 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.144147    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ac788d27-88a8-4ed2-9363-fe4f0aa51013-serviceca\") pod \"node-ca-wczf6\" (UID: \"ac788d27-88a8-4ed2-9363-fe4f0aa51013\") " pod="openshift-image-registry/node-ca-wczf6"
Sep 18 13:44:14.146891 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.144177    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f5d1bf14-b322-49d0-9fc8-b3a9a0aca036-ovn-node-metrics-cert\") pod \"ovnkube-node-wbv96\" (UID: \"f5d1bf14-b322-49d0-9fc8-b3a9a0aca036\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbv96"
Sep 18 13:44:14.146891 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.144194    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/47923c57-2f7a-46f5-b8c4-10efd987fa8b-cni-binary-copy\") pod \"multus-additional-cni-plugins-szg9k\" (UID: \"47923c57-2f7a-46f5-b8c4-10efd987fa8b\") " pod="openshift-multus/multus-additional-cni-plugins-szg9k"
Sep 18 13:44:14.146891 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.144222    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8902efdf-cf1d-4513-bfbb-cdf2b9a2b922-konnectivity-ca\") pod \"konnectivity-agent-7cm48\" (UID: \"8902efdf-cf1d-4513-bfbb-cdf2b9a2b922\") " pod="kube-system/konnectivity-agent-7cm48"
Sep 18 13:44:14.146891 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.144477    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbd84791-daa9-43f0-9f32-8f4415eceeca-tmp-dir\") pod \"node-resolver-vjv4c\" (UID: \"dbd84791-daa9-43f0-9f32-8f4415eceeca\") " pod="openshift-dns/node-resolver-vjv4c"
Sep 18 13:44:14.146891 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.144504    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b4829a7a-35e0-44e4-b738-4ba4bd403929-host-run-k8s-cni-cncf-io\") pod \"multus-ztfrw\" (UID: \"b4829a7a-35e0-44e4-b738-4ba4bd403929\") " pod="openshift-multus/multus-ztfrw"
Sep 18 13:44:14.146891 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.144536    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e0b6f508-1744-4ab7-8e37-0f2f4df3a816-run\") pod \"tuned-8k94g\" (UID: \"e0b6f508-1744-4ab7-8e37-0f2f4df3a816\") " pod="openshift-cluster-node-tuning-operator/tuned-8k94g"
Sep 18 13:44:14.146891 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.144568    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87eba103-cfec-4000-a10c-fd4fd512ac90-metrics-certs\") pod \"network-metrics-daemon-nb9bg\" (UID: \"87eba103-cfec-4000-a10c-fd4fd512ac90\") " pod="openshift-multus/network-metrics-daemon-nb9bg"
Sep 18 13:44:14.146891 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.144572    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f5d1bf14-b322-49d0-9fc8-b3a9a0aca036-host-cni-bin\") pod \"ovnkube-node-wbv96\" (UID: \"f5d1bf14-b322-49d0-9fc8-b3a9a0aca036\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbv96"
Sep 18 13:44:14.147795 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.144598    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnjtp\" (UniqueName: \"kubernetes.io/projected/87eba103-cfec-4000-a10c-fd4fd512ac90-kube-api-access-xnjtp\") pod \"network-metrics-daemon-nb9bg\" (UID: \"87eba103-cfec-4000-a10c-fd4fd512ac90\") " pod="openshift-multus/network-metrics-daemon-nb9bg"
Sep 18 13:44:14.147795 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.144643    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2e756a4b-a7f8-4688-9a3a-efecba8641d2-iptables-alerter-script\") pod \"iptables-alerter-vc9jp\" (UID: \"2e756a4b-a7f8-4688-9a3a-efecba8641d2\") " pod="openshift-network-operator/iptables-alerter-vc9jp"
Sep 18 13:44:14.147795 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.144674    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b4829a7a-35e0-44e4-b738-4ba4bd403929-host-var-lib-cni-multus\") pod \"multus-ztfrw\" (UID: \"b4829a7a-35e0-44e4-b738-4ba4bd403929\") " pod="openshift-multus/multus-ztfrw"
Sep 18 13:44:14.147795 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.144700    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f5d1bf14-b322-49d0-9fc8-b3a9a0aca036-host-kubelet\") pod \"ovnkube-node-wbv96\" (UID: \"f5d1bf14-b322-49d0-9fc8-b3a9a0aca036\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbv96"
Sep 18 13:44:14.147795 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.144729    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f5d1bf14-b322-49d0-9fc8-b3a9a0aca036-host-run-ovn-kubernetes\") pod \"ovnkube-node-wbv96\" (UID: \"f5d1bf14-b322-49d0-9fc8-b3a9a0aca036\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbv96"
Sep 18 13:44:14.147795 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.144759    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f5d1bf14-b322-49d0-9fc8-b3a9a0aca036-ovnkube-config\") pod \"ovnkube-node-wbv96\" (UID: \"f5d1bf14-b322-49d0-9fc8-b3a9a0aca036\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbv96"
Sep 18 13:44:14.147795 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.144798    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/802e0a84-ff96-44d6-9aa2-db34555f46eb-dbus\") pod \"global-pull-secret-syncer-bdwmn\" (UID: \"802e0a84-ff96-44d6-9aa2-db34555f46eb\") " pod="kube-system/global-pull-secret-syncer-bdwmn"
Sep 18 13:44:14.147795 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.144828    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b4829a7a-35e0-44e4-b738-4ba4bd403929-multus-cni-dir\") pod \"multus-ztfrw\" (UID: \"b4829a7a-35e0-44e4-b738-4ba4bd403929\") " pod="openshift-multus/multus-ztfrw"
Sep 18 13:44:14.147795 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.144857    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f5d1bf14-b322-49d0-9fc8-b3a9a0aca036-run-systemd\") pod \"ovnkube-node-wbv96\" (UID: \"f5d1bf14-b322-49d0-9fc8-b3a9a0aca036\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbv96"
Sep 18 13:44:14.147795 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.144888    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f5d1bf14-b322-49d0-9fc8-b3a9a0aca036-ovnkube-script-lib\") pod \"ovnkube-node-wbv96\" (UID: \"f5d1bf14-b322-49d0-9fc8-b3a9a0aca036\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbv96"
Sep 18 13:44:14.147795 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.145030    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/66b62a87-76ae-41e9-8e0c-dd48ff323e5d-etc-selinux\") pod \"aws-ebs-csi-driver-node-nscch\" (UID: \"66b62a87-76ae-41e9-8e0c-dd48ff323e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nscch"
Sep 18 13:44:14.147795 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.145039    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ac788d27-88a8-4ed2-9363-fe4f0aa51013-serviceca\") pod \"node-ca-wczf6\" (UID: \"ac788d27-88a8-4ed2-9363-fe4f0aa51013\") " pod="openshift-image-registry/node-ca-wczf6"
Sep 18 13:44:14.147795 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.145406    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b4829a7a-35e0-44e4-b738-4ba4bd403929-etc-kubernetes\") pod \"multus-ztfrw\" (UID: \"b4829a7a-35e0-44e4-b738-4ba4bd403929\") " pod="openshift-multus/multus-ztfrw"
Sep 18 13:44:14.147795 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.145477    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f5d1bf14-b322-49d0-9fc8-b3a9a0aca036-ovnkube-script-lib\") pod \"ovnkube-node-wbv96\" (UID: \"f5d1bf14-b322-49d0-9fc8-b3a9a0aca036\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbv96"
Sep 18 13:44:14.147795 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.145606    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2e756a4b-a7f8-4688-9a3a-efecba8641d2-iptables-alerter-script\") pod \"iptables-alerter-vc9jp\" (UID: \"2e756a4b-a7f8-4688-9a3a-efecba8641d2\") " pod="openshift-network-operator/iptables-alerter-vc9jp"
Sep 18 13:44:14.147795 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.145658    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8902efdf-cf1d-4513-bfbb-cdf2b9a2b922-konnectivity-ca\") pod \"konnectivity-agent-7cm48\" (UID: \"8902efdf-cf1d-4513-bfbb-cdf2b9a2b922\") " pod="kube-system/konnectivity-agent-7cm48"
Sep 18 13:44:14.147795 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.145663    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b4829a7a-35e0-44e4-b738-4ba4bd403929-host-var-lib-cni-multus\") pod \"multus-ztfrw\" (UID: \"b4829a7a-35e0-44e4-b738-4ba4bd403929\") " pod="openshift-multus/multus-ztfrw"
Sep 18 13:44:14.148439 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.145700    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f5d1bf14-b322-49d0-9fc8-b3a9a0aca036-host-kubelet\") pod \"ovnkube-node-wbv96\" (UID: \"f5d1bf14-b322-49d0-9fc8-b3a9a0aca036\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbv96"
Sep 18 13:44:14.148439 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.145706    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e0b6f508-1744-4ab7-8e37-0f2f4df3a816-etc-systemd\") pod \"tuned-8k94g\" (UID: \"e0b6f508-1744-4ab7-8e37-0f2f4df3a816\") " pod="openshift-cluster-node-tuning-operator/tuned-8k94g"
Sep 18 13:44:14.148439 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.145755    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e0b6f508-1744-4ab7-8e37-0f2f4df3a816-run\") pod \"tuned-8k94g\" (UID: \"e0b6f508-1744-4ab7-8e37-0f2f4df3a816\") " pod="openshift-cluster-node-tuning-operator/tuned-8k94g"
Sep 18 13:44:14.148439 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:14.145793    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:44:14.148439 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.145799    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b4829a7a-35e0-44e4-b738-4ba4bd403929-multus-cni-dir\") pod \"multus-ztfrw\" (UID: \"b4829a7a-35e0-44e4-b738-4ba4bd403929\") " pod="openshift-multus/multus-ztfrw"
Sep 18 13:44:14.148439 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:14.145857    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87eba103-cfec-4000-a10c-fd4fd512ac90-metrics-certs podName:87eba103-cfec-4000-a10c-fd4fd512ac90 nodeName:}" failed. No retries permitted until 2026-09-18 13:44:14.645838692 +0000 UTC m=+3.698498014 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/87eba103-cfec-4000-a10c-fd4fd512ac90-metrics-certs") pod "network-metrics-daemon-nb9bg" (UID: "87eba103-cfec-4000-a10c-fd4fd512ac90") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:44:14.148439 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.145855    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/802e0a84-ff96-44d6-9aa2-db34555f46eb-dbus\") pod \"global-pull-secret-syncer-bdwmn\" (UID: \"802e0a84-ff96-44d6-9aa2-db34555f46eb\") " pod="kube-system/global-pull-secret-syncer-bdwmn"
Sep 18 13:44:14.148439 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.145888    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f5d1bf14-b322-49d0-9fc8-b3a9a0aca036-host-run-ovn-kubernetes\") pod \"ovnkube-node-wbv96\" (UID: \"f5d1bf14-b322-49d0-9fc8-b3a9a0aca036\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbv96"
Sep 18 13:44:14.148439 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.145883    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f5d1bf14-b322-49d0-9fc8-b3a9a0aca036-run-systemd\") pod \"ovnkube-node-wbv96\" (UID: \"f5d1bf14-b322-49d0-9fc8-b3a9a0aca036\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbv96"
Sep 18 13:44:14.148439 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.145970    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f5d1bf14-b322-49d0-9fc8-b3a9a0aca036-ovnkube-config\") pod \"ovnkube-node-wbv96\" (UID: \"f5d1bf14-b322-49d0-9fc8-b3a9a0aca036\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbv96"
Sep 18 13:44:14.148439 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.147355    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0b6f508-1744-4ab7-8e37-0f2f4df3a816-tmp\") pod \"tuned-8k94g\" (UID: \"e0b6f508-1744-4ab7-8e37-0f2f4df3a816\") " pod="openshift-cluster-node-tuning-operator/tuned-8k94g"
Sep 18 13:44:14.148439 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.147730    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f5d1bf14-b322-49d0-9fc8-b3a9a0aca036-ovn-node-metrics-cert\") pod \"ovnkube-node-wbv96\" (UID: \"f5d1bf14-b322-49d0-9fc8-b3a9a0aca036\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbv96"
Sep 18 13:44:14.148439 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.147960    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e0b6f508-1744-4ab7-8e37-0f2f4df3a816-etc-tuned\") pod \"tuned-8k94g\" (UID: \"e0b6f508-1744-4ab7-8e37-0f2f4df3a816\") " pod="openshift-cluster-node-tuning-operator/tuned-8k94g"
Sep 18 13:44:14.148439 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.148122    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8902efdf-cf1d-4513-bfbb-cdf2b9a2b922-agent-certs\") pod \"konnectivity-agent-7cm48\" (UID: \"8902efdf-cf1d-4513-bfbb-cdf2b9a2b922\") " pod="kube-system/konnectivity-agent-7cm48"
Sep 18 13:44:14.153837 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.153807    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-t8cvj"
Sep 18 13:44:14.157755 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.157737    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cm5q\" (UniqueName: \"kubernetes.io/projected/2e756a4b-a7f8-4688-9a3a-efecba8641d2-kube-api-access-6cm5q\") pod \"iptables-alerter-vc9jp\" (UID: \"2e756a4b-a7f8-4688-9a3a-efecba8641d2\") " pod="openshift-network-operator/iptables-alerter-vc9jp"
Sep 18 13:44:14.171681 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.171656    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9txp\" (UniqueName: \"kubernetes.io/projected/b4829a7a-35e0-44e4-b738-4ba4bd403929-kube-api-access-s9txp\") pod \"multus-ztfrw\" (UID: \"b4829a7a-35e0-44e4-b738-4ba4bd403929\") " pod="openshift-multus/multus-ztfrw"
Sep 18 13:44:14.173374 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.173358    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrxhq\" (UniqueName: \"kubernetes.io/projected/e0b6f508-1744-4ab7-8e37-0f2f4df3a816-kube-api-access-lrxhq\") pod \"tuned-8k94g\" (UID: \"e0b6f508-1744-4ab7-8e37-0f2f4df3a816\") " pod="openshift-cluster-node-tuning-operator/tuned-8k94g"
Sep 18 13:44:14.175722 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.175699    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wqns\" (UniqueName: \"kubernetes.io/projected/66b62a87-76ae-41e9-8e0c-dd48ff323e5d-kube-api-access-5wqns\") pod \"aws-ebs-csi-driver-node-nscch\" (UID: \"66b62a87-76ae-41e9-8e0c-dd48ff323e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nscch"
Sep 18 13:44:14.176786 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.176764    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtkdz\" (UniqueName: \"kubernetes.io/projected/47923c57-2f7a-46f5-b8c4-10efd987fa8b-kube-api-access-mtkdz\") pod \"multus-additional-cni-plugins-szg9k\" (UID: \"47923c57-2f7a-46f5-b8c4-10efd987fa8b\") " pod="openshift-multus/multus-additional-cni-plugins-szg9k"
Sep 18 13:44:14.178494 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.178475    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnjtp\" (UniqueName: \"kubernetes.io/projected/87eba103-cfec-4000-a10c-fd4fd512ac90-kube-api-access-xnjtp\") pod \"network-metrics-daemon-nb9bg\" (UID: \"87eba103-cfec-4000-a10c-fd4fd512ac90\") " pod="openshift-multus/network-metrics-daemon-nb9bg"
Sep 18 13:44:14.178965 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:14.178947    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 13:44:14.178965 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:14.178967    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 13:44:14.179119 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:14.178978    2577 projected.go:196] Error preparing data for projected volume kube-api-access-dmqn6 for pod openshift-network-diagnostics/network-check-target-hj8lg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:44:14.179119 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:14.179039    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f382898-0370-425b-8c01-6587a985e3f0-kube-api-access-dmqn6 podName:6f382898-0370-425b-8c01-6587a985e3f0 nodeName:}" failed. No retries permitted until 2026-09-18 13:44:14.679026086 +0000 UTC m=+3.731685393 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-dmqn6" (UniqueName: "kubernetes.io/projected/6f382898-0370-425b-8c01-6587a985e3f0-kube-api-access-dmqn6") pod "network-check-target-hj8lg" (UID: "6f382898-0370-425b-8c01-6587a985e3f0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:44:14.179956 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.179934    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h4d5\" (UniqueName: \"kubernetes.io/projected/dbd84791-daa9-43f0-9f32-8f4415eceeca-kube-api-access-7h4d5\") pod \"node-resolver-vjv4c\" (UID: \"dbd84791-daa9-43f0-9f32-8f4415eceeca\") " pod="openshift-dns/node-resolver-vjv4c"
Sep 18 13:44:14.181336 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.181315    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdv8r\" (UniqueName: \"kubernetes.io/projected/f5d1bf14-b322-49d0-9fc8-b3a9a0aca036-kube-api-access-kdv8r\") pod \"ovnkube-node-wbv96\" (UID: \"f5d1bf14-b322-49d0-9fc8-b3a9a0aca036\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbv96"
Sep 18 13:44:14.182832 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.182815    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b4zq\" (UniqueName: \"kubernetes.io/projected/ac788d27-88a8-4ed2-9363-fe4f0aa51013-kube-api-access-9b4zq\") pod \"node-ca-wczf6\" (UID: \"ac788d27-88a8-4ed2-9363-fe4f0aa51013\") " pod="openshift-image-registry/node-ca-wczf6"
Sep 18 13:44:14.320809 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.320778    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ztfrw"
Sep 18 13:44:14.325936 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.325915    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wbv96"
Sep 18 13:44:14.359355 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.359299    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7cm48"
Sep 18 13:44:14.378484 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.378455    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vjv4c"
Sep 18 13:44:14.397160 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.397138    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vc9jp"
Sep 18 13:44:14.410347 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.410326    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nscch"
Sep 18 13:44:14.412107 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:14.412079    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbd84791_daa9_43f0_9f32_8f4415eceeca.slice/crio-03569325bac276b4855e51e020a83c5943f155524dda4dd02bda93854ea7eb9c WatchSource:0}: Error finding container 03569325bac276b4855e51e020a83c5943f155524dda4dd02bda93854ea7eb9c: Status 404 returned error can't find the container with id 03569325bac276b4855e51e020a83c5943f155524dda4dd02bda93854ea7eb9c
Sep 18 13:44:14.412733 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:14.412708    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a4b0c9f3c3ac36b289317453f80d9d6.slice/crio-d4326a2d550267f65179c675e83785f0826ce890faea35e1feeb5325aada40cd WatchSource:0}: Error finding container d4326a2d550267f65179c675e83785f0826ce890faea35e1feeb5325aada40cd: Status 404 returned error can't find the container with id d4326a2d550267f65179c675e83785f0826ce890faea35e1feeb5325aada40cd
Sep 18 13:44:14.413312 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:14.413286    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8902efdf_cf1d_4513_bfbb_cdf2b9a2b922.slice/crio-7d7b3583cef2929285b7d19a82f93d479f11fea8151fc05aaa1d2cdd0f557e1c WatchSource:0}: Error finding container 7d7b3583cef2929285b7d19a82f93d479f11fea8151fc05aaa1d2cdd0f557e1c: Status 404 returned error can't find the container with id 7d7b3583cef2929285b7d19a82f93d479f11fea8151fc05aaa1d2cdd0f557e1c
Sep 18 13:44:14.413991 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:14.413967    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e756a4b_a7f8_4688_9a3a_efecba8641d2.slice/crio-76f5b5e269f612e3f1d4c54fe200c7c418422dc87dcf6343b0ed6587eb8cf033 WatchSource:0}: Error finding container 76f5b5e269f612e3f1d4c54fe200c7c418422dc87dcf6343b0ed6587eb8cf033: Status 404 returned error can't find the container with id 76f5b5e269f612e3f1d4c54fe200c7c418422dc87dcf6343b0ed6587eb8cf033
Sep 18 13:44:14.414928 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:14.414887    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00bbb25ba7cee607198e2c37fb5d2b3f.slice/crio-0346935664d66da823a643635e700ad82629bf508b2cd6ee738bf4ef26b4ac09 WatchSource:0}: Error finding container 0346935664d66da823a643635e700ad82629bf508b2cd6ee738bf4ef26b4ac09: Status 404 returned error can't find the container with id 0346935664d66da823a643635e700ad82629bf508b2cd6ee738bf4ef26b4ac09
Sep 18 13:44:14.418221 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.418198    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 13:44:14.421250 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:14.421209    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66b62a87_76ae_41e9_8e0c_dd48ff323e5d.slice/crio-7f2febee20069e009f6fb4330acd2bf2d883367a3b3396fa225795cca23401ee WatchSource:0}: Error finding container 7f2febee20069e009f6fb4330acd2bf2d883367a3b3396fa225795cca23401ee: Status 404 returned error can't find the container with id 7f2febee20069e009f6fb4330acd2bf2d883367a3b3396fa225795cca23401ee
Sep 18 13:44:14.425014 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.424994    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8k94g"
Sep 18 13:44:14.431441 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.431422    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wczf6"
Sep 18 13:44:14.435281 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:14.435250    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0b6f508_1744_4ab7_8e37_0f2f4df3a816.slice/crio-b57caa9b37e6c860c90740ae5675140db538acc6e15f958f69b2970aaa1d3177 WatchSource:0}: Error finding container b57caa9b37e6c860c90740ae5675140db538acc6e15f958f69b2970aaa1d3177: Status 404 returned error can't find the container with id b57caa9b37e6c860c90740ae5675140db538acc6e15f958f69b2970aaa1d3177
Sep 18 13:44:14.440590 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:14.440569    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac788d27_88a8_4ed2_9363_fe4f0aa51013.slice/crio-289165edc749d9c30b48f7eae0f3cbdab1cef8adf585d249c2ed09a1cf77abe9 WatchSource:0}: Error finding container 289165edc749d9c30b48f7eae0f3cbdab1cef8adf585d249c2ed09a1cf77abe9: Status 404 returned error can't find the container with id 289165edc749d9c30b48f7eae0f3cbdab1cef8adf585d249c2ed09a1cf77abe9
Sep 18 13:44:14.448507 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.448485    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-szg9k"
Sep 18 13:44:14.460212 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:14.460190    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47923c57_2f7a_46f5_b8c4_10efd987fa8b.slice/crio-8acd0d1ff4e9395af5424f0190d1a922b87441187e7b122f252f59e04ff53e4a WatchSource:0}: Error finding container 8acd0d1ff4e9395af5424f0190d1a922b87441187e7b122f252f59e04ff53e4a: Status 404 returned error can't find the container with id 8acd0d1ff4e9395af5424f0190d1a922b87441187e7b122f252f59e04ff53e4a
Sep 18 13:44:14.485327 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.485306    2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 13:44:14.651144 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.651062    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87eba103-cfec-4000-a10c-fd4fd512ac90-metrics-certs\") pod \"network-metrics-daemon-nb9bg\" (UID: \"87eba103-cfec-4000-a10c-fd4fd512ac90\") " pod="openshift-multus/network-metrics-daemon-nb9bg"
Sep 18 13:44:14.651144 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.651108    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/802e0a84-ff96-44d6-9aa2-db34555f46eb-original-pull-secret\") pod \"global-pull-secret-syncer-bdwmn\" (UID: \"802e0a84-ff96-44d6-9aa2-db34555f46eb\") " pod="kube-system/global-pull-secret-syncer-bdwmn"
Sep 18 13:44:14.651321 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:14.651202    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 13:44:14.651321 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:14.651203    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:44:14.651321 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:14.651260    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/802e0a84-ff96-44d6-9aa2-db34555f46eb-original-pull-secret podName:802e0a84-ff96-44d6-9aa2-db34555f46eb nodeName:}" failed. No retries permitted until 2026-09-18 13:44:15.651244733 +0000 UTC m=+4.703904039 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/802e0a84-ff96-44d6-9aa2-db34555f46eb-original-pull-secret") pod "global-pull-secret-syncer-bdwmn" (UID: "802e0a84-ff96-44d6-9aa2-db34555f46eb") : object "kube-system"/"original-pull-secret" not registered
Sep 18 13:44:14.651321 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:14.651277    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87eba103-cfec-4000-a10c-fd4fd512ac90-metrics-certs podName:87eba103-cfec-4000-a10c-fd4fd512ac90 nodeName:}" failed. No retries permitted until 2026-09-18 13:44:15.651269756 +0000 UTC m=+4.703929062 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/87eba103-cfec-4000-a10c-fd4fd512ac90-metrics-certs") pod "network-metrics-daemon-nb9bg" (UID: "87eba103-cfec-4000-a10c-fd4fd512ac90") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:44:14.732475 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:14.732442    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4829a7a_35e0_44e4_b738_4ba4bd403929.slice/crio-a67c8846164946687bb3b0f6de64ef9a6ef86559028215978db7df8396d8a717 WatchSource:0}: Error finding container a67c8846164946687bb3b0f6de64ef9a6ef86559028215978db7df8396d8a717: Status 404 returned error can't find the container with id a67c8846164946687bb3b0f6de64ef9a6ef86559028215978db7df8396d8a717
Sep 18 13:44:14.751626 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:14.751598    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmqn6\" (UniqueName: \"kubernetes.io/projected/6f382898-0370-425b-8c01-6587a985e3f0-kube-api-access-dmqn6\") pod \"network-check-target-hj8lg\" (UID: \"6f382898-0370-425b-8c01-6587a985e3f0\") " pod="openshift-network-diagnostics/network-check-target-hj8lg"
Sep 18 13:44:14.751723 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:14.751714    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 13:44:14.751771 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:14.751729    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 13:44:14.751771 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:14.751738    2577 projected.go:196] Error preparing data for projected volume kube-api-access-dmqn6 for pod openshift-network-diagnostics/network-check-target-hj8lg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:44:14.751842 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:14.751784    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f382898-0370-425b-8c01-6587a985e3f0-kube-api-access-dmqn6 podName:6f382898-0370-425b-8c01-6587a985e3f0 nodeName:}" failed. No retries permitted until 2026-09-18 13:44:15.751771406 +0000 UTC m=+4.804430717 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-dmqn6" (UniqueName: "kubernetes.io/projected/6f382898-0370-425b-8c01-6587a985e3f0-kube-api-access-dmqn6") pod "network-check-target-hj8lg" (UID: "6f382898-0370-425b-8c01-6587a985e3f0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:44:14.847137 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:14.847101    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5d1bf14_b322_49d0_9fc8_b3a9a0aca036.slice/crio-4e8ed5093e1b903f5247950e4d3ff0a72590cba7c9bfca9d9c789fe83c4e4705 WatchSource:0}: Error finding container 4e8ed5093e1b903f5247950e4d3ff0a72590cba7c9bfca9d9c789fe83c4e4705: Status 404 returned error can't find the container with id 4e8ed5093e1b903f5247950e4d3ff0a72590cba7c9bfca9d9c789fe83c4e4705
Sep 18 13:44:15.109388 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:15.109280    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vjv4c" event={"ID":"dbd84791-daa9-43f0-9f32-8f4415eceeca","Type":"ContainerStarted","Data":"03569325bac276b4855e51e020a83c5943f155524dda4dd02bda93854ea7eb9c"}
Sep 18 13:44:15.132402 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:15.132289    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-222.ec2.internal" event={"ID":"4a4b0c9f3c3ac36b289317453f80d9d6","Type":"ContainerStarted","Data":"d4326a2d550267f65179c675e83785f0826ce890faea35e1feeb5325aada40cd"}
Sep 18 13:44:15.134957 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:15.134928    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ztfrw" event={"ID":"b4829a7a-35e0-44e4-b738-4ba4bd403929","Type":"ContainerStarted","Data":"a67c8846164946687bb3b0f6de64ef9a6ef86559028215978db7df8396d8a717"}
Sep 18 13:44:15.138828 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:15.138800    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8k94g" event={"ID":"e0b6f508-1744-4ab7-8e37-0f2f4df3a816","Type":"ContainerStarted","Data":"b57caa9b37e6c860c90740ae5675140db538acc6e15f958f69b2970aaa1d3177"}
Sep 18 13:44:15.142094 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:15.142067    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7cm48" event={"ID":"8902efdf-cf1d-4513-bfbb-cdf2b9a2b922","Type":"ContainerStarted","Data":"7d7b3583cef2929285b7d19a82f93d479f11fea8151fc05aaa1d2cdd0f557e1c"}
Sep 18 13:44:15.144998 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:15.144969    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-222.ec2.internal" event={"ID":"00bbb25ba7cee607198e2c37fb5d2b3f","Type":"ContainerStarted","Data":"0346935664d66da823a643635e700ad82629bf508b2cd6ee738bf4ef26b4ac09"}
Sep 18 13:44:15.152177 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:15.152124    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wbv96" event={"ID":"f5d1bf14-b322-49d0-9fc8-b3a9a0aca036","Type":"ContainerStarted","Data":"4e8ed5093e1b903f5247950e4d3ff0a72590cba7c9bfca9d9c789fe83c4e4705"}
Sep 18 13:44:15.158459 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:15.158380    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 13:39:14 +0000 UTC" deadline="2028-07-02 00:02:20.835092318 +0000 UTC"
Sep 18 13:44:15.158459 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:15.158412    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15658h18m5.676685445s"
Sep 18 13:44:15.161992 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:15.161935    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-szg9k" event={"ID":"47923c57-2f7a-46f5-b8c4-10efd987fa8b","Type":"ContainerStarted","Data":"8acd0d1ff4e9395af5424f0190d1a922b87441187e7b122f252f59e04ff53e4a"}
Sep 18 13:44:15.166793 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:15.166765    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wczf6" event={"ID":"ac788d27-88a8-4ed2-9363-fe4f0aa51013","Type":"ContainerStarted","Data":"289165edc749d9c30b48f7eae0f3cbdab1cef8adf585d249c2ed09a1cf77abe9"}
Sep 18 13:44:15.170742 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:15.170659    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nscch" event={"ID":"66b62a87-76ae-41e9-8e0c-dd48ff323e5d","Type":"ContainerStarted","Data":"7f2febee20069e009f6fb4330acd2bf2d883367a3b3396fa225795cca23401ee"}
Sep 18 13:44:15.191572 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:15.191536    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vc9jp" event={"ID":"2e756a4b-a7f8-4688-9a3a-efecba8641d2","Type":"ContainerStarted","Data":"76f5b5e269f612e3f1d4c54fe200c7c418422dc87dcf6343b0ed6587eb8cf033"}
Sep 18 13:44:15.256760 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:15.256726    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 13:44:15.659345 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:15.659080    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87eba103-cfec-4000-a10c-fd4fd512ac90-metrics-certs\") pod \"network-metrics-daemon-nb9bg\" (UID: \"87eba103-cfec-4000-a10c-fd4fd512ac90\") " pod="openshift-multus/network-metrics-daemon-nb9bg"
Sep 18 13:44:15.659345 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:15.659151    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/802e0a84-ff96-44d6-9aa2-db34555f46eb-original-pull-secret\") pod \"global-pull-secret-syncer-bdwmn\" (UID: \"802e0a84-ff96-44d6-9aa2-db34555f46eb\") " pod="kube-system/global-pull-secret-syncer-bdwmn"
Sep 18 13:44:15.659345 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:15.659291    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 13:44:15.659345 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:15.659357    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/802e0a84-ff96-44d6-9aa2-db34555f46eb-original-pull-secret podName:802e0a84-ff96-44d6-9aa2-db34555f46eb nodeName:}" failed. No retries permitted until 2026-09-18 13:44:17.659338454 +0000 UTC m=+6.711997776 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/802e0a84-ff96-44d6-9aa2-db34555f46eb-original-pull-secret") pod "global-pull-secret-syncer-bdwmn" (UID: "802e0a84-ff96-44d6-9aa2-db34555f46eb") : object "kube-system"/"original-pull-secret" not registered
Sep 18 13:44:15.659791 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:15.659766    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:44:15.659861 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:15.659821    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87eba103-cfec-4000-a10c-fd4fd512ac90-metrics-certs podName:87eba103-cfec-4000-a10c-fd4fd512ac90 nodeName:}" failed. No retries permitted until 2026-09-18 13:44:17.659804431 +0000 UTC m=+6.712463758 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/87eba103-cfec-4000-a10c-fd4fd512ac90-metrics-certs") pod "network-metrics-daemon-nb9bg" (UID: "87eba103-cfec-4000-a10c-fd4fd512ac90") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:44:15.761957 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:15.761907    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmqn6\" (UniqueName: \"kubernetes.io/projected/6f382898-0370-425b-8c01-6587a985e3f0-kube-api-access-dmqn6\") pod \"network-check-target-hj8lg\" (UID: \"6f382898-0370-425b-8c01-6587a985e3f0\") " pod="openshift-network-diagnostics/network-check-target-hj8lg"
Sep 18 13:44:15.762156 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:15.762097    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 13:44:15.762156 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:15.762118    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 13:44:15.762156 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:15.762131    2577 projected.go:196] Error preparing data for projected volume kube-api-access-dmqn6 for pod openshift-network-diagnostics/network-check-target-hj8lg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:44:15.762342 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:15.762199    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f382898-0370-425b-8c01-6587a985e3f0-kube-api-access-dmqn6 podName:6f382898-0370-425b-8c01-6587a985e3f0 nodeName:}" failed. No retries permitted until 2026-09-18 13:44:17.762178929 +0000 UTC m=+6.814838259 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-dmqn6" (UniqueName: "kubernetes.io/projected/6f382898-0370-425b-8c01-6587a985e3f0-kube-api-access-dmqn6") pod "network-check-target-hj8lg" (UID: "6f382898-0370-425b-8c01-6587a985e3f0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:44:16.101221 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:16.101182    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdwmn"
Sep 18 13:44:16.101437 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:16.101330    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdwmn" podUID="802e0a84-ff96-44d6-9aa2-db34555f46eb"
Sep 18 13:44:16.101818 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:16.101772    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hj8lg"
Sep 18 13:44:16.101940 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:16.101876    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hj8lg" podUID="6f382898-0370-425b-8c01-6587a985e3f0"
Sep 18 13:44:16.102004 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:16.101965    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nb9bg"
Sep 18 13:44:16.102068 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:16.102054    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nb9bg" podUID="87eba103-cfec-4000-a10c-fd4fd512ac90"
Sep 18 13:44:16.158805 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:16.158761    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 13:39:14 +0000 UTC" deadline="2028-06-01 16:41:19.725588708 +0000 UTC"
Sep 18 13:44:16.158805 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:16.158804    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14930h57m3.566789887s"
Sep 18 13:44:17.679734 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:17.679697    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87eba103-cfec-4000-a10c-fd4fd512ac90-metrics-certs\") pod \"network-metrics-daemon-nb9bg\" (UID: \"87eba103-cfec-4000-a10c-fd4fd512ac90\") " pod="openshift-multus/network-metrics-daemon-nb9bg"
Sep 18 13:44:17.680336 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:17.679758    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/802e0a84-ff96-44d6-9aa2-db34555f46eb-original-pull-secret\") pod \"global-pull-secret-syncer-bdwmn\" (UID: \"802e0a84-ff96-44d6-9aa2-db34555f46eb\") " pod="kube-system/global-pull-secret-syncer-bdwmn"
Sep 18 13:44:17.680336 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:17.679928    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 13:44:17.680336 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:17.679994    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/802e0a84-ff96-44d6-9aa2-db34555f46eb-original-pull-secret podName:802e0a84-ff96-44d6-9aa2-db34555f46eb nodeName:}" failed. No retries permitted until 2026-09-18 13:44:21.679975607 +0000 UTC m=+10.732634935 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/802e0a84-ff96-44d6-9aa2-db34555f46eb-original-pull-secret") pod "global-pull-secret-syncer-bdwmn" (UID: "802e0a84-ff96-44d6-9aa2-db34555f46eb") : object "kube-system"/"original-pull-secret" not registered
Sep 18 13:44:17.680516 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:17.680392    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:44:17.680516 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:17.680441    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87eba103-cfec-4000-a10c-fd4fd512ac90-metrics-certs podName:87eba103-cfec-4000-a10c-fd4fd512ac90 nodeName:}" failed. No retries permitted until 2026-09-18 13:44:21.680424741 +0000 UTC m=+10.733084052 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/87eba103-cfec-4000-a10c-fd4fd512ac90-metrics-certs") pod "network-metrics-daemon-nb9bg" (UID: "87eba103-cfec-4000-a10c-fd4fd512ac90") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:44:17.780146 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:17.780108    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmqn6\" (UniqueName: \"kubernetes.io/projected/6f382898-0370-425b-8c01-6587a985e3f0-kube-api-access-dmqn6\") pod \"network-check-target-hj8lg\" (UID: \"6f382898-0370-425b-8c01-6587a985e3f0\") " pod="openshift-network-diagnostics/network-check-target-hj8lg"
Sep 18 13:44:17.780313 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:17.780301    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 13:44:17.780386 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:17.780320    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 13:44:17.780386 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:17.780333    2577 projected.go:196] Error preparing data for projected volume kube-api-access-dmqn6 for pod openshift-network-diagnostics/network-check-target-hj8lg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:44:17.780496 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:17.780388    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f382898-0370-425b-8c01-6587a985e3f0-kube-api-access-dmqn6 podName:6f382898-0370-425b-8c01-6587a985e3f0 nodeName:}" failed. No retries permitted until 2026-09-18 13:44:21.780370766 +0000 UTC m=+10.833030076 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-dmqn6" (UniqueName: "kubernetes.io/projected/6f382898-0370-425b-8c01-6587a985e3f0-kube-api-access-dmqn6") pod "network-check-target-hj8lg" (UID: "6f382898-0370-425b-8c01-6587a985e3f0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:44:18.101651 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:18.101622    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nb9bg"
Sep 18 13:44:18.101857 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:18.101767    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nb9bg" podUID="87eba103-cfec-4000-a10c-fd4fd512ac90"
Sep 18 13:44:18.102197 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:18.102175    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdwmn"
Sep 18 13:44:18.102293 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:18.102268    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdwmn" podUID="802e0a84-ff96-44d6-9aa2-db34555f46eb"
Sep 18 13:44:18.102350 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:18.102308    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hj8lg"
Sep 18 13:44:18.102404 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:18.102378    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hj8lg" podUID="6f382898-0370-425b-8c01-6587a985e3f0"
Sep 18 13:44:20.101785 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:20.101739    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nb9bg"
Sep 18 13:44:20.102260 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:20.101740    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdwmn"
Sep 18 13:44:20.102260 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:20.101739    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hj8lg"
Sep 18 13:44:20.102260 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:20.101955    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdwmn" podUID="802e0a84-ff96-44d6-9aa2-db34555f46eb"
Sep 18 13:44:20.102260 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:20.102042    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hj8lg" podUID="6f382898-0370-425b-8c01-6587a985e3f0"
Sep 18 13:44:20.102260 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:20.101906    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nb9bg" podUID="87eba103-cfec-4000-a10c-fd4fd512ac90"
Sep 18 13:44:21.714365 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:21.714320    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87eba103-cfec-4000-a10c-fd4fd512ac90-metrics-certs\") pod \"network-metrics-daemon-nb9bg\" (UID: \"87eba103-cfec-4000-a10c-fd4fd512ac90\") " pod="openshift-multus/network-metrics-daemon-nb9bg"
Sep 18 13:44:21.714903 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:21.714391    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/802e0a84-ff96-44d6-9aa2-db34555f46eb-original-pull-secret\") pod \"global-pull-secret-syncer-bdwmn\" (UID: \"802e0a84-ff96-44d6-9aa2-db34555f46eb\") " pod="kube-system/global-pull-secret-syncer-bdwmn"
Sep 18 13:44:21.714903 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:21.714519    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 13:44:21.714903 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:21.714586    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/802e0a84-ff96-44d6-9aa2-db34555f46eb-original-pull-secret podName:802e0a84-ff96-44d6-9aa2-db34555f46eb nodeName:}" failed. No retries permitted until 2026-09-18 13:44:29.714568783 +0000 UTC m=+18.767228092 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/802e0a84-ff96-44d6-9aa2-db34555f46eb-original-pull-secret") pod "global-pull-secret-syncer-bdwmn" (UID: "802e0a84-ff96-44d6-9aa2-db34555f46eb") : object "kube-system"/"original-pull-secret" not registered
Sep 18 13:44:21.714903 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:21.714603    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:44:21.714903 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:21.714654    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87eba103-cfec-4000-a10c-fd4fd512ac90-metrics-certs podName:87eba103-cfec-4000-a10c-fd4fd512ac90 nodeName:}" failed. No retries permitted until 2026-09-18 13:44:29.714630813 +0000 UTC m=+18.767290133 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/87eba103-cfec-4000-a10c-fd4fd512ac90-metrics-certs") pod "network-metrics-daemon-nb9bg" (UID: "87eba103-cfec-4000-a10c-fd4fd512ac90") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:44:21.815765 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:21.815727    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmqn6\" (UniqueName: \"kubernetes.io/projected/6f382898-0370-425b-8c01-6587a985e3f0-kube-api-access-dmqn6\") pod \"network-check-target-hj8lg\" (UID: \"6f382898-0370-425b-8c01-6587a985e3f0\") " pod="openshift-network-diagnostics/network-check-target-hj8lg"
Sep 18 13:44:21.815956 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:21.815920    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 13:44:21.815956 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:21.815949    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 13:44:21.816037 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:21.815963    2577 projected.go:196] Error preparing data for projected volume kube-api-access-dmqn6 for pod openshift-network-diagnostics/network-check-target-hj8lg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:44:21.816037 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:21.816033    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f382898-0370-425b-8c01-6587a985e3f0-kube-api-access-dmqn6 podName:6f382898-0370-425b-8c01-6587a985e3f0 nodeName:}" failed. No retries permitted until 2026-09-18 13:44:29.81601469 +0000 UTC m=+18.868674016 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-dmqn6" (UniqueName: "kubernetes.io/projected/6f382898-0370-425b-8c01-6587a985e3f0-kube-api-access-dmqn6") pod "network-check-target-hj8lg" (UID: "6f382898-0370-425b-8c01-6587a985e3f0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:44:22.101645 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:22.101400    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hj8lg"
Sep 18 13:44:22.101645 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:22.101400    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nb9bg"
Sep 18 13:44:22.101645 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:22.101514    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hj8lg" podUID="6f382898-0370-425b-8c01-6587a985e3f0"
Sep 18 13:44:22.101645 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:22.101633    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nb9bg" podUID="87eba103-cfec-4000-a10c-fd4fd512ac90"
Sep 18 13:44:22.101645 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:22.101406    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdwmn"
Sep 18 13:44:22.102041 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:22.101729    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdwmn" podUID="802e0a84-ff96-44d6-9aa2-db34555f46eb"
Sep 18 13:44:24.101072 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:24.101040    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nb9bg"
Sep 18 13:44:24.101072 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:24.101068    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdwmn"
Sep 18 13:44:24.101590 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:24.101185    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hj8lg"
Sep 18 13:44:24.101590 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:24.101195    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nb9bg" podUID="87eba103-cfec-4000-a10c-fd4fd512ac90"
Sep 18 13:44:24.101590 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:24.101263    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdwmn" podUID="802e0a84-ff96-44d6-9aa2-db34555f46eb"
Sep 18 13:44:24.101590 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:24.101333    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hj8lg" podUID="6f382898-0370-425b-8c01-6587a985e3f0"
Sep 18 13:44:26.101185 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:26.101148    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hj8lg"
Sep 18 13:44:26.101185 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:26.101187    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nb9bg"
Sep 18 13:44:26.101693 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:26.101187    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdwmn"
Sep 18 13:44:26.101693 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:26.101288    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hj8lg" podUID="6f382898-0370-425b-8c01-6587a985e3f0"
Sep 18 13:44:26.101693 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:26.101396    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nb9bg" podUID="87eba103-cfec-4000-a10c-fd4fd512ac90"
Sep 18 13:44:26.101693 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:26.101482    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdwmn" podUID="802e0a84-ff96-44d6-9aa2-db34555f46eb"
Sep 18 13:44:28.101406 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:28.101376    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdwmn"
Sep 18 13:44:28.101406 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:28.101404    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nb9bg"
Sep 18 13:44:28.101983 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:28.101385    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hj8lg"
Sep 18 13:44:28.101983 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:28.101494    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdwmn" podUID="802e0a84-ff96-44d6-9aa2-db34555f46eb"
Sep 18 13:44:28.101983 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:28.101575    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hj8lg" podUID="6f382898-0370-425b-8c01-6587a985e3f0"
Sep 18 13:44:28.101983 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:28.101662    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nb9bg" podUID="87eba103-cfec-4000-a10c-fd4fd512ac90"
Sep 18 13:44:29.771949 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:29.771910    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87eba103-cfec-4000-a10c-fd4fd512ac90-metrics-certs\") pod \"network-metrics-daemon-nb9bg\" (UID: \"87eba103-cfec-4000-a10c-fd4fd512ac90\") " pod="openshift-multus/network-metrics-daemon-nb9bg"
Sep 18 13:44:29.772266 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:29.771991    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/802e0a84-ff96-44d6-9aa2-db34555f46eb-original-pull-secret\") pod \"global-pull-secret-syncer-bdwmn\" (UID: \"802e0a84-ff96-44d6-9aa2-db34555f46eb\") " pod="kube-system/global-pull-secret-syncer-bdwmn"
Sep 18 13:44:29.772266 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:29.772105    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 13:44:29.772266 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:29.772119    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:44:29.772266 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:29.772168    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/802e0a84-ff96-44d6-9aa2-db34555f46eb-original-pull-secret podName:802e0a84-ff96-44d6-9aa2-db34555f46eb nodeName:}" failed. No retries permitted until 2026-09-18 13:44:45.772152048 +0000 UTC m=+34.824811369 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/802e0a84-ff96-44d6-9aa2-db34555f46eb-original-pull-secret") pod "global-pull-secret-syncer-bdwmn" (UID: "802e0a84-ff96-44d6-9aa2-db34555f46eb") : object "kube-system"/"original-pull-secret" not registered
Sep 18 13:44:29.772266 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:29.772183    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87eba103-cfec-4000-a10c-fd4fd512ac90-metrics-certs podName:87eba103-cfec-4000-a10c-fd4fd512ac90 nodeName:}" failed. No retries permitted until 2026-09-18 13:44:45.772176125 +0000 UTC m=+34.824835433 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/87eba103-cfec-4000-a10c-fd4fd512ac90-metrics-certs") pod "network-metrics-daemon-nb9bg" (UID: "87eba103-cfec-4000-a10c-fd4fd512ac90") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:44:29.872955 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:29.872925    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmqn6\" (UniqueName: \"kubernetes.io/projected/6f382898-0370-425b-8c01-6587a985e3f0-kube-api-access-dmqn6\") pod \"network-check-target-hj8lg\" (UID: \"6f382898-0370-425b-8c01-6587a985e3f0\") " pod="openshift-network-diagnostics/network-check-target-hj8lg"
Sep 18 13:44:29.873097 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:29.873054    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 13:44:29.873097 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:29.873071    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 13:44:29.873097 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:29.873080    2577 projected.go:196] Error preparing data for projected volume kube-api-access-dmqn6 for pod openshift-network-diagnostics/network-check-target-hj8lg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:44:29.873256 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:29.873127    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f382898-0370-425b-8c01-6587a985e3f0-kube-api-access-dmqn6 podName:6f382898-0370-425b-8c01-6587a985e3f0 nodeName:}" failed. No retries permitted until 2026-09-18 13:44:45.87311517 +0000 UTC m=+34.925774488 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-dmqn6" (UniqueName: "kubernetes.io/projected/6f382898-0370-425b-8c01-6587a985e3f0-kube-api-access-dmqn6") pod "network-check-target-hj8lg" (UID: "6f382898-0370-425b-8c01-6587a985e3f0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:44:30.101064 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:30.101035    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nb9bg"
Sep 18 13:44:30.101252 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:30.101035    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hj8lg"
Sep 18 13:44:30.101252 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:30.101035    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdwmn"
Sep 18 13:44:30.101252 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:30.101203    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nb9bg" podUID="87eba103-cfec-4000-a10c-fd4fd512ac90"
Sep 18 13:44:30.101420 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:30.101289    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdwmn" podUID="802e0a84-ff96-44d6-9aa2-db34555f46eb"
Sep 18 13:44:30.101420 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:30.101370    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hj8lg" podUID="6f382898-0370-425b-8c01-6587a985e3f0"
Sep 18 13:44:32.101340 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:32.101306    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nb9bg"
Sep 18 13:44:32.101340 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:32.101324    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hj8lg"
Sep 18 13:44:32.101925 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:32.101323    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdwmn"
Sep 18 13:44:32.101925 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:32.101426    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nb9bg" podUID="87eba103-cfec-4000-a10c-fd4fd512ac90"
Sep 18 13:44:32.101925 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:32.101486    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hj8lg" podUID="6f382898-0370-425b-8c01-6587a985e3f0"
Sep 18 13:44:32.101925 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:32.101584    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdwmn" podUID="802e0a84-ff96-44d6-9aa2-db34555f46eb"
Sep 18 13:44:34.101892 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:34.101555    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdwmn"
Sep 18 13:44:34.102540 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:34.101607    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hj8lg"
Sep 18 13:44:34.102540 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:34.101973    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdwmn" podUID="802e0a84-ff96-44d6-9aa2-db34555f46eb"
Sep 18 13:44:34.102540 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:34.101611    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nb9bg"
Sep 18 13:44:34.102540 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:34.102060    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hj8lg" podUID="6f382898-0370-425b-8c01-6587a985e3f0"
Sep 18 13:44:34.102540 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:34.102157    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nb9bg" podUID="87eba103-cfec-4000-a10c-fd4fd512ac90"
Sep 18 13:44:34.242467 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:34.242387    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wbv96" event={"ID":"f5d1bf14-b322-49d0-9fc8-b3a9a0aca036","Type":"ContainerStarted","Data":"c290a2be0a658d708bd0b2e327157c1f1568a84db818abf8b1ff61bb4273b355"}
Sep 18 13:44:34.242467 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:34.242457    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wbv96" event={"ID":"f5d1bf14-b322-49d0-9fc8-b3a9a0aca036","Type":"ContainerStarted","Data":"269e5b07db03b1a6cafd5411486012dbfdb16bcbb5dd3713d94a1dacbb5d4163"}
Sep 18 13:44:34.242658 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:34.242488    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wbv96" event={"ID":"f5d1bf14-b322-49d0-9fc8-b3a9a0aca036","Type":"ContainerStarted","Data":"7eac0a13036cba370a9bc3df648d8a6d7aa423d3747e1e413d1b9024edaf2da0"}
Sep 18 13:44:34.242658 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:34.242524    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wbv96" event={"ID":"f5d1bf14-b322-49d0-9fc8-b3a9a0aca036","Type":"ContainerStarted","Data":"4f688fd7ab1be1a38d408d44a02e4d415cd5553202ce318825dcfeb215b28960"}
Sep 18 13:44:34.242658 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:34.242554    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wbv96" event={"ID":"f5d1bf14-b322-49d0-9fc8-b3a9a0aca036","Type":"ContainerStarted","Data":"abc1769321e173e1fbad3fb3a950f7888d766abf9597c8cbde3c429c73f360b0"}
Sep 18 13:44:34.242658 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:34.242588    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wbv96" event={"ID":"f5d1bf14-b322-49d0-9fc8-b3a9a0aca036","Type":"ContainerStarted","Data":"dd76ddb6be8dc98c68bdddfcab9f950fd9549c2128ce06e4bf9c2f7fd93f7ac0"}
Sep 18 13:44:34.243955 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:34.243930    2577 generic.go:356] "Generic (PLEG): container finished" podID="47923c57-2f7a-46f5-b8c4-10efd987fa8b" containerID="72867d46ea83f8efd81b80e05b98a65b3a1b98b350dc539844d882be2fab55f2" exitCode=0
Sep 18 13:44:34.244069 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:34.244013    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-szg9k" event={"ID":"47923c57-2f7a-46f5-b8c4-10efd987fa8b","Type":"ContainerDied","Data":"72867d46ea83f8efd81b80e05b98a65b3a1b98b350dc539844d882be2fab55f2"}
Sep 18 13:44:34.245547 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:34.245524    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wczf6" event={"ID":"ac788d27-88a8-4ed2-9363-fe4f0aa51013","Type":"ContainerStarted","Data":"a8ad25f1fffb2595e4bbf00069c5a53baee9b79d42821b92121edffd23280ee6"}
Sep 18 13:44:34.247086 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:34.247062    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nscch" event={"ID":"66b62a87-76ae-41e9-8e0c-dd48ff323e5d","Type":"ContainerStarted","Data":"8fca2770a156b064c6a0e389c477ec2262b762425422c42d9dbbb2400839cf01"}
Sep 18 13:44:34.248481 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:34.248461    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vjv4c" event={"ID":"dbd84791-daa9-43f0-9f32-8f4415eceeca","Type":"ContainerStarted","Data":"321a6721f28df92b74fb0ce2d360a27fd1c19f83a67a5a41dc5bb1a871a09e36"}
Sep 18 13:44:34.250011 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:34.249984    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-222.ec2.internal" event={"ID":"4a4b0c9f3c3ac36b289317453f80d9d6","Type":"ContainerStarted","Data":"52e54ffb4a7f08769ef6b708ffb1d23e90b1de1e9ab1bc2f4845ad30b319a35c"}
Sep 18 13:44:34.251465 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:34.251444    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ztfrw" event={"ID":"b4829a7a-35e0-44e4-b738-4ba4bd403929","Type":"ContainerStarted","Data":"5f3b45e65e481eb9bd6c6499ac281a70e0924243d554a5310ccd8bcc2fd7d83c"}
Sep 18 13:44:34.252954 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:34.252935    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8k94g" event={"ID":"e0b6f508-1744-4ab7-8e37-0f2f4df3a816","Type":"ContainerStarted","Data":"b66cd9316c1acdce3d734c26b6b58781c7744a6a51fb88c7bbb2ea498624fd28"}
Sep 18 13:44:34.256339 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:34.256306    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7cm48" event={"ID":"8902efdf-cf1d-4513-bfbb-cdf2b9a2b922","Type":"ContainerStarted","Data":"375cf31e86f963293876de2187c5de68898fffc9b6d8d04145b34e5664ccc4ed"}
Sep 18 13:44:34.257878 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:34.257859    2577 generic.go:356] "Generic (PLEG): container finished" podID="00bbb25ba7cee607198e2c37fb5d2b3f" containerID="53c509831d6b8405c9773849b7070998e4b414e66c4d10b324980e2245415bcb" exitCode=0
Sep 18 13:44:34.257956 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:34.257888    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-222.ec2.internal" event={"ID":"00bbb25ba7cee607198e2c37fb5d2b3f","Type":"ContainerDied","Data":"53c509831d6b8405c9773849b7070998e4b414e66c4d10b324980e2245415bcb"}
Sep 18 13:44:34.290961 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:34.290923    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-wczf6" podStartSLOduration=2.541012053 podStartE2EDuration="21.290909745s" totalImagesPullingTime="18.749897692s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:44:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:44:34.290495478 +0000 UTC m=+23.343154804" watchObservedRunningTime="2026-09-18 13:44:34.290909745 +0000 UTC m=+23.343569072"
Sep 18 13:44:34.310922 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:34.310884    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-222.ec2.internal" podStartSLOduration=20.310871622 podStartE2EDuration="20.310871622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:44:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:44:34.310201319 +0000 UTC m=+23.362860647" watchObservedRunningTime="2026-09-18 13:44:34.310871622 +0000 UTC m=+23.363530949"
Sep 18 13:44:34.329586 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:34.329546    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-ztfrw" podStartSLOduration=2.8351260700000003 podStartE2EDuration="21.329532641s" totalImagesPullingTime="18.494406571s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:44:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:44:34.329300117 +0000 UTC m=+23.381959443" watchObservedRunningTime="2026-09-18 13:44:34.329532641 +0000 UTC m=+23.382191967"
Sep 18 13:44:34.374484 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:34.374439    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-7cm48" podStartSLOduration=2.565851547 podStartE2EDuration="21.374426619s" totalImagesPullingTime="18.808575072s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:44:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:44:34.351352312 +0000 UTC m=+23.404011639" watchObservedRunningTime="2026-09-18 13:44:34.374426619 +0000 UTC m=+23.427085946"
Sep 18 13:44:34.374608 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:34.374510    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-vjv4c" podStartSLOduration=2.6555176019999998 podStartE2EDuration="21.374506015s" totalImagesPullingTime="18.718988413s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:44:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:44:34.374087239 +0000 UTC m=+23.426746566" watchObservedRunningTime="2026-09-18 13:44:34.374506015 +0000 UTC m=+23.427165341"
Sep 18 13:44:34.394981 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:34.394946    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-8k94g" podStartSLOduration=2.601747398 podStartE2EDuration="21.394933484s" totalImagesPullingTime="18.793186086s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:44:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:44:34.394498928 +0000 UTC m=+23.447158249" watchObservedRunningTime="2026-09-18 13:44:34.394933484 +0000 UTC m=+23.447592811"
Sep 18 13:44:34.959635 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:34.959376    2577 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 18 13:44:35.104815 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:35.104717    2577 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-18T13:44:34.959581245Z","UUID":"f1457c08-85ef-474d-9add-8d591b6026ff","Handler":null,"Name":"","Endpoint":""}
Sep 18 13:44:35.106880 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:35.106861    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 18 13:44:35.106997 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:35.106891    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 18 13:44:35.263128 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:35.263056    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nscch" event={"ID":"66b62a87-76ae-41e9-8e0c-dd48ff323e5d","Type":"ContainerStarted","Data":"a724b5f87ece47432861203a0fe22f5340f7f74e86cc4ec3ad21bf98e91b2f8f"}
Sep 18 13:44:35.264867 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:35.264836    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vc9jp" event={"ID":"2e756a4b-a7f8-4688-9a3a-efecba8641d2","Type":"ContainerStarted","Data":"dce37289f7b436297a7785590d9b3f8ae521cd6b9c96bf73791be1153bbfe898"}
Sep 18 13:44:35.267075 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:35.267050    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-222.ec2.internal" event={"ID":"00bbb25ba7cee607198e2c37fb5d2b3f","Type":"ContainerStarted","Data":"92abf02b0cfa08721fb9c5f9248e5377cf4c53ea44594f246e46e498ab2c3494"}
Sep 18 13:44:35.279026 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:35.278982    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vc9jp" podStartSLOduration=3.459375763 podStartE2EDuration="22.278966187s" totalImagesPullingTime="18.819590424s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:44:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:44:35.278753848 +0000 UTC m=+24.331413174" watchObservedRunningTime="2026-09-18 13:44:35.278966187 +0000 UTC m=+24.331625516"
Sep 18 13:44:35.292583 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:35.292544    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-222.ec2.internal" podStartSLOduration=21.292530124 podStartE2EDuration="21.292530124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:44:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:44:35.292073871 +0000 UTC m=+24.344733197" watchObservedRunningTime="2026-09-18 13:44:35.292530124 +0000 UTC m=+24.345189454"
Sep 18 13:44:36.101910 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:36.101882    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdwmn"
Sep 18 13:44:36.102067 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:36.101882    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nb9bg"
Sep 18 13:44:36.102067 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:36.101988    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdwmn" podUID="802e0a84-ff96-44d6-9aa2-db34555f46eb"
Sep 18 13:44:36.102187 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:36.102099    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nb9bg" podUID="87eba103-cfec-4000-a10c-fd4fd512ac90"
Sep 18 13:44:36.102187 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:36.101882    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hj8lg"
Sep 18 13:44:36.102263 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:36.102213    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hj8lg" podUID="6f382898-0370-425b-8c01-6587a985e3f0"
Sep 18 13:44:36.274344 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:36.274305    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wbv96" event={"ID":"f5d1bf14-b322-49d0-9fc8-b3a9a0aca036","Type":"ContainerStarted","Data":"7f668c40ffe2ae325943abe0c26a0068dce2ba9149303e27ffe1a47cb505dff0"}
Sep 18 13:44:36.276844 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:36.276769    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nscch" event={"ID":"66b62a87-76ae-41e9-8e0c-dd48ff323e5d","Type":"ContainerStarted","Data":"ab4edbfe8b71479d1ffa1e67cc518e73873aa87953347d84ee25cb593e1a65e4"}
Sep 18 13:44:36.295598 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:36.293939    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nscch" podStartSLOduration=2.696759494 podStartE2EDuration="23.293921565s" totalImagesPullingTime="20.597162071s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:44:13 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:44:36.293485143 +0000 UTC m=+25.346144471" watchObservedRunningTime="2026-09-18 13:44:36.293921565 +0000 UTC m=+25.346580902"
Sep 18 13:44:36.792279 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:36.792244    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-7cm48"
Sep 18 13:44:36.792934 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:36.792917    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-7cm48"
Sep 18 13:44:37.279017 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:37.278988    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-7cm48"
Sep 18 13:44:37.279430 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:37.279288    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-7cm48"
Sep 18 13:44:38.101696 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:38.101664    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nb9bg"
Sep 18 13:44:38.101696 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:38.101693    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdwmn"
Sep 18 13:44:38.101988 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:38.101707    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hj8lg"
Sep 18 13:44:38.101988 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:38.101788    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nb9bg" podUID="87eba103-cfec-4000-a10c-fd4fd512ac90"
Sep 18 13:44:38.101988 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:38.101871    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdwmn" podUID="802e0a84-ff96-44d6-9aa2-db34555f46eb"
Sep 18 13:44:38.101988 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:38.101948    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hj8lg" podUID="6f382898-0370-425b-8c01-6587a985e3f0"
Sep 18 13:44:39.287036 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:39.286799    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wbv96" event={"ID":"f5d1bf14-b322-49d0-9fc8-b3a9a0aca036","Type":"ContainerStarted","Data":"4d0fc82a8228fdda0d0ba9483a769401faf4c8e80fd21739040212c2eee5620a"}
Sep 18 13:44:39.287589 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:39.287074    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wbv96"
Sep 18 13:44:39.287589 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:39.287125    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wbv96"
Sep 18 13:44:39.288728 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:39.288706    2577 generic.go:356] "Generic (PLEG): container finished" podID="47923c57-2f7a-46f5-b8c4-10efd987fa8b" containerID="0b25d3cbc084ad4dd4328096c696dfb311a61d454e153cd115764cb4b3cceb97" exitCode=0
Sep 18 13:44:39.288868 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:39.288786    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-szg9k" event={"ID":"47923c57-2f7a-46f5-b8c4-10efd987fa8b","Type":"ContainerDied","Data":"0b25d3cbc084ad4dd4328096c696dfb311a61d454e153cd115764cb4b3cceb97"}
Sep 18 13:44:39.302915 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:39.302896    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wbv96"
Sep 18 13:44:39.317030 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:39.316989    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-wbv96" podStartSLOduration=7.572948785 podStartE2EDuration="26.316976493s" totalImagesPullingTime="18.744027708s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:44:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:44:39.316770557 +0000 UTC m=+28.369429884" watchObservedRunningTime="2026-09-18 13:44:39.316976493 +0000 UTC m=+28.369635819"
Sep 18 13:44:40.101333 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:40.101310    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nb9bg"
Sep 18 13:44:40.101425 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:40.101341    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdwmn"
Sep 18 13:44:40.101425 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:40.101402    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hj8lg"
Sep 18 13:44:40.101425 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:40.101415    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nb9bg" podUID="87eba103-cfec-4000-a10c-fd4fd512ac90"
Sep 18 13:44:40.101524 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:40.101470    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hj8lg" podUID="6f382898-0370-425b-8c01-6587a985e3f0"
Sep 18 13:44:40.101588 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:40.101563    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdwmn" podUID="802e0a84-ff96-44d6-9aa2-db34555f46eb"
Sep 18 13:44:40.272899 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:40.272805    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bdwmn"]
Sep 18 13:44:40.274897 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:40.274866    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nb9bg"]
Sep 18 13:44:40.277573 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:40.277549    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hj8lg"]
Sep 18 13:44:40.293416 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:40.293385    2577 generic.go:356] "Generic (PLEG): container finished" podID="47923c57-2f7a-46f5-b8c4-10efd987fa8b" containerID="82c837e83ef10fa09eea118c5347170be1ae7c8655254f9ce77408f5ac84ba5a" exitCode=0
Sep 18 13:44:40.294003 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:40.293426    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-szg9k" event={"ID":"47923c57-2f7a-46f5-b8c4-10efd987fa8b","Type":"ContainerDied","Data":"82c837e83ef10fa09eea118c5347170be1ae7c8655254f9ce77408f5ac84ba5a"}
Sep 18 13:44:40.294003 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:40.293475    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hj8lg"
Sep 18 13:44:40.294003 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:40.293540    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdwmn"
Sep 18 13:44:40.294003 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:40.293620    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hj8lg" podUID="6f382898-0370-425b-8c01-6587a985e3f0"
Sep 18 13:44:40.294003 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:40.293681    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nb9bg"
Sep 18 13:44:40.294003 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:40.293677    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdwmn" podUID="802e0a84-ff96-44d6-9aa2-db34555f46eb"
Sep 18 13:44:40.294003 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:40.293774    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nb9bg" podUID="87eba103-cfec-4000-a10c-fd4fd512ac90"
Sep 18 13:44:40.294369 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:40.294342    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wbv96"
Sep 18 13:44:40.311372 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:40.311348    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wbv96"
Sep 18 13:44:41.298059 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:41.298024    2577 generic.go:356] "Generic (PLEG): container finished" podID="47923c57-2f7a-46f5-b8c4-10efd987fa8b" containerID="d31d57e0152434c9e0ef8cc5ee486abac7ff19baba98123dbad03a7e38072abe" exitCode=0
Sep 18 13:44:41.298452 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:41.298108    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-szg9k" event={"ID":"47923c57-2f7a-46f5-b8c4-10efd987fa8b","Type":"ContainerDied","Data":"d31d57e0152434c9e0ef8cc5ee486abac7ff19baba98123dbad03a7e38072abe"}
Sep 18 13:44:42.101973 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:42.101940    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdwmn"
Sep 18 13:44:42.102150 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:42.101940    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nb9bg"
Sep 18 13:44:42.102150 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:42.102058    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdwmn" podUID="802e0a84-ff96-44d6-9aa2-db34555f46eb"
Sep 18 13:44:42.102150 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:42.102078    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hj8lg"
Sep 18 13:44:42.102257 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:42.102150    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nb9bg" podUID="87eba103-cfec-4000-a10c-fd4fd512ac90"
Sep 18 13:44:42.102257 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:42.102209    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hj8lg" podUID="6f382898-0370-425b-8c01-6587a985e3f0"
Sep 18 13:44:44.101709 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:44.101508    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nb9bg"
Sep 18 13:44:44.102180 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:44.101549    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdwmn"
Sep 18 13:44:44.102180 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:44.101804    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nb9bg" podUID="87eba103-cfec-4000-a10c-fd4fd512ac90"
Sep 18 13:44:44.102180 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:44.101549    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hj8lg"
Sep 18 13:44:44.102180 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:44.101930    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdwmn" podUID="802e0a84-ff96-44d6-9aa2-db34555f46eb"
Sep 18 13:44:44.102180 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:44.101981    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hj8lg" podUID="6f382898-0370-425b-8c01-6587a985e3f0"
Sep 18 13:44:45.779667 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:45.779631    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87eba103-cfec-4000-a10c-fd4fd512ac90-metrics-certs\") pod \"network-metrics-daemon-nb9bg\" (UID: \"87eba103-cfec-4000-a10c-fd4fd512ac90\") " pod="openshift-multus/network-metrics-daemon-nb9bg"
Sep 18 13:44:45.780142 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:45.779693    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/802e0a84-ff96-44d6-9aa2-db34555f46eb-original-pull-secret\") pod \"global-pull-secret-syncer-bdwmn\" (UID: \"802e0a84-ff96-44d6-9aa2-db34555f46eb\") " pod="kube-system/global-pull-secret-syncer-bdwmn"
Sep 18 13:44:45.780142 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:45.779796    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:44:45.780142 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:45.779848    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 13:44:45.780142 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:45.779880    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87eba103-cfec-4000-a10c-fd4fd512ac90-metrics-certs podName:87eba103-cfec-4000-a10c-fd4fd512ac90 nodeName:}" failed. No retries permitted until 2026-09-18 13:45:17.77986357 +0000 UTC m=+66.832522889 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/87eba103-cfec-4000-a10c-fd4fd512ac90-metrics-certs") pod "network-metrics-daemon-nb9bg" (UID: "87eba103-cfec-4000-a10c-fd4fd512ac90") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:44:45.780142 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:45.779897    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/802e0a84-ff96-44d6-9aa2-db34555f46eb-original-pull-secret podName:802e0a84-ff96-44d6-9aa2-db34555f46eb nodeName:}" failed. No retries permitted until 2026-09-18 13:45:17.779889669 +0000 UTC m=+66.832548974 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/802e0a84-ff96-44d6-9aa2-db34555f46eb-original-pull-secret") pod "global-pull-secret-syncer-bdwmn" (UID: "802e0a84-ff96-44d6-9aa2-db34555f46eb") : object "kube-system"/"original-pull-secret" not registered
Sep 18 13:44:45.880452 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:45.880412    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmqn6\" (UniqueName: \"kubernetes.io/projected/6f382898-0370-425b-8c01-6587a985e3f0-kube-api-access-dmqn6\") pod \"network-check-target-hj8lg\" (UID: \"6f382898-0370-425b-8c01-6587a985e3f0\") " pod="openshift-network-diagnostics/network-check-target-hj8lg"
Sep 18 13:44:45.880633 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:45.880609    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 13:44:45.880633 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:45.880633    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 13:44:45.880733 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:45.880645    2577 projected.go:196] Error preparing data for projected volume kube-api-access-dmqn6 for pod openshift-network-diagnostics/network-check-target-hj8lg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:44:45.880733 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:45.880713    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f382898-0370-425b-8c01-6587a985e3f0-kube-api-access-dmqn6 podName:6f382898-0370-425b-8c01-6587a985e3f0 nodeName:}" failed. No retries permitted until 2026-09-18 13:45:17.88069319 +0000 UTC m=+66.933352518 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-dmqn6" (UniqueName: "kubernetes.io/projected/6f382898-0370-425b-8c01-6587a985e3f0-kube-api-access-dmqn6") pod "network-check-target-hj8lg" (UID: "6f382898-0370-425b-8c01-6587a985e3f0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:44:46.102031 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:46.101978    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nb9bg"
Sep 18 13:44:46.102199 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:46.102036    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hj8lg"
Sep 18 13:44:46.102199 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:46.102005    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdwmn"
Sep 18 13:44:46.102199 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:46.102117    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nb9bg" podUID="87eba103-cfec-4000-a10c-fd4fd512ac90"
Sep 18 13:44:46.102347 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:46.102194    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdwmn" podUID="802e0a84-ff96-44d6-9aa2-db34555f46eb"
Sep 18 13:44:46.102347 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:46.102272    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hj8lg" podUID="6f382898-0370-425b-8c01-6587a985e3f0"
Sep 18 13:44:46.320259 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:46.320238    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-222.ec2.internal" event="NodeReady"
Sep 18 13:44:46.320362 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:46.320350    2577 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 18 13:44:46.369133 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:46.369064    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-59dccbbcb7-ql9ft"]
Sep 18 13:44:46.391712 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:46.391691    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59dccbbcb7-ql9ft"
Sep 18 13:44:46.395514 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:46.394876    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 18 13:44:46.395514 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:46.395183    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 18 13:44:46.396198 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:46.396175    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 18 13:44:46.396300 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:46.396232    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rp2z9\""
Sep 18 13:44:46.401463 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:46.401444    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 18 13:44:46.402486 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:46.402467    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-fcbr5"]
Sep 18 13:44:46.422176 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:46.422155    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-59dccbbcb7-ql9ft"]
Sep 18 13:44:46.422284 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:46.422271    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hlqkp"]
Sep 18 13:44:46.422444 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:46.422423    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fcbr5"
Sep 18 13:44:46.424803 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:46.424785    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 18 13:44:46.424803 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:46.424797    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5gmms\""
Sep 18 13:44:46.425126 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:46.425110    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 18 13:44:46.436416 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:46.436391    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fcbr5"]
Sep 18 13:44:46.436511 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:46.436475    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hlqkp"
Sep 18 13:44:46.438834 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:46.438804    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 18 13:44:46.439803 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:46.439786    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 18 13:44:46.439970 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:46.439948    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 18 13:44:46.441035 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:46.441017    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-4kzqf\""
Sep 18 13:44:46.444893 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:46.444872    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hlqkp"]
Sep 18 13:44:46.486549 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:46.486520    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/65250207-5996-4ed6-b0f3-555bf3213f85-installation-pull-secrets\") pod \"image-registry-59dccbbcb7-ql9ft\" (UID: \"65250207-5996-4ed6-b0f3-555bf3213f85\") " pod="openshift-image-registry/image-registry-59dccbbcb7-ql9ft"
Sep 18 13:44:46.486708 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:46.486554    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/65250207-5996-4ed6-b0f3-555bf3213f85-image-registry-private-configuration\") pod \"image-registry-59dccbbcb7-ql9ft\" (UID: \"65250207-5996-4ed6-b0f3-555bf3213f85\") " pod="openshift-image-registry/image-registry-59dccbbcb7-ql9ft"
Sep 18 13:44:46.486708 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:46.486574    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce0ee0f8-b52b-4ddc-86f9-41f650a590b2-tmp-dir\") pod \"dns-default-fcbr5\" (UID: \"ce0ee0f8-b52b-4ddc-86f9-41f650a590b2\") " pod="openshift-dns/dns-default-fcbr5"
Sep 18 13:44:46.486708 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:46.486602    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/65250207-5996-4ed6-b0f3-555bf3213f85-registry-certificates\") pod \"image-registry-59dccbbcb7-ql9ft\" (UID: \"65250207-5996-4ed6-b0f3-555bf3213f85\") " pod="openshift-image-registry/image-registry-59dccbbcb7-ql9ft"
Sep 18 13:44:46.486708 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:46.486657    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/65250207-5996-4ed6-b0f3-555bf3213f85-ca-trust-extracted\") pod \"image-registry-59dccbbcb7-ql9ft\" (UID: \"65250207-5996-4ed6-b0f3-555bf3213f85\") " pod="openshift-image-registry/image-registry-59dccbbcb7-ql9ft"
Sep 18 13:44:46.486708 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:46.486700    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r8cf\" (UniqueName: \"kubernetes.io/projected/65250207-5996-4ed6-b0f3-555bf3213f85-kube-api-access-8r8cf\") pod \"image-registry-59dccbbcb7-ql9ft\" (UID: \"65250207-5996-4ed6-b0f3-555bf3213f85\") " pod="openshift-image-registry/image-registry-59dccbbcb7-ql9ft"
Sep 18 13:44:46.486940 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:46.486720    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/65250207-5996-4ed6-b0f3-555bf3213f85-trusted-ca\") pod \"image-registry-59dccbbcb7-ql9ft\" (UID: \"65250207-5996-4ed6-b0f3-555bf3213f85\") " pod="openshift-image-registry/image-registry-59dccbbcb7-ql9ft"
Sep 18 13:44:46.486940 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:46.486747    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp2w8\" (UniqueName: \"kubernetes.io/projected/ce0ee0f8-b52b-4ddc-86f9-41f650a590b2-kube-api-access-rp2w8\") pod \"dns-default-fcbr5\" (UID: \"ce0ee0f8-b52b-4ddc-86f9-41f650a590b2\") " pod="openshift-dns/dns-default-fcbr5"
Sep 18 13:44:46.486940 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:46.486779    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/65250207-5996-4ed6-b0f3-555bf3213f85-registry-tls\") pod \"image-registry-59dccbbcb7-ql9ft\" (UID: \"65250207-5996-4ed6-b0f3-555bf3213f85\") " pod="openshift-image-registry/image-registry-59dccbbcb7-ql9ft"
Sep 18 13:44:46.486940 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:46.486794    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ce0ee0f8-b52b-4ddc-86f9-41f650a590b2-config-volume\") pod \"dns-default-fcbr5\" (UID: \"ce0ee0f8-b52b-4ddc-86f9-41f650a590b2\") " pod="openshift-dns/dns-default-fcbr5"
Sep 18 13:44:46.486940 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:46.486813    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce0ee0f8-b52b-4ddc-86f9-41f650a590b2-metrics-tls\") pod \"dns-default-fcbr5\" (UID: \"ce0ee0f8-b52b-4ddc-86f9-41f650a590b2\") " pod="openshift-dns/dns-default-fcbr5"
Sep 18 13:44:46.486940 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:46.486877    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/65250207-5996-4ed6-b0f3-555bf3213f85-bound-sa-token\") pod \"image-registry-59dccbbcb7-ql9ft\" (UID: \"65250207-5996-4ed6-b0f3-555bf3213f85\") " pod="openshift-image-registry/image-registry-59dccbbcb7-ql9ft"
Sep 18 13:44:46.587692 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:46.587663    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/65250207-5996-4ed6-b0f3-555bf3213f85-registry-certificates\") pod \"image-registry-59dccbbcb7-ql9ft\" (UID: \"65250207-5996-4ed6-b0f3-555bf3213f85\") " pod="openshift-image-registry/image-registry-59dccbbcb7-ql9ft"
Sep 18 13:44:46.587818 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:46.587716    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/65250207-5996-4ed6-b0f3-555bf3213f85-ca-trust-extracted\") pod \"image-registry-59dccbbcb7-ql9ft\" (UID: \"65250207-5996-4ed6-b0f3-555bf3213f85\") " pod="openshift-image-registry/image-registry-59dccbbcb7-ql9ft"
Sep 18 13:44:46.587896 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:46.587849    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e5510d2c-e7d9-4512-934a-a2044c0ba084-cert\") pod \"ingress-canary-hlqkp\" (UID: \"e5510d2c-e7d9-4512-934a-a2044c0ba084\") " pod="openshift-ingress-canary/ingress-canary-hlqkp"
Sep 18 13:44:46.587952 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:46.587899    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r8cf\" (UniqueName: \"kubernetes.io/projected/65250207-5996-4ed6-b0f3-555bf3213f85-kube-api-access-8r8cf\") pod \"image-registry-59dccbbcb7-ql9ft\" (UID: \"65250207-5996-4ed6-b0f3-555bf3213f85\") " pod="openshift-image-registry/image-registry-59dccbbcb7-ql9ft"
Sep 18 13:44:46.587952 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:46.587917    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/65250207-5996-4ed6-b0f3-555bf3213f85-trusted-ca\") pod \"image-registry-59dccbbcb7-ql9ft\" (UID: \"65250207-5996-4ed6-b0f3-555bf3213f85\") " pod="openshift-image-registry/image-registry-59dccbbcb7-ql9ft"
Sep 18 13:44:46.587952 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:46.587941    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp2w8\" (UniqueName: \"kubernetes.io/projected/ce0ee0f8-b52b-4ddc-86f9-41f650a590b2-kube-api-access-rp2w8\") pod \"dns-default-fcbr5\" (UID: \"ce0ee0f8-b52b-4ddc-86f9-41f650a590b2\") " pod="openshift-dns/dns-default-fcbr5"
Sep 18 13:44:46.588098 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:46.587960    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/65250207-5996-4ed6-b0f3-555bf3213f85-registry-tls\") pod \"image-registry-59dccbbcb7-ql9ft\" (UID: \"65250207-5996-4ed6-b0f3-555bf3213f85\") " pod="openshift-image-registry/image-registry-59dccbbcb7-ql9ft"
Sep 18 13:44:46.588098 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:46.587979    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ce0ee0f8-b52b-4ddc-86f9-41f650a590b2-config-volume\") pod \"dns-default-fcbr5\" (UID: \"ce0ee0f8-b52b-4ddc-86f9-41f650a590b2\") " pod="openshift-dns/dns-default-fcbr5"
Sep 18 13:44:46.588098 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:46.588006    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce0ee0f8-b52b-4ddc-86f9-41f650a590b2-metrics-tls\") pod \"dns-default-fcbr5\" (UID: \"ce0ee0f8-b52b-4ddc-86f9-41f650a590b2\") " pod="openshift-dns/dns-default-fcbr5"
Sep 18 13:44:46.588098 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:46.588036    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/65250207-5996-4ed6-b0f3-555bf3213f85-bound-sa-token\") pod \"image-registry-59dccbbcb7-ql9ft\" (UID: \"65250207-5996-4ed6-b0f3-555bf3213f85\") " pod="openshift-image-registry/image-registry-59dccbbcb7-ql9ft"
Sep 18 13:44:46.588098 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:46.588064    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpjv2\" (UniqueName: \"kubernetes.io/projected/e5510d2c-e7d9-4512-934a-a2044c0ba084-kube-api-access-xpjv2\") pod \"ingress-canary-hlqkp\" (UID: \"e5510d2c-e7d9-4512-934a-a2044c0ba084\") " pod="openshift-ingress-canary/ingress-canary-hlqkp"
Sep 18 13:44:46.588098 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:46.588077    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 13:44:46.588098 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:46.588094    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59dccbbcb7-ql9ft: secret "image-registry-tls" not found
Sep 18 13:44:46.588453 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:46.588124    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/65250207-5996-4ed6-b0f3-555bf3213f85-installation-pull-secrets\") pod \"image-registry-59dccbbcb7-ql9ft\" (UID: \"65250207-5996-4ed6-b0f3-555bf3213f85\") " pod="openshift-image-registry/image-registry-59dccbbcb7-ql9ft"
Sep 18 13:44:46.588453 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:46.588152    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/65250207-5996-4ed6-b0f3-555bf3213f85-registry-tls podName:65250207-5996-4ed6-b0f3-555bf3213f85 nodeName:}" failed. No retries permitted until 2026-09-18 13:44:47.088128695 +0000 UTC m=+36.140788017 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/65250207-5996-4ed6-b0f3-555bf3213f85-registry-tls") pod "image-registry-59dccbbcb7-ql9ft" (UID: "65250207-5996-4ed6-b0f3-555bf3213f85") : secret "image-registry-tls" not found
Sep 18 13:44:46.588453 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:46.588175    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/65250207-5996-4ed6-b0f3-555bf3213f85-ca-trust-extracted\") pod \"image-registry-59dccbbcb7-ql9ft\" (UID: \"65250207-5996-4ed6-b0f3-555bf3213f85\") " pod="openshift-image-registry/image-registry-59dccbbcb7-ql9ft"
Sep 18 13:44:46.588453 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:46.588213    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 13:44:46.588453 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:46.588232    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/65250207-5996-4ed6-b0f3-555bf3213f85-image-registry-private-configuration\") pod \"image-registry-59dccbbcb7-ql9ft\" (UID: \"65250207-5996-4ed6-b0f3-555bf3213f85\") " pod="openshift-image-registry/image-registry-59dccbbcb7-ql9ft"
Sep 18 13:44:46.588453 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:46.588376    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/65250207-5996-4ed6-b0f3-555bf3213f85-registry-certificates\") pod \"image-registry-59dccbbcb7-ql9ft\" (UID: \"65250207-5996-4ed6-b0f3-555bf3213f85\") " pod="openshift-image-registry/image-registry-59dccbbcb7-ql9ft"
Sep 18 13:44:46.588453 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:46.588390    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce0ee0f8-b52b-4ddc-86f9-41f650a590b2-metrics-tls podName:ce0ee0f8-b52b-4ddc-86f9-41f650a590b2 nodeName:}" failed. No retries permitted until 2026-09-18 13:44:47.088370704 +0000 UTC m=+36.141030012 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ce0ee0f8-b52b-4ddc-86f9-41f650a590b2-metrics-tls") pod "dns-default-fcbr5" (UID: "ce0ee0f8-b52b-4ddc-86f9-41f650a590b2") : secret "dns-default-metrics-tls" not found
Sep 18 13:44:46.588453 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:46.588416    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce0ee0f8-b52b-4ddc-86f9-41f650a590b2-tmp-dir\") pod \"dns-default-fcbr5\" (UID: \"ce0ee0f8-b52b-4ddc-86f9-41f650a590b2\") " pod="openshift-dns/dns-default-fcbr5"
Sep 18 13:44:46.588731 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:46.588677    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce0ee0f8-b52b-4ddc-86f9-41f650a590b2-tmp-dir\") pod \"dns-default-fcbr5\" (UID: \"ce0ee0f8-b52b-4ddc-86f9-41f650a590b2\") " pod="openshift-dns/dns-default-fcbr5"
Sep 18 13:44:46.588775 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:46.588760    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ce0ee0f8-b52b-4ddc-86f9-41f650a590b2-config-volume\") pod \"dns-default-fcbr5\" (UID: \"ce0ee0f8-b52b-4ddc-86f9-41f650a590b2\") " pod="openshift-dns/dns-default-fcbr5"
Sep 18 13:44:46.589024 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:46.588999    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/65250207-5996-4ed6-b0f3-555bf3213f85-trusted-ca\") pod \"image-registry-59dccbbcb7-ql9ft\" (UID: \"65250207-5996-4ed6-b0f3-555bf3213f85\") " pod="openshift-image-registry/image-registry-59dccbbcb7-ql9ft"
Sep 18 13:44:46.591968 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:46.591945    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/65250207-5996-4ed6-b0f3-555bf3213f85-image-registry-private-configuration\") pod \"image-registry-59dccbbcb7-ql9ft\" (UID: \"65250207-5996-4ed6-b0f3-555bf3213f85\") " pod="openshift-image-registry/image-registry-59dccbbcb7-ql9ft"
Sep 18 13:44:46.591968 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:46.591958    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/65250207-5996-4ed6-b0f3-555bf3213f85-installation-pull-secrets\") pod \"image-registry-59dccbbcb7-ql9ft\" (UID: \"65250207-5996-4ed6-b0f3-555bf3213f85\") " pod="openshift-image-registry/image-registry-59dccbbcb7-ql9ft"
Sep 18 13:44:46.599269 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:46.599244    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp2w8\" (UniqueName: \"kubernetes.io/projected/ce0ee0f8-b52b-4ddc-86f9-41f650a590b2-kube-api-access-rp2w8\") pod \"dns-default-fcbr5\" (UID: \"ce0ee0f8-b52b-4ddc-86f9-41f650a590b2\") " pod="openshift-dns/dns-default-fcbr5"
Sep 18 13:44:46.599549 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:46.599531    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r8cf\" (UniqueName: \"kubernetes.io/projected/65250207-5996-4ed6-b0f3-555bf3213f85-kube-api-access-8r8cf\") pod \"image-registry-59dccbbcb7-ql9ft\" (UID: \"65250207-5996-4ed6-b0f3-555bf3213f85\") " pod="openshift-image-registry/image-registry-59dccbbcb7-ql9ft"
Sep 18 13:44:46.600742 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:46.600722    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/65250207-5996-4ed6-b0f3-555bf3213f85-bound-sa-token\") pod \"image-registry-59dccbbcb7-ql9ft\" (UID: \"65250207-5996-4ed6-b0f3-555bf3213f85\") " pod="openshift-image-registry/image-registry-59dccbbcb7-ql9ft"
Sep 18 13:44:46.618955 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:46.618933    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:44:46.689274 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:46.689197    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e5510d2c-e7d9-4512-934a-a2044c0ba084-cert\") pod \"ingress-canary-hlqkp\" (UID: \"e5510d2c-e7d9-4512-934a-a2044c0ba084\") " pod="openshift-ingress-canary/ingress-canary-hlqkp"
Sep 18 13:44:46.689274 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:46.689266    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpjv2\" (UniqueName: \"kubernetes.io/projected/e5510d2c-e7d9-4512-934a-a2044c0ba084-kube-api-access-xpjv2\") pod \"ingress-canary-hlqkp\" (UID: \"e5510d2c-e7d9-4512-934a-a2044c0ba084\") " pod="openshift-ingress-canary/ingress-canary-hlqkp"
Sep 18 13:44:46.691634 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:46.691610    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e5510d2c-e7d9-4512-934a-a2044c0ba084-cert\") pod \"ingress-canary-hlqkp\" (UID: \"e5510d2c-e7d9-4512-934a-a2044c0ba084\") " pod="openshift-ingress-canary/ingress-canary-hlqkp"
Sep 18 13:44:46.698207 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:46.698185    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpjv2\" (UniqueName: \"kubernetes.io/projected/e5510d2c-e7d9-4512-934a-a2044c0ba084-kube-api-access-xpjv2\") pod \"ingress-canary-hlqkp\" (UID: \"e5510d2c-e7d9-4512-934a-a2044c0ba084\") " pod="openshift-ingress-canary/ingress-canary-hlqkp"
Sep 18 13:44:46.745454 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:46.745421    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hlqkp"
Sep 18 13:44:46.765520 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:46.765484    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:44:47.057255 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:47.057180    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hlqkp"]
Sep 18 13:44:47.062555 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:44:47.062402    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5510d2c_e7d9_4512_934a_a2044c0ba084.slice/crio-2a9fe6ddf7e3093ca07d54bbbb104255df2e0903eef7687f0157431cae632a80 WatchSource:0}: Error finding container 2a9fe6ddf7e3093ca07d54bbbb104255df2e0903eef7687f0157431cae632a80: Status 404 returned error can't find the container with id 2a9fe6ddf7e3093ca07d54bbbb104255df2e0903eef7687f0157431cae632a80
Sep 18 13:44:47.092704 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:47.092682    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/65250207-5996-4ed6-b0f3-555bf3213f85-registry-tls\") pod \"image-registry-59dccbbcb7-ql9ft\" (UID: \"65250207-5996-4ed6-b0f3-555bf3213f85\") " pod="openshift-image-registry/image-registry-59dccbbcb7-ql9ft"
Sep 18 13:44:47.092816 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:47.092712    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce0ee0f8-b52b-4ddc-86f9-41f650a590b2-metrics-tls\") pod \"dns-default-fcbr5\" (UID: \"ce0ee0f8-b52b-4ddc-86f9-41f650a590b2\") " pod="openshift-dns/dns-default-fcbr5"
Sep 18 13:44:47.092886 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:47.092863    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 13:44:47.092886 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:47.092867    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 13:44:47.092886 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:47.092883    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59dccbbcb7-ql9ft: secret "image-registry-tls" not found
Sep 18 13:44:47.092984 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:47.092924    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce0ee0f8-b52b-4ddc-86f9-41f650a590b2-metrics-tls podName:ce0ee0f8-b52b-4ddc-86f9-41f650a590b2 nodeName:}" failed. No retries permitted until 2026-09-18 13:44:48.092905682 +0000 UTC m=+37.145564987 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ce0ee0f8-b52b-4ddc-86f9-41f650a590b2-metrics-tls") pod "dns-default-fcbr5" (UID: "ce0ee0f8-b52b-4ddc-86f9-41f650a590b2") : secret "dns-default-metrics-tls" not found
Sep 18 13:44:47.092984 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:47.092937    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/65250207-5996-4ed6-b0f3-555bf3213f85-registry-tls podName:65250207-5996-4ed6-b0f3-555bf3213f85 nodeName:}" failed. No retries permitted until 2026-09-18 13:44:48.092931178 +0000 UTC m=+37.145590482 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/65250207-5996-4ed6-b0f3-555bf3213f85-registry-tls") pod "image-registry-59dccbbcb7-ql9ft" (UID: "65250207-5996-4ed6-b0f3-555bf3213f85") : secret "image-registry-tls" not found
Sep 18 13:44:47.312806 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:47.312715    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hlqkp" event={"ID":"e5510d2c-e7d9-4512-934a-a2044c0ba084","Type":"ContainerStarted","Data":"2a9fe6ddf7e3093ca07d54bbbb104255df2e0903eef7687f0157431cae632a80"}
Sep 18 13:44:47.315471 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:47.315444    2577 generic.go:356] "Generic (PLEG): container finished" podID="47923c57-2f7a-46f5-b8c4-10efd987fa8b" containerID="229b2cd5945ccf0e19e770b165948f6bb924ca5dc480b741ef1d948fe1310aa2" exitCode=0
Sep 18 13:44:47.315589 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:47.315487    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-szg9k" event={"ID":"47923c57-2f7a-46f5-b8c4-10efd987fa8b","Type":"ContainerDied","Data":"229b2cd5945ccf0e19e770b165948f6bb924ca5dc480b741ef1d948fe1310aa2"}
Sep 18 13:44:48.101253 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:48.101222    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdwmn"
Sep 18 13:44:48.101964 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:48.101361    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hj8lg"
Sep 18 13:44:48.101964 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:48.101396    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nb9bg"
Sep 18 13:44:48.101964 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:48.101893    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/65250207-5996-4ed6-b0f3-555bf3213f85-registry-tls\") pod \"image-registry-59dccbbcb7-ql9ft\" (UID: \"65250207-5996-4ed6-b0f3-555bf3213f85\") " pod="openshift-image-registry/image-registry-59dccbbcb7-ql9ft"
Sep 18 13:44:48.101964 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:48.101925    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce0ee0f8-b52b-4ddc-86f9-41f650a590b2-metrics-tls\") pod \"dns-default-fcbr5\" (UID: \"ce0ee0f8-b52b-4ddc-86f9-41f650a590b2\") " pod="openshift-dns/dns-default-fcbr5"
Sep 18 13:44:48.102216 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:48.102026    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 13:44:48.102216 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:48.102077    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce0ee0f8-b52b-4ddc-86f9-41f650a590b2-metrics-tls podName:ce0ee0f8-b52b-4ddc-86f9-41f650a590b2 nodeName:}" failed. No retries permitted until 2026-09-18 13:44:50.102058296 +0000 UTC m=+39.154717600 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ce0ee0f8-b52b-4ddc-86f9-41f650a590b2-metrics-tls") pod "dns-default-fcbr5" (UID: "ce0ee0f8-b52b-4ddc-86f9-41f650a590b2") : secret "dns-default-metrics-tls" not found
Sep 18 13:44:48.102216 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:48.102025    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 13:44:48.102216 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:48.102114    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59dccbbcb7-ql9ft: secret "image-registry-tls" not found
Sep 18 13:44:48.102216 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:48.102188    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/65250207-5996-4ed6-b0f3-555bf3213f85-registry-tls podName:65250207-5996-4ed6-b0f3-555bf3213f85 nodeName:}" failed. No retries permitted until 2026-09-18 13:44:50.102172884 +0000 UTC m=+39.154832204 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/65250207-5996-4ed6-b0f3-555bf3213f85-registry-tls") pod "image-registry-59dccbbcb7-ql9ft" (UID: "65250207-5996-4ed6-b0f3-555bf3213f85") : secret "image-registry-tls" not found
Sep 18 13:44:48.104194 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:48.104174    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 13:44:48.104757 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:48.104730    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 13:44:48.104876 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:48.104754    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tqxkk\""
Sep 18 13:44:48.104876 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:48.104736    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 13:44:48.104876 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:48.104764    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 13:44:48.104876 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:48.104859    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8ktpw\""
Sep 18 13:44:48.322548 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:48.322509    2577 generic.go:356] "Generic (PLEG): container finished" podID="47923c57-2f7a-46f5-b8c4-10efd987fa8b" containerID="acb7e646db7876040f3f73a2a789cfab9732dd30c95993537080e07e3d60646a" exitCode=0
Sep 18 13:44:48.322721 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:48.322567    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-szg9k" event={"ID":"47923c57-2f7a-46f5-b8c4-10efd987fa8b","Type":"ContainerDied","Data":"acb7e646db7876040f3f73a2a789cfab9732dd30c95993537080e07e3d60646a"}
Sep 18 13:44:49.329860 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:49.329807    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-szg9k" event={"ID":"47923c57-2f7a-46f5-b8c4-10efd987fa8b","Type":"ContainerStarted","Data":"d03a065e456187fc709a3f430a4bdd99dc866152803af40772005b20358d2c31"}
Sep 18 13:44:49.354554 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:49.354489    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-szg9k" podStartSLOduration=6.915513256 podStartE2EDuration="36.354465693s" totalImagesPullingTime="29.438952437s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:44:13 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:44:49.353285122 +0000 UTC m=+38.405944451" watchObservedRunningTime="2026-09-18 13:44:49.354465693 +0000 UTC m=+38.407125038"
Sep 18 13:44:49.480863 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:49.480807    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:44:49.594642 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:49.594615    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:44:50.117284 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:50.117202    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/65250207-5996-4ed6-b0f3-555bf3213f85-registry-tls\") pod \"image-registry-59dccbbcb7-ql9ft\" (UID: \"65250207-5996-4ed6-b0f3-555bf3213f85\") " pod="openshift-image-registry/image-registry-59dccbbcb7-ql9ft"
Sep 18 13:44:50.117284 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:50.117255    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce0ee0f8-b52b-4ddc-86f9-41f650a590b2-metrics-tls\") pod \"dns-default-fcbr5\" (UID: \"ce0ee0f8-b52b-4ddc-86f9-41f650a590b2\") " pod="openshift-dns/dns-default-fcbr5"
Sep 18 13:44:50.117609 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:50.117319    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 13:44:50.117609 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:50.117337    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59dccbbcb7-ql9ft: secret "image-registry-tls" not found
Sep 18 13:44:50.117609 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:50.117385    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/65250207-5996-4ed6-b0f3-555bf3213f85-registry-tls podName:65250207-5996-4ed6-b0f3-555bf3213f85 nodeName:}" failed. No retries permitted until 2026-09-18 13:44:54.117369083 +0000 UTC m=+43.170028392 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/65250207-5996-4ed6-b0f3-555bf3213f85-registry-tls") pod "image-registry-59dccbbcb7-ql9ft" (UID: "65250207-5996-4ed6-b0f3-555bf3213f85") : secret "image-registry-tls" not found
Sep 18 13:44:50.117609 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:50.117396    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 13:44:50.117609 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:50.117473    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce0ee0f8-b52b-4ddc-86f9-41f650a590b2-metrics-tls podName:ce0ee0f8-b52b-4ddc-86f9-41f650a590b2 nodeName:}" failed. No retries permitted until 2026-09-18 13:44:54.117455811 +0000 UTC m=+43.170115130 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ce0ee0f8-b52b-4ddc-86f9-41f650a590b2-metrics-tls") pod "dns-default-fcbr5" (UID: "ce0ee0f8-b52b-4ddc-86f9-41f650a590b2") : secret "dns-default-metrics-tls" not found
Sep 18 13:44:50.334018 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:50.333980    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hlqkp" event={"ID":"e5510d2c-e7d9-4512-934a-a2044c0ba084","Type":"ContainerStarted","Data":"65513bea6c853fda65c1e51036fbc5707df8f306f5931e97292b57affc58f73b"}
Sep 18 13:44:50.349118 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:50.349075    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-hlqkp" podStartSLOduration=1.651914782 podStartE2EDuration="4.349061116s" totalImagesPullingTime="2.697146334s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:44:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:44:50.348958136 +0000 UTC m=+39.401617466" watchObservedRunningTime="2026-09-18 13:44:50.349061116 +0000 UTC m=+39.401720443"
Sep 18 13:44:51.494343 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:51.494312    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:44:52.608728 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:52.608696    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:44:52.670250 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:52.670223    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:44:53.905232 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:53.905202    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:44:54.004205 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:54.004171    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:44:54.146104 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:54.146069    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/65250207-5996-4ed6-b0f3-555bf3213f85-registry-tls\") pod \"image-registry-59dccbbcb7-ql9ft\" (UID: \"65250207-5996-4ed6-b0f3-555bf3213f85\") " pod="openshift-image-registry/image-registry-59dccbbcb7-ql9ft"
Sep 18 13:44:54.146104 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:54.146106    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce0ee0f8-b52b-4ddc-86f9-41f650a590b2-metrics-tls\") pod \"dns-default-fcbr5\" (UID: \"ce0ee0f8-b52b-4ddc-86f9-41f650a590b2\") " pod="openshift-dns/dns-default-fcbr5"
Sep 18 13:44:54.146301 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:54.146223    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 13:44:54.146301 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:54.146228    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 13:44:54.146301 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:54.146244    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59dccbbcb7-ql9ft: secret "image-registry-tls" not found
Sep 18 13:44:54.146301 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:54.146288    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/65250207-5996-4ed6-b0f3-555bf3213f85-registry-tls podName:65250207-5996-4ed6-b0f3-555bf3213f85 nodeName:}" failed. No retries permitted until 2026-09-18 13:45:02.14627047 +0000 UTC m=+51.198929792 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/65250207-5996-4ed6-b0f3-555bf3213f85-registry-tls") pod "image-registry-59dccbbcb7-ql9ft" (UID: "65250207-5996-4ed6-b0f3-555bf3213f85") : secret "image-registry-tls" not found
Sep 18 13:44:54.146427 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:44:54.146305    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce0ee0f8-b52b-4ddc-86f9-41f650a590b2-metrics-tls podName:ce0ee0f8-b52b-4ddc-86f9-41f650a590b2 nodeName:}" failed. No retries permitted until 2026-09-18 13:45:02.146296187 +0000 UTC m=+51.198955496 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ce0ee0f8-b52b-4ddc-86f9-41f650a590b2-metrics-tls") pod "dns-default-fcbr5" (UID: "ce0ee0f8-b52b-4ddc-86f9-41f650a590b2") : secret "dns-default-metrics-tls" not found
Sep 18 13:44:54.452023 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:54.451993    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:44:54.596127 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:54.596098    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:44:56.222031 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:56.221995    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:44:56.285994 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:56.285965    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:44:57.702928 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:57.702891    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:44:57.800540 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:57.800509    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:44:58.673022 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:58.672988    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:44:58.758866 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:44:58.758837    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:00.293595 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.293564    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-7g9jh"]
Sep 18 13:45:00.323416 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.323385    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-8tnxl"]
Sep 18 13:45:00.323552 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.323496    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-7g9jh"
Sep 18 13:45:00.327313 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.327278    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 18 13:45:00.327484 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.327462    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 18 13:45:00.328268 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.328253    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 18 13:45:00.328374 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.328283    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 18 13:45:00.331856 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.331836    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-85jmd\""
Sep 18 13:45:00.336918 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.336901    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-7g9jh"]
Sep 18 13:45:00.336981 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.336930    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-8tnxl"]
Sep 18 13:45:00.337015 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.336992    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-8tnxl"
Sep 18 13:45:00.339204 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.339187    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 18 13:45:00.341413 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.341395    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 18 13:45:00.341514 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.341418    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 18 13:45:00.341514 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.341458    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-4nqb4\""
Sep 18 13:45:00.341673 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.341657    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 18 13:45:00.347223 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.347205    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 18 13:45:00.388835 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.388799    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-4xlbs"]
Sep 18 13:45:00.415093 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.415071    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-fvx98"]
Sep 18 13:45:00.415228 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.415210    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-4xlbs"
Sep 18 13:45:00.418384 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.418337    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-q9rrm\""
Sep 18 13:45:00.418587 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.418568    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 18 13:45:00.418675 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.418587    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 18 13:45:00.418675 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.418667    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 18 13:45:00.438919 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.438901    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-4xlbs"]
Sep 18 13:45:00.439013 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.438930    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-fvx98"]
Sep 18 13:45:00.439013 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.438994    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-fvx98"
Sep 18 13:45:00.440748 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.440729    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-pnt24\""
Sep 18 13:45:00.483702 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.483676    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-v99sv"]
Sep 18 13:45:00.493098 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.493080    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd8tg\" (UniqueName: \"kubernetes.io/projected/69d3178a-5769-4b5f-a243-30a1bedc5bd6-kube-api-access-nd8tg\") pod \"console-operator-fd4bb46c-8tnxl\" (UID: \"69d3178a-5769-4b5f-a243-30a1bedc5bd6\") " pod="openshift-console-operator/console-operator-fd4bb46c-8tnxl"
Sep 18 13:45:00.493176 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.493122    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/69d3178a-5769-4b5f-a243-30a1bedc5bd6-trusted-ca\") pod \"console-operator-fd4bb46c-8tnxl\" (UID: \"69d3178a-5769-4b5f-a243-30a1bedc5bd6\") " pod="openshift-console-operator/console-operator-fd4bb46c-8tnxl"
Sep 18 13:45:00.493215 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.493176    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/69d3178a-5769-4b5f-a243-30a1bedc5bd6-config\") pod \"console-operator-fd4bb46c-8tnxl\" (UID: \"69d3178a-5769-4b5f-a243-30a1bedc5bd6\") " pod="openshift-console-operator/console-operator-fd4bb46c-8tnxl"
Sep 18 13:45:00.493215 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.493204    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/38fce007-dfca-4566-bea9-3e8c2ea32974-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-7g9jh\" (UID: \"38fce007-dfca-4566-bea9-3e8c2ea32974\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-7g9jh"
Sep 18 13:45:00.493278 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.493265    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4ng6\" (UniqueName: \"kubernetes.io/projected/38fce007-dfca-4566-bea9-3e8c2ea32974-kube-api-access-m4ng6\") pod \"cluster-monitoring-operator-847b8bd8d7-7g9jh\" (UID: \"38fce007-dfca-4566-bea9-3e8c2ea32974\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-7g9jh"
Sep 18 13:45:00.493308 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.493280    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/69d3178a-5769-4b5f-a243-30a1bedc5bd6-serving-cert\") pod \"console-operator-fd4bb46c-8tnxl\" (UID: \"69d3178a-5769-4b5f-a243-30a1bedc5bd6\") " pod="openshift-console-operator/console-operator-fd4bb46c-8tnxl"
Sep 18 13:45:00.493340 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.493315    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/38fce007-dfca-4566-bea9-3e8c2ea32974-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-7g9jh\" (UID: \"38fce007-dfca-4566-bea9-3e8c2ea32974\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-7g9jh"
Sep 18 13:45:00.498994 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.498975    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-v99sv"]
Sep 18 13:45:00.499077 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.499059    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-v99sv"
Sep 18 13:45:00.501524 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.501501    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 18 13:45:00.501682 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.501661    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-pblc8\""
Sep 18 13:45:00.501783 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.501692    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 18 13:45:00.501783 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.501692    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 18 13:45:00.501783 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.501750    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 18 13:45:00.538593 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.538536    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:00.593606 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.593582    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a20817cc-9558-434d-9061-86be074f9ad2-config\") pod \"service-ca-operator-57cdc4b768-v99sv\" (UID: \"a20817cc-9558-434d-9061-86be074f9ad2\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-v99sv"
Sep 18 13:45:00.593714 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.593620    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd8tg\" (UniqueName: \"kubernetes.io/projected/69d3178a-5769-4b5f-a243-30a1bedc5bd6-kube-api-access-nd8tg\") pod \"console-operator-fd4bb46c-8tnxl\" (UID: \"69d3178a-5769-4b5f-a243-30a1bedc5bd6\") " pod="openshift-console-operator/console-operator-fd4bb46c-8tnxl"
Sep 18 13:45:00.593714 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.593661    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/69d3178a-5769-4b5f-a243-30a1bedc5bd6-trusted-ca\") pod \"console-operator-fd4bb46c-8tnxl\" (UID: \"69d3178a-5769-4b5f-a243-30a1bedc5bd6\") " pod="openshift-console-operator/console-operator-fd4bb46c-8tnxl"
Sep 18 13:45:00.593714 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.593692    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mccns\" (UniqueName: \"kubernetes.io/projected/369561d5-6f05-4034-a7e1-24138261ca29-kube-api-access-mccns\") pod \"cluster-samples-operator-685d7d68b5-4xlbs\" (UID: \"369561d5-6f05-4034-a7e1-24138261ca29\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-4xlbs"
Sep 18 13:45:00.593854 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.593711    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/69d3178a-5769-4b5f-a243-30a1bedc5bd6-config\") pod \"console-operator-fd4bb46c-8tnxl\" (UID: \"69d3178a-5769-4b5f-a243-30a1bedc5bd6\") " pod="openshift-console-operator/console-operator-fd4bb46c-8tnxl"
Sep 18 13:45:00.593854 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.593737    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/38fce007-dfca-4566-bea9-3e8c2ea32974-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-7g9jh\" (UID: \"38fce007-dfca-4566-bea9-3e8c2ea32974\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-7g9jh"
Sep 18 13:45:00.593854 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.593792    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a20817cc-9558-434d-9061-86be074f9ad2-serving-cert\") pod \"service-ca-operator-57cdc4b768-v99sv\" (UID: \"a20817cc-9558-434d-9061-86be074f9ad2\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-v99sv"
Sep 18 13:45:00.593989 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.593850    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4ng6\" (UniqueName: \"kubernetes.io/projected/38fce007-dfca-4566-bea9-3e8c2ea32974-kube-api-access-m4ng6\") pod \"cluster-monitoring-operator-847b8bd8d7-7g9jh\" (UID: \"38fce007-dfca-4566-bea9-3e8c2ea32974\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-7g9jh"
Sep 18 13:45:00.593989 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.593875    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/69d3178a-5769-4b5f-a243-30a1bedc5bd6-serving-cert\") pod \"console-operator-fd4bb46c-8tnxl\" (UID: \"69d3178a-5769-4b5f-a243-30a1bedc5bd6\") " pod="openshift-console-operator/console-operator-fd4bb46c-8tnxl"
Sep 18 13:45:00.593989 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.593898    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtv45\" (UniqueName: \"kubernetes.io/projected/b47b0c48-ce7a-4511-8f7c-d10cb8dd83b8-kube-api-access-mtv45\") pod \"network-check-source-7cbc45fddc-fvx98\" (UID: \"b47b0c48-ce7a-4511-8f7c-d10cb8dd83b8\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-fvx98"
Sep 18 13:45:00.593989 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:45:00.593910    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 13:45:00.594183 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:45:00.593994    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38fce007-dfca-4566-bea9-3e8c2ea32974-cluster-monitoring-operator-tls podName:38fce007-dfca-4566-bea9-3e8c2ea32974 nodeName:}" failed. No retries permitted until 2026-09-18 13:45:01.093971653 +0000 UTC m=+50.146630963 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/38fce007-dfca-4566-bea9-3e8c2ea32974-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-7g9jh" (UID: "38fce007-dfca-4566-bea9-3e8c2ea32974") : secret "cluster-monitoring-operator-tls" not found
Sep 18 13:45:00.594183 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.594030    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhzwp\" (UniqueName: \"kubernetes.io/projected/a20817cc-9558-434d-9061-86be074f9ad2-kube-api-access-nhzwp\") pod \"service-ca-operator-57cdc4b768-v99sv\" (UID: \"a20817cc-9558-434d-9061-86be074f9ad2\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-v99sv"
Sep 18 13:45:00.594183 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.594081    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/38fce007-dfca-4566-bea9-3e8c2ea32974-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-7g9jh\" (UID: \"38fce007-dfca-4566-bea9-3e8c2ea32974\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-7g9jh"
Sep 18 13:45:00.594183 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.594107    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/369561d5-6f05-4034-a7e1-24138261ca29-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-4xlbs\" (UID: \"369561d5-6f05-4034-a7e1-24138261ca29\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-4xlbs"
Sep 18 13:45:00.594433 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.594415    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/69d3178a-5769-4b5f-a243-30a1bedc5bd6-config\") pod \"console-operator-fd4bb46c-8tnxl\" (UID: \"69d3178a-5769-4b5f-a243-30a1bedc5bd6\") " pod="openshift-console-operator/console-operator-fd4bb46c-8tnxl"
Sep 18 13:45:00.594571 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.594553    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/69d3178a-5769-4b5f-a243-30a1bedc5bd6-trusted-ca\") pod \"console-operator-fd4bb46c-8tnxl\" (UID: \"69d3178a-5769-4b5f-a243-30a1bedc5bd6\") " pod="openshift-console-operator/console-operator-fd4bb46c-8tnxl"
Sep 18 13:45:00.594807 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.594787    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/38fce007-dfca-4566-bea9-3e8c2ea32974-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-7g9jh\" (UID: \"38fce007-dfca-4566-bea9-3e8c2ea32974\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-7g9jh"
Sep 18 13:45:00.597214 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.597195    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/69d3178a-5769-4b5f-a243-30a1bedc5bd6-serving-cert\") pod \"console-operator-fd4bb46c-8tnxl\" (UID: \"69d3178a-5769-4b5f-a243-30a1bedc5bd6\") " pod="openshift-console-operator/console-operator-fd4bb46c-8tnxl"
Sep 18 13:45:00.601986 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.601963    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd8tg\" (UniqueName: \"kubernetes.io/projected/69d3178a-5769-4b5f-a243-30a1bedc5bd6-kube-api-access-nd8tg\") pod \"console-operator-fd4bb46c-8tnxl\" (UID: \"69d3178a-5769-4b5f-a243-30a1bedc5bd6\") " pod="openshift-console-operator/console-operator-fd4bb46c-8tnxl"
Sep 18 13:45:00.602706 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.602682    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4ng6\" (UniqueName: \"kubernetes.io/projected/38fce007-dfca-4566-bea9-3e8c2ea32974-kube-api-access-m4ng6\") pod \"cluster-monitoring-operator-847b8bd8d7-7g9jh\" (UID: \"38fce007-dfca-4566-bea9-3e8c2ea32974\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-7g9jh"
Sep 18 13:45:00.613753 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.613730    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:00.644799 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.644771    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-8tnxl"
Sep 18 13:45:00.694929 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.694887    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mccns\" (UniqueName: \"kubernetes.io/projected/369561d5-6f05-4034-a7e1-24138261ca29-kube-api-access-mccns\") pod \"cluster-samples-operator-685d7d68b5-4xlbs\" (UID: \"369561d5-6f05-4034-a7e1-24138261ca29\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-4xlbs"
Sep 18 13:45:00.695110 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.695083    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a20817cc-9558-434d-9061-86be074f9ad2-serving-cert\") pod \"service-ca-operator-57cdc4b768-v99sv\" (UID: \"a20817cc-9558-434d-9061-86be074f9ad2\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-v99sv"
Sep 18 13:45:00.695159 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.695137    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtv45\" (UniqueName: \"kubernetes.io/projected/b47b0c48-ce7a-4511-8f7c-d10cb8dd83b8-kube-api-access-mtv45\") pod \"network-check-source-7cbc45fddc-fvx98\" (UID: \"b47b0c48-ce7a-4511-8f7c-d10cb8dd83b8\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-fvx98"
Sep 18 13:45:00.695202 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.695186    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhzwp\" (UniqueName: \"kubernetes.io/projected/a20817cc-9558-434d-9061-86be074f9ad2-kube-api-access-nhzwp\") pod \"service-ca-operator-57cdc4b768-v99sv\" (UID: \"a20817cc-9558-434d-9061-86be074f9ad2\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-v99sv"
Sep 18 13:45:00.695248 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.695224    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/369561d5-6f05-4034-a7e1-24138261ca29-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-4xlbs\" (UID: \"369561d5-6f05-4034-a7e1-24138261ca29\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-4xlbs"
Sep 18 13:45:00.695293 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.695246    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a20817cc-9558-434d-9061-86be074f9ad2-config\") pod \"service-ca-operator-57cdc4b768-v99sv\" (UID: \"a20817cc-9558-434d-9061-86be074f9ad2\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-v99sv"
Sep 18 13:45:00.695343 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:45:00.695321    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 13:45:00.695411 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:45:00.695396    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/369561d5-6f05-4034-a7e1-24138261ca29-samples-operator-tls podName:369561d5-6f05-4034-a7e1-24138261ca29 nodeName:}" failed. No retries permitted until 2026-09-18 13:45:01.195377306 +0000 UTC m=+50.248036628 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/369561d5-6f05-4034-a7e1-24138261ca29-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-4xlbs" (UID: "369561d5-6f05-4034-a7e1-24138261ca29") : secret "samples-operator-tls" not found
Sep 18 13:45:00.695915 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.695894    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a20817cc-9558-434d-9061-86be074f9ad2-config\") pod \"service-ca-operator-57cdc4b768-v99sv\" (UID: \"a20817cc-9558-434d-9061-86be074f9ad2\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-v99sv"
Sep 18 13:45:00.697223 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.697207    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a20817cc-9558-434d-9061-86be074f9ad2-serving-cert\") pod \"service-ca-operator-57cdc4b768-v99sv\" (UID: \"a20817cc-9558-434d-9061-86be074f9ad2\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-v99sv"
Sep 18 13:45:00.705578 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.705546    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mccns\" (UniqueName: \"kubernetes.io/projected/369561d5-6f05-4034-a7e1-24138261ca29-kube-api-access-mccns\") pod \"cluster-samples-operator-685d7d68b5-4xlbs\" (UID: \"369561d5-6f05-4034-a7e1-24138261ca29\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-4xlbs"
Sep 18 13:45:00.705706 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.705550    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtv45\" (UniqueName: \"kubernetes.io/projected/b47b0c48-ce7a-4511-8f7c-d10cb8dd83b8-kube-api-access-mtv45\") pod \"network-check-source-7cbc45fddc-fvx98\" (UID: \"b47b0c48-ce7a-4511-8f7c-d10cb8dd83b8\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-fvx98"
Sep 18 13:45:00.714198 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.714177    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhzwp\" (UniqueName: \"kubernetes.io/projected/a20817cc-9558-434d-9061-86be074f9ad2-kube-api-access-nhzwp\") pod \"service-ca-operator-57cdc4b768-v99sv\" (UID: \"a20817cc-9558-434d-9061-86be074f9ad2\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-v99sv"
Sep 18 13:45:00.717389 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.717363    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:00.757857 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.757813    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-fvx98"
Sep 18 13:45:00.807606 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.807570    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-v99sv"
Sep 18 13:45:00.815763 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.815703    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:00.933346 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:00.933310    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-8tnxl"]
Sep 18 13:45:00.936262 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:45:00.936234    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69d3178a_5769_4b5f_a243_30a1bedc5bd6.slice/crio-6ed6d65a30c3e2880bd7ca479f42e68e9e88249f0a18dcb57148a717cebd1e5b WatchSource:0}: Error finding container 6ed6d65a30c3e2880bd7ca479f42e68e9e88249f0a18dcb57148a717cebd1e5b: Status 404 returned error can't find the container with id 6ed6d65a30c3e2880bd7ca479f42e68e9e88249f0a18dcb57148a717cebd1e5b
Sep 18 13:45:01.039731 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:01.039699    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-fvx98"]
Sep 18 13:45:01.042762 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:45:01.042740    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb47b0c48_ce7a_4511_8f7c_d10cb8dd83b8.slice/crio-41b194c78050c5754503d5c72bee8a2b1bc2029d796dea6101abc9c9aed6ae74 WatchSource:0}: Error finding container 41b194c78050c5754503d5c72bee8a2b1bc2029d796dea6101abc9c9aed6ae74: Status 404 returned error can't find the container with id 41b194c78050c5754503d5c72bee8a2b1bc2029d796dea6101abc9c9aed6ae74
Sep 18 13:45:01.089096 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:01.088722    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-v99sv"]
Sep 18 13:45:01.093004 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:45:01.092977    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda20817cc_9558_434d_9061_86be074f9ad2.slice/crio-3da40db3257891d93713a697792c80b28c6e35ce4e38db4c0c561db42224e1b1 WatchSource:0}: Error finding container 3da40db3257891d93713a697792c80b28c6e35ce4e38db4c0c561db42224e1b1: Status 404 returned error can't find the container with id 3da40db3257891d93713a697792c80b28c6e35ce4e38db4c0c561db42224e1b1
Sep 18 13:45:01.098816 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:01.098794    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/38fce007-dfca-4566-bea9-3e8c2ea32974-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-7g9jh\" (UID: \"38fce007-dfca-4566-bea9-3e8c2ea32974\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-7g9jh"
Sep 18 13:45:01.098924 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:45:01.098907    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 13:45:01.098976 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:45:01.098965    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38fce007-dfca-4566-bea9-3e8c2ea32974-cluster-monitoring-operator-tls podName:38fce007-dfca-4566-bea9-3e8c2ea32974 nodeName:}" failed. No retries permitted until 2026-09-18 13:45:02.098949565 +0000 UTC m=+51.151608872 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/38fce007-dfca-4566-bea9-3e8c2ea32974-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-7g9jh" (UID: "38fce007-dfca-4566-bea9-3e8c2ea32974") : secret "cluster-monitoring-operator-tls" not found
Sep 18 13:45:01.199916 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:01.199817    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/369561d5-6f05-4034-a7e1-24138261ca29-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-4xlbs\" (UID: \"369561d5-6f05-4034-a7e1-24138261ca29\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-4xlbs"
Sep 18 13:45:01.200057 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:45:01.199963    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 13:45:01.200057 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:45:01.200032    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/369561d5-6f05-4034-a7e1-24138261ca29-samples-operator-tls podName:369561d5-6f05-4034-a7e1-24138261ca29 nodeName:}" failed. No retries permitted until 2026-09-18 13:45:02.20001467 +0000 UTC m=+51.252673976 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/369561d5-6f05-4034-a7e1-24138261ca29-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-4xlbs" (UID: "369561d5-6f05-4034-a7e1-24138261ca29") : secret "samples-operator-tls" not found
Sep 18 13:45:01.362003 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:01.361966    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-fvx98" event={"ID":"b47b0c48-ce7a-4511-8f7c-d10cb8dd83b8","Type":"ContainerStarted","Data":"41b194c78050c5754503d5c72bee8a2b1bc2029d796dea6101abc9c9aed6ae74"}
Sep 18 13:45:01.363122 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:01.363097    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-v99sv" event={"ID":"a20817cc-9558-434d-9061-86be074f9ad2","Type":"ContainerStarted","Data":"3da40db3257891d93713a697792c80b28c6e35ce4e38db4c0c561db42224e1b1"}
Sep 18 13:45:01.364063 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:01.364044    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-8tnxl" event={"ID":"69d3178a-5769-4b5f-a243-30a1bedc5bd6","Type":"ContainerStarted","Data":"6ed6d65a30c3e2880bd7ca479f42e68e9e88249f0a18dcb57148a717cebd1e5b"}
Sep 18 13:45:01.617249 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:01.617206    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:02.108206 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:02.108125    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/38fce007-dfca-4566-bea9-3e8c2ea32974-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-7g9jh\" (UID: \"38fce007-dfca-4566-bea9-3e8c2ea32974\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-7g9jh"
Sep 18 13:45:02.108373 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:45:02.108294    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 13:45:02.108373 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:45:02.108368    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38fce007-dfca-4566-bea9-3e8c2ea32974-cluster-monitoring-operator-tls podName:38fce007-dfca-4566-bea9-3e8c2ea32974 nodeName:}" failed. No retries permitted until 2026-09-18 13:45:04.108346987 +0000 UTC m=+53.161006313 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/38fce007-dfca-4566-bea9-3e8c2ea32974-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-7g9jh" (UID: "38fce007-dfca-4566-bea9-3e8c2ea32974") : secret "cluster-monitoring-operator-tls" not found
Sep 18 13:45:02.208948 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:02.208912    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/65250207-5996-4ed6-b0f3-555bf3213f85-registry-tls\") pod \"image-registry-59dccbbcb7-ql9ft\" (UID: \"65250207-5996-4ed6-b0f3-555bf3213f85\") " pod="openshift-image-registry/image-registry-59dccbbcb7-ql9ft"
Sep 18 13:45:02.209128 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:02.208964    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce0ee0f8-b52b-4ddc-86f9-41f650a590b2-metrics-tls\") pod \"dns-default-fcbr5\" (UID: \"ce0ee0f8-b52b-4ddc-86f9-41f650a590b2\") " pod="openshift-dns/dns-default-fcbr5"
Sep 18 13:45:02.209128 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:02.209027    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/369561d5-6f05-4034-a7e1-24138261ca29-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-4xlbs\" (UID: \"369561d5-6f05-4034-a7e1-24138261ca29\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-4xlbs"
Sep 18 13:45:02.209230 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:45:02.209178    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 13:45:02.209284 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:45:02.209240    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/369561d5-6f05-4034-a7e1-24138261ca29-samples-operator-tls podName:369561d5-6f05-4034-a7e1-24138261ca29 nodeName:}" failed. No retries permitted until 2026-09-18 13:45:04.209221345 +0000 UTC m=+53.261880671 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/369561d5-6f05-4034-a7e1-24138261ca29-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-4xlbs" (UID: "369561d5-6f05-4034-a7e1-24138261ca29") : secret "samples-operator-tls" not found
Sep 18 13:45:02.209348 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:45:02.209318    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 13:45:02.209348 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:45:02.209330    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59dccbbcb7-ql9ft: secret "image-registry-tls" not found
Sep 18 13:45:02.209442 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:45:02.209363    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/65250207-5996-4ed6-b0f3-555bf3213f85-registry-tls podName:65250207-5996-4ed6-b0f3-555bf3213f85 nodeName:}" failed. No retries permitted until 2026-09-18 13:45:18.209351939 +0000 UTC m=+67.262011244 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/65250207-5996-4ed6-b0f3-555bf3213f85-registry-tls") pod "image-registry-59dccbbcb7-ql9ft" (UID: "65250207-5996-4ed6-b0f3-555bf3213f85") : secret "image-registry-tls" not found
Sep 18 13:45:02.209442 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:45:02.209422    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 13:45:02.209538 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:45:02.209452    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce0ee0f8-b52b-4ddc-86f9-41f650a590b2-metrics-tls podName:ce0ee0f8-b52b-4ddc-86f9-41f650a590b2 nodeName:}" failed. No retries permitted until 2026-09-18 13:45:18.209442322 +0000 UTC m=+67.262101627 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ce0ee0f8-b52b-4ddc-86f9-41f650a590b2-metrics-tls") pod "dns-default-fcbr5" (UID: "ce0ee0f8-b52b-4ddc-86f9-41f650a590b2") : secret "dns-default-metrics-tls" not found
Sep 18 13:45:04.124753 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:04.124716    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/38fce007-dfca-4566-bea9-3e8c2ea32974-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-7g9jh\" (UID: \"38fce007-dfca-4566-bea9-3e8c2ea32974\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-7g9jh"
Sep 18 13:45:04.125248 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:45:04.124910    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 13:45:04.125248 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:45:04.124985    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38fce007-dfca-4566-bea9-3e8c2ea32974-cluster-monitoring-operator-tls podName:38fce007-dfca-4566-bea9-3e8c2ea32974 nodeName:}" failed. No retries permitted until 2026-09-18 13:45:08.124965824 +0000 UTC m=+57.177625144 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/38fce007-dfca-4566-bea9-3e8c2ea32974-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-7g9jh" (UID: "38fce007-dfca-4566-bea9-3e8c2ea32974") : secret "cluster-monitoring-operator-tls" not found
Sep 18 13:45:04.228807 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:04.226175    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/369561d5-6f05-4034-a7e1-24138261ca29-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-4xlbs\" (UID: \"369561d5-6f05-4034-a7e1-24138261ca29\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-4xlbs"
Sep 18 13:45:04.228807 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:45:04.226525    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 13:45:04.228807 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:45:04.226602    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/369561d5-6f05-4034-a7e1-24138261ca29-samples-operator-tls podName:369561d5-6f05-4034-a7e1-24138261ca29 nodeName:}" failed. No retries permitted until 2026-09-18 13:45:08.226581784 +0000 UTC m=+57.279241107 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/369561d5-6f05-4034-a7e1-24138261ca29-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-4xlbs" (UID: "369561d5-6f05-4034-a7e1-24138261ca29") : secret "samples-operator-tls" not found
Sep 18 13:45:04.909275 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:04.909185    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:04.972570 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:04.972535    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:05.238110 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:05.238033    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-6lm8v"]
Sep 18 13:45:05.240134 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:05.240114    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6lm8v"
Sep 18 13:45:05.242346 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:05.242128    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 18 13:45:05.242346 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:05.242159    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 18 13:45:05.242346 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:05.242194    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 18 13:45:05.242567 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:05.242484    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-gfxkd\""
Sep 18 13:45:05.242567 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:05.242529    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 18 13:45:05.251422 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:05.251399    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6lm8v"]
Sep 18 13:45:05.268320 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:05.268298    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:05.335598 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:05.335573    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9d9c9436-8b6c-4966-95f5-2cf258e95cc4-crio-socket\") pod \"insights-runtime-extractor-6lm8v\" (UID: \"9d9c9436-8b6c-4966-95f5-2cf258e95cc4\") " pod="openshift-insights/insights-runtime-extractor-6lm8v"
Sep 18 13:45:05.335753 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:05.335604    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9d9c9436-8b6c-4966-95f5-2cf258e95cc4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6lm8v\" (UID: \"9d9c9436-8b6c-4966-95f5-2cf258e95cc4\") " pod="openshift-insights/insights-runtime-extractor-6lm8v"
Sep 18 13:45:05.335753 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:05.335640    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbq6d\" (UniqueName: \"kubernetes.io/projected/9d9c9436-8b6c-4966-95f5-2cf258e95cc4-kube-api-access-cbq6d\") pod \"insights-runtime-extractor-6lm8v\" (UID: \"9d9c9436-8b6c-4966-95f5-2cf258e95cc4\") " pod="openshift-insights/insights-runtime-extractor-6lm8v"
Sep 18 13:45:05.335753 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:05.335692    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d9c9436-8b6c-4966-95f5-2cf258e95cc4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6lm8v\" (UID: \"9d9c9436-8b6c-4966-95f5-2cf258e95cc4\") " pod="openshift-insights/insights-runtime-extractor-6lm8v"
Sep 18 13:45:05.335884 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:05.335842    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9d9c9436-8b6c-4966-95f5-2cf258e95cc4-data-volume\") pod \"insights-runtime-extractor-6lm8v\" (UID: \"9d9c9436-8b6c-4966-95f5-2cf258e95cc4\") " pod="openshift-insights/insights-runtime-extractor-6lm8v"
Sep 18 13:45:05.362266 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:05.362240    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:05.377046 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:05.377016    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-v99sv" event={"ID":"a20817cc-9558-434d-9061-86be074f9ad2","Type":"ContainerStarted","Data":"46cc6d4d3c4d13fba11a8a4ed55daf5ab54d53e5e6f49b28594947015610a579"}
Sep 18 13:45:05.378766 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:05.378749    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-8tnxl_69d3178a-5769-4b5f-a243-30a1bedc5bd6/console-operator/0.log"
Sep 18 13:45:05.378878 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:05.378783    2577 generic.go:356] "Generic (PLEG): container finished" podID="69d3178a-5769-4b5f-a243-30a1bedc5bd6" containerID="467a0c7e4cfff7a587d706bd2a171fe30c6b25065e6305ac121aa2517530a183" exitCode=255
Sep 18 13:45:05.378944 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:05.378881    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-8tnxl" event={"ID":"69d3178a-5769-4b5f-a243-30a1bedc5bd6","Type":"ContainerDied","Data":"467a0c7e4cfff7a587d706bd2a171fe30c6b25065e6305ac121aa2517530a183"}
Sep 18 13:45:05.379080 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:05.379065    2577 scope.go:117] "RemoveContainer" containerID="467a0c7e4cfff7a587d706bd2a171fe30c6b25065e6305ac121aa2517530a183"
Sep 18 13:45:05.380451 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:05.380433    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-fvx98" event={"ID":"b47b0c48-ce7a-4511-8f7c-d10cb8dd83b8","Type":"ContainerStarted","Data":"be98eed58155768360dca35953c034498b707988cd5f7e6e2f78f7ff82a9e206"}
Sep 18 13:45:05.384488 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:05.384468    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-b77cp"]
Sep 18 13:45:05.386597 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:05.386581    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-b77cp"
Sep 18 13:45:05.390069 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:05.390040    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 18 13:45:05.390168 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:05.390152    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 18 13:45:05.390321 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:05.390304    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-6qtgk\""
Sep 18 13:45:05.398536 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:05.398513    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-b77cp"]
Sep 18 13:45:05.401018 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:05.400974    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-v99sv" podStartSLOduration=1.84970757 podStartE2EDuration="5.400961284s" totalImagesPullingTime="3.551253714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:45:05.400366331 +0000 UTC m=+54.453025659" watchObservedRunningTime="2026-09-18 13:45:05.400961284 +0000 UTC m=+54.453620661"
Sep 18 13:45:05.436576 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:05.436546    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9d9c9436-8b6c-4966-95f5-2cf258e95cc4-data-volume\") pod \"insights-runtime-extractor-6lm8v\" (UID: \"9d9c9436-8b6c-4966-95f5-2cf258e95cc4\") " pod="openshift-insights/insights-runtime-extractor-6lm8v"
Sep 18 13:45:05.436699 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:05.436627    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9d9c9436-8b6c-4966-95f5-2cf258e95cc4-crio-socket\") pod \"insights-runtime-extractor-6lm8v\" (UID: \"9d9c9436-8b6c-4966-95f5-2cf258e95cc4\") " pod="openshift-insights/insights-runtime-extractor-6lm8v"
Sep 18 13:45:05.436699 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:05.436654    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9d9c9436-8b6c-4966-95f5-2cf258e95cc4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6lm8v\" (UID: \"9d9c9436-8b6c-4966-95f5-2cf258e95cc4\") " pod="openshift-insights/insights-runtime-extractor-6lm8v"
Sep 18 13:45:05.436812 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:05.436705    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbq6d\" (UniqueName: \"kubernetes.io/projected/9d9c9436-8b6c-4966-95f5-2cf258e95cc4-kube-api-access-cbq6d\") pod \"insights-runtime-extractor-6lm8v\" (UID: \"9d9c9436-8b6c-4966-95f5-2cf258e95cc4\") " pod="openshift-insights/insights-runtime-extractor-6lm8v"
Sep 18 13:45:05.436812 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:05.436731    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d9c9436-8b6c-4966-95f5-2cf258e95cc4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6lm8v\" (UID: \"9d9c9436-8b6c-4966-95f5-2cf258e95cc4\") " pod="openshift-insights/insights-runtime-extractor-6lm8v"
Sep 18 13:45:05.436943 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:45:05.436927    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 13:45:05.436998 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:05.436950    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9d9c9436-8b6c-4966-95f5-2cf258e95cc4-data-volume\") pod \"insights-runtime-extractor-6lm8v\" (UID: \"9d9c9436-8b6c-4966-95f5-2cf258e95cc4\") " pod="openshift-insights/insights-runtime-extractor-6lm8v"
Sep 18 13:45:05.436998 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:45:05.436980    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d9c9436-8b6c-4966-95f5-2cf258e95cc4-insights-runtime-extractor-tls podName:9d9c9436-8b6c-4966-95f5-2cf258e95cc4 nodeName:}" failed. No retries permitted until 2026-09-18 13:45:05.936961811 +0000 UTC m=+54.989621123 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9d9c9436-8b6c-4966-95f5-2cf258e95cc4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6lm8v" (UID: "9d9c9436-8b6c-4966-95f5-2cf258e95cc4") : secret "insights-runtime-extractor-tls" not found
Sep 18 13:45:05.437109 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:45:05.437091    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 13:45:05.437158 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:45:05.437151    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9d9c9436-8b6c-4966-95f5-2cf258e95cc4-kube-rbac-proxy-cm podName:9d9c9436-8b6c-4966-95f5-2cf258e95cc4 nodeName:}" failed. No retries permitted until 2026-09-18 13:45:05.937134075 +0000 UTC m=+54.989793400 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9d9c9436-8b6c-4966-95f5-2cf258e95cc4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6lm8v" (UID: "9d9c9436-8b6c-4966-95f5-2cf258e95cc4") : configmap "kube-rbac-proxy" not found
Sep 18 13:45:05.437271 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:05.437253    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9d9c9436-8b6c-4966-95f5-2cf258e95cc4-crio-socket\") pod \"insights-runtime-extractor-6lm8v\" (UID: \"9d9c9436-8b6c-4966-95f5-2cf258e95cc4\") " pod="openshift-insights/insights-runtime-extractor-6lm8v"
Sep 18 13:45:05.439759 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:05.439712    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-fvx98" podStartSLOduration=1.831760655 podStartE2EDuration="5.439696563s" totalImagesPullingTime="3.607935908s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:45:05.438202556 +0000 UTC m=+54.490861883" watchObservedRunningTime="2026-09-18 13:45:05.439696563 +0000 UTC m=+54.492355894"
Sep 18 13:45:05.445339 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:05.445292    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbq6d\" (UniqueName: \"kubernetes.io/projected/9d9c9436-8b6c-4966-95f5-2cf258e95cc4-kube-api-access-cbq6d\") pod \"insights-runtime-extractor-6lm8v\" (UID: \"9d9c9436-8b6c-4966-95f5-2cf258e95cc4\") " pod="openshift-insights/insights-runtime-extractor-6lm8v"
Sep 18 13:45:05.445636 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:05.445611    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:05.518197 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:05.518142    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:05.537596 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:05.537563    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjnxz\" (UniqueName: \"kubernetes.io/projected/0c3e7539-d533-47e4-ad17-dbe6fccdd6ba-kube-api-access-qjnxz\") pod \"migrator-76794cbd6c-b77cp\" (UID: \"0c3e7539-d533-47e4-ad17-dbe6fccdd6ba\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-b77cp"
Sep 18 13:45:05.638414 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:05.638380    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjnxz\" (UniqueName: \"kubernetes.io/projected/0c3e7539-d533-47e4-ad17-dbe6fccdd6ba-kube-api-access-qjnxz\") pod \"migrator-76794cbd6c-b77cp\" (UID: \"0c3e7539-d533-47e4-ad17-dbe6fccdd6ba\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-b77cp"
Sep 18 13:45:05.646812 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:05.646786    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjnxz\" (UniqueName: \"kubernetes.io/projected/0c3e7539-d533-47e4-ad17-dbe6fccdd6ba-kube-api-access-qjnxz\") pod \"migrator-76794cbd6c-b77cp\" (UID: \"0c3e7539-d533-47e4-ad17-dbe6fccdd6ba\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-b77cp"
Sep 18 13:45:05.710672 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:05.710644    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-b77cp"
Sep 18 13:45:05.753309 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:05.753253    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:05.853390 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:05.853358    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:05.940424 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:05.940386    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9d9c9436-8b6c-4966-95f5-2cf258e95cc4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6lm8v\" (UID: \"9d9c9436-8b6c-4966-95f5-2cf258e95cc4\") " pod="openshift-insights/insights-runtime-extractor-6lm8v"
Sep 18 13:45:05.940577 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:05.940439    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d9c9436-8b6c-4966-95f5-2cf258e95cc4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6lm8v\" (UID: \"9d9c9436-8b6c-4966-95f5-2cf258e95cc4\") " pod="openshift-insights/insights-runtime-extractor-6lm8v"
Sep 18 13:45:05.940577 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:45:05.940510    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 13:45:05.940577 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:45:05.940552    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 13:45:05.940680 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:45:05.940583    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9d9c9436-8b6c-4966-95f5-2cf258e95cc4-kube-rbac-proxy-cm podName:9d9c9436-8b6c-4966-95f5-2cf258e95cc4 nodeName:}" failed. No retries permitted until 2026-09-18 13:45:06.940565608 +0000 UTC m=+55.993224925 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9d9c9436-8b6c-4966-95f5-2cf258e95cc4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6lm8v" (UID: "9d9c9436-8b6c-4966-95f5-2cf258e95cc4") : configmap "kube-rbac-proxy" not found
Sep 18 13:45:05.940680 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:45:05.940598    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d9c9436-8b6c-4966-95f5-2cf258e95cc4-insights-runtime-extractor-tls podName:9d9c9436-8b6c-4966-95f5-2cf258e95cc4 nodeName:}" failed. No retries permitted until 2026-09-18 13:45:06.940592128 +0000 UTC m=+55.993251434 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9d9c9436-8b6c-4966-95f5-2cf258e95cc4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6lm8v" (UID: "9d9c9436-8b6c-4966-95f5-2cf258e95cc4") : secret "insights-runtime-extractor-tls" not found
Sep 18 13:45:05.978201 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:05.978013    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-b77cp"]
Sep 18 13:45:05.980903 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:45:05.980879    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c3e7539_d533_47e4_ad17_dbe6fccdd6ba.slice/crio-e1607c117e2445ce5abd2e4c83f4e8f8bd68744935aa57ce24a9a677cc1c2dc9 WatchSource:0}: Error finding container e1607c117e2445ce5abd2e4c83f4e8f8bd68744935aa57ce24a9a677cc1c2dc9: Status 404 returned error can't find the container with id e1607c117e2445ce5abd2e4c83f4e8f8bd68744935aa57ce24a9a677cc1c2dc9
Sep 18 13:45:06.189291 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:06.189211    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:06.290390 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:06.290357    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:06.384347 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:06.384310    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-b77cp" event={"ID":"0c3e7539-d533-47e4-ad17-dbe6fccdd6ba","Type":"ContainerStarted","Data":"e1607c117e2445ce5abd2e4c83f4e8f8bd68744935aa57ce24a9a677cc1c2dc9"}
Sep 18 13:45:06.385786 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:06.385767    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-8tnxl_69d3178a-5769-4b5f-a243-30a1bedc5bd6/console-operator/1.log"
Sep 18 13:45:06.386208 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:06.386193    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-8tnxl_69d3178a-5769-4b5f-a243-30a1bedc5bd6/console-operator/0.log"
Sep 18 13:45:06.386270 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:06.386224    2577 generic.go:356] "Generic (PLEG): container finished" podID="69d3178a-5769-4b5f-a243-30a1bedc5bd6" containerID="0cac49ff1e95aaecdf9b815c5841cd55d51312c78d10834c1eab9174722fb097" exitCode=255
Sep 18 13:45:06.386270 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:06.386252    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-8tnxl" event={"ID":"69d3178a-5769-4b5f-a243-30a1bedc5bd6","Type":"ContainerDied","Data":"0cac49ff1e95aaecdf9b815c5841cd55d51312c78d10834c1eab9174722fb097"}
Sep 18 13:45:06.386356 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:06.386306    2577 scope.go:117] "RemoveContainer" containerID="467a0c7e4cfff7a587d706bd2a171fe30c6b25065e6305ac121aa2517530a183"
Sep 18 13:45:06.386536 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:06.386519    2577 scope.go:117] "RemoveContainer" containerID="0cac49ff1e95aaecdf9b815c5841cd55d51312c78d10834c1eab9174722fb097"
Sep 18 13:45:06.386752 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:45:06.386729    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-8tnxl_openshift-console-operator(69d3178a-5769-4b5f-a243-30a1bedc5bd6)\"" pod="openshift-console-operator/console-operator-fd4bb46c-8tnxl" podUID="69d3178a-5769-4b5f-a243-30a1bedc5bd6"
Sep 18 13:45:06.458130 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:06.458047    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:06.560767 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:06.560730    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:06.807156 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:06.807050    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vjv4c_dbd84791-daa9-43f0-9f32-8f4415eceeca/dns-node-resolver/0.log"
Sep 18 13:45:06.949201 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:06.949159    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9d9c9436-8b6c-4966-95f5-2cf258e95cc4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6lm8v\" (UID: \"9d9c9436-8b6c-4966-95f5-2cf258e95cc4\") " pod="openshift-insights/insights-runtime-extractor-6lm8v"
Sep 18 13:45:06.949372 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:06.949238    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d9c9436-8b6c-4966-95f5-2cf258e95cc4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6lm8v\" (UID: \"9d9c9436-8b6c-4966-95f5-2cf258e95cc4\") " pod="openshift-insights/insights-runtime-extractor-6lm8v"
Sep 18 13:45:06.949372 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:45:06.949247    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 13:45:06.949372 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:45:06.949343    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9d9c9436-8b6c-4966-95f5-2cf258e95cc4-kube-rbac-proxy-cm podName:9d9c9436-8b6c-4966-95f5-2cf258e95cc4 nodeName:}" failed. No retries permitted until 2026-09-18 13:45:08.949315957 +0000 UTC m=+58.001975303 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9d9c9436-8b6c-4966-95f5-2cf258e95cc4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6lm8v" (UID: "9d9c9436-8b6c-4966-95f5-2cf258e95cc4") : configmap "kube-rbac-proxy" not found
Sep 18 13:45:06.949559 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:45:06.949385    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 13:45:06.949559 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:45:06.949448    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d9c9436-8b6c-4966-95f5-2cf258e95cc4-insights-runtime-extractor-tls podName:9d9c9436-8b6c-4966-95f5-2cf258e95cc4 nodeName:}" failed. No retries permitted until 2026-09-18 13:45:08.9494304 +0000 UTC m=+58.002089723 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9d9c9436-8b6c-4966-95f5-2cf258e95cc4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6lm8v" (UID: "9d9c9436-8b6c-4966-95f5-2cf258e95cc4") : secret "insights-runtime-extractor-tls" not found
Sep 18 13:45:07.391117 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:07.391090    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-8tnxl_69d3178a-5769-4b5f-a243-30a1bedc5bd6/console-operator/1.log"
Sep 18 13:45:07.391555 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:07.391452    2577 scope.go:117] "RemoveContainer" containerID="0cac49ff1e95aaecdf9b815c5841cd55d51312c78d10834c1eab9174722fb097"
Sep 18 13:45:07.391650 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:45:07.391625    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-8tnxl_openshift-console-operator(69d3178a-5769-4b5f-a243-30a1bedc5bd6)\"" pod="openshift-console-operator/console-operator-fd4bb46c-8tnxl" podUID="69d3178a-5769-4b5f-a243-30a1bedc5bd6"
Sep 18 13:45:07.393104 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:07.393082    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-b77cp" event={"ID":"0c3e7539-d533-47e4-ad17-dbe6fccdd6ba","Type":"ContainerStarted","Data":"f1c321e192519f556437a38394956a23addd2fe774e38840de113548831a6c00"}
Sep 18 13:45:07.393199 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:07.393123    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-b77cp" event={"ID":"0c3e7539-d533-47e4-ad17-dbe6fccdd6ba","Type":"ContainerStarted","Data":"be5d6e27dbc702545963b36e66c1538f1b6730ce2195f96918432b2503d4dc39"}
Sep 18 13:45:07.422174 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:07.422113    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-b77cp" podStartSLOduration=1.3233322570000001 podStartE2EDuration="2.422096526s" totalImagesPullingTime="1.098764269s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:45:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:45:07.421169006 +0000 UTC m=+56.473828332" watchObservedRunningTime="2026-09-18 13:45:07.422096526 +0000 UTC m=+56.474755854"
Sep 18 13:45:07.495971 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:07.495938    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:07.561900 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:07.561869    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:08.008965 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:08.008937    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wczf6_ac788d27-88a8-4ed2-9363-fe4f0aa51013/node-ca/0.log"
Sep 18 13:45:08.092901 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:08.092855    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:08.151049 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:08.151021    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:08.161257 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:08.161226    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/38fce007-dfca-4566-bea9-3e8c2ea32974-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-7g9jh\" (UID: \"38fce007-dfca-4566-bea9-3e8c2ea32974\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-7g9jh"
Sep 18 13:45:08.161385 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:45:08.161367    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 13:45:08.161450 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:45:08.161440    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38fce007-dfca-4566-bea9-3e8c2ea32974-cluster-monitoring-operator-tls podName:38fce007-dfca-4566-bea9-3e8c2ea32974 nodeName:}" failed. No retries permitted until 2026-09-18 13:45:16.161426253 +0000 UTC m=+65.214085564 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/38fce007-dfca-4566-bea9-3e8c2ea32974-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-7g9jh" (UID: "38fce007-dfca-4566-bea9-3e8c2ea32974") : secret "cluster-monitoring-operator-tls" not found
Sep 18 13:45:08.262256 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:08.262169    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/369561d5-6f05-4034-a7e1-24138261ca29-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-4xlbs\" (UID: \"369561d5-6f05-4034-a7e1-24138261ca29\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-4xlbs"
Sep 18 13:45:08.262396 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:45:08.262299    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 13:45:08.262396 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:45:08.262355    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/369561d5-6f05-4034-a7e1-24138261ca29-samples-operator-tls podName:369561d5-6f05-4034-a7e1-24138261ca29 nodeName:}" failed. No retries permitted until 2026-09-18 13:45:16.262339285 +0000 UTC m=+65.314998589 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/369561d5-6f05-4034-a7e1-24138261ca29-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-4xlbs" (UID: "369561d5-6f05-4034-a7e1-24138261ca29") : secret "samples-operator-tls" not found
Sep 18 13:45:08.607291 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:08.607265    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hlqkp_e5510d2c-e7d9-4512-934a-a2044c0ba084/serve-healthcheck-canary/0.log"
Sep 18 13:45:08.968436 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:08.968338    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9d9c9436-8b6c-4966-95f5-2cf258e95cc4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6lm8v\" (UID: \"9d9c9436-8b6c-4966-95f5-2cf258e95cc4\") " pod="openshift-insights/insights-runtime-extractor-6lm8v"
Sep 18 13:45:08.968436 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:08.968396    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d9c9436-8b6c-4966-95f5-2cf258e95cc4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6lm8v\" (UID: \"9d9c9436-8b6c-4966-95f5-2cf258e95cc4\") " pod="openshift-insights/insights-runtime-extractor-6lm8v"
Sep 18 13:45:08.968661 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:45:08.968480    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 13:45:08.968661 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:45:08.968569    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 13:45:08.968661 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:45:08.968583    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9d9c9436-8b6c-4966-95f5-2cf258e95cc4-kube-rbac-proxy-cm podName:9d9c9436-8b6c-4966-95f5-2cf258e95cc4 nodeName:}" failed. No retries permitted until 2026-09-18 13:45:12.968560759 +0000 UTC m=+62.021220078 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9d9c9436-8b6c-4966-95f5-2cf258e95cc4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6lm8v" (UID: "9d9c9436-8b6c-4966-95f5-2cf258e95cc4") : configmap "kube-rbac-proxy" not found
Sep 18 13:45:08.968661 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:45:08.968631    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d9c9436-8b6c-4966-95f5-2cf258e95cc4-insights-runtime-extractor-tls podName:9d9c9436-8b6c-4966-95f5-2cf258e95cc4 nodeName:}" failed. No retries permitted until 2026-09-18 13:45:12.968614348 +0000 UTC m=+62.021273672 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9d9c9436-8b6c-4966-95f5-2cf258e95cc4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6lm8v" (UID: "9d9c9436-8b6c-4966-95f5-2cf258e95cc4") : secret "insights-runtime-extractor-tls" not found
Sep 18 13:45:09.084931 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:09.084896    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:10.645818 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:10.645782    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-8tnxl"
Sep 18 13:45:10.646194 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:10.645855    2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-8tnxl"
Sep 18 13:45:10.646194 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:10.646179    2577 scope.go:117] "RemoveContainer" containerID="0cac49ff1e95aaecdf9b815c5841cd55d51312c78d10834c1eab9174722fb097"
Sep 18 13:45:10.646357 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:45:10.646337    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-8tnxl_openshift-console-operator(69d3178a-5769-4b5f-a243-30a1bedc5bd6)\"" pod="openshift-console-operator/console-operator-fd4bb46c-8tnxl" podUID="69d3178a-5769-4b5f-a243-30a1bedc5bd6"
Sep 18 13:45:12.304781 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:12.304748    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:12.311277 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:12.311252    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wbv96"
Sep 18 13:45:12.406350 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:12.406310    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:12.471323 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:12.471291    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:12.680539 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:12.680491    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:12.819750 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:12.819717    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:12.993740 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:12.993667    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:12.998704 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:12.998680    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d9c9436-8b6c-4966-95f5-2cf258e95cc4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6lm8v\" (UID: \"9d9c9436-8b6c-4966-95f5-2cf258e95cc4\") " pod="openshift-insights/insights-runtime-extractor-6lm8v"
Sep 18 13:45:12.998848 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:12.998787    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9d9c9436-8b6c-4966-95f5-2cf258e95cc4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6lm8v\" (UID: \"9d9c9436-8b6c-4966-95f5-2cf258e95cc4\") " pod="openshift-insights/insights-runtime-extractor-6lm8v"
Sep 18 13:45:12.998848 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:45:12.998830    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 13:45:12.998924 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:45:12.998869    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 13:45:12.998924 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:45:12.998884    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d9c9436-8b6c-4966-95f5-2cf258e95cc4-insights-runtime-extractor-tls podName:9d9c9436-8b6c-4966-95f5-2cf258e95cc4 nodeName:}" failed. No retries permitted until 2026-09-18 13:45:20.998869801 +0000 UTC m=+70.051529106 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9d9c9436-8b6c-4966-95f5-2cf258e95cc4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6lm8v" (UID: "9d9c9436-8b6c-4966-95f5-2cf258e95cc4") : secret "insights-runtime-extractor-tls" not found
Sep 18 13:45:12.998924 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:45:12.998906    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9d9c9436-8b6c-4966-95f5-2cf258e95cc4-kube-rbac-proxy-cm podName:9d9c9436-8b6c-4966-95f5-2cf258e95cc4 nodeName:}" failed. No retries permitted until 2026-09-18 13:45:20.998896467 +0000 UTC m=+70.051555773 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9d9c9436-8b6c-4966-95f5-2cf258e95cc4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6lm8v" (UID: "9d9c9436-8b6c-4966-95f5-2cf258e95cc4") : configmap "kube-rbac-proxy" not found
Sep 18 13:45:14.191036 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:14.191001    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:14.393709 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:14.393651    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:14.450238 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:14.450161    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:14.584808 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:14.584767    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:14.826985 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:14.826882    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:14.994944 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:14.994899    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:15.894085 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:15.894052    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:15.993868 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:15.993842    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:16.186316 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:16.186228    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:16.224596 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:16.224562    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/38fce007-dfca-4566-bea9-3e8c2ea32974-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-7g9jh\" (UID: \"38fce007-dfca-4566-bea9-3e8c2ea32974\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-7g9jh"
Sep 18 13:45:16.224747 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:45:16.224708    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 13:45:16.224789 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:45:16.224772    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38fce007-dfca-4566-bea9-3e8c2ea32974-cluster-monitoring-operator-tls podName:38fce007-dfca-4566-bea9-3e8c2ea32974 nodeName:}" failed. No retries permitted until 2026-09-18 13:45:32.224756669 +0000 UTC m=+81.277415977 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/38fce007-dfca-4566-bea9-3e8c2ea32974-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-7g9jh" (UID: "38fce007-dfca-4566-bea9-3e8c2ea32974") : secret "cluster-monitoring-operator-tls" not found
Sep 18 13:45:16.325720 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:16.325683    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/369561d5-6f05-4034-a7e1-24138261ca29-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-4xlbs\" (UID: \"369561d5-6f05-4034-a7e1-24138261ca29\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-4xlbs"
Sep 18 13:45:16.328102 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:16.328072    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/369561d5-6f05-4034-a7e1-24138261ca29-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-4xlbs\" (UID: \"369561d5-6f05-4034-a7e1-24138261ca29\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-4xlbs"
Sep 18 13:45:16.394036 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:16.394005    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:16.589670 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:16.589639    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:16.625900 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:16.625877    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-q9rrm\""
Sep 18 13:45:16.634585 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:16.634558    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-4xlbs"
Sep 18 13:45:16.783809 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:16.783771    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:16.839791 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:16.839709    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:16.908376 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:16.908344    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-4xlbs"]
Sep 18 13:45:16.992931 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:16.992904    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:17.424560 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:17.424522    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-4xlbs" event={"ID":"369561d5-6f05-4034-a7e1-24138261ca29","Type":"ContainerStarted","Data":"4d2a1b01ca6937fc5c566230b66a00e16f070c7ce08bcf4857239ce09efb2988"}
Sep 18 13:45:17.839685 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:17.839654    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87eba103-cfec-4000-a10c-fd4fd512ac90-metrics-certs\") pod \"network-metrics-daemon-nb9bg\" (UID: \"87eba103-cfec-4000-a10c-fd4fd512ac90\") " pod="openshift-multus/network-metrics-daemon-nb9bg"
Sep 18 13:45:17.839882 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:17.839697    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/802e0a84-ff96-44d6-9aa2-db34555f46eb-original-pull-secret\") pod \"global-pull-secret-syncer-bdwmn\" (UID: \"802e0a84-ff96-44d6-9aa2-db34555f46eb\") " pod="kube-system/global-pull-secret-syncer-bdwmn"
Sep 18 13:45:17.842177 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:17.842146    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 13:45:17.842301 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:17.842174    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 13:45:17.850045 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:45:17.850023    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 18 13:45:17.850159 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:45:17.850095    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87eba103-cfec-4000-a10c-fd4fd512ac90-metrics-certs podName:87eba103-cfec-4000-a10c-fd4fd512ac90 nodeName:}" failed. No retries permitted until 2026-09-18 13:46:21.850075285 +0000 UTC m=+130.902734593 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/87eba103-cfec-4000-a10c-fd4fd512ac90-metrics-certs") pod "network-metrics-daemon-nb9bg" (UID: "87eba103-cfec-4000-a10c-fd4fd512ac90") : secret "metrics-daemon-secret" not found
Sep 18 13:45:17.852554 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:17.852535    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/802e0a84-ff96-44d6-9aa2-db34555f46eb-original-pull-secret\") pod \"global-pull-secret-syncer-bdwmn\" (UID: \"802e0a84-ff96-44d6-9aa2-db34555f46eb\") " pod="kube-system/global-pull-secret-syncer-bdwmn"
Sep 18 13:45:17.940553 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:17.940518    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmqn6\" (UniqueName: \"kubernetes.io/projected/6f382898-0370-425b-8c01-6587a985e3f0-kube-api-access-dmqn6\") pod \"network-check-target-hj8lg\" (UID: \"6f382898-0370-425b-8c01-6587a985e3f0\") " pod="openshift-network-diagnostics/network-check-target-hj8lg"
Sep 18 13:45:17.943152 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:17.943128    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmqn6\" (UniqueName: \"kubernetes.io/projected/6f382898-0370-425b-8c01-6587a985e3f0-kube-api-access-dmqn6\") pod \"network-check-target-hj8lg\" (UID: \"6f382898-0370-425b-8c01-6587a985e3f0\") " pod="openshift-network-diagnostics/network-check-target-hj8lg"
Sep 18 13:45:17.992437 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:17.992408    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:18.113747 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:18.113672    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdwmn"
Sep 18 13:45:18.122037 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:18.122018    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8ktpw\""
Sep 18 13:45:18.130460 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:18.130437    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hj8lg"
Sep 18 13:45:18.186351 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:18.186285    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:18.242903 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:18.242873    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce0ee0f8-b52b-4ddc-86f9-41f650a590b2-metrics-tls\") pod \"dns-default-fcbr5\" (UID: \"ce0ee0f8-b52b-4ddc-86f9-41f650a590b2\") " pod="openshift-dns/dns-default-fcbr5"
Sep 18 13:45:18.243025 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:18.242948    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/65250207-5996-4ed6-b0f3-555bf3213f85-registry-tls\") pod \"image-registry-59dccbbcb7-ql9ft\" (UID: \"65250207-5996-4ed6-b0f3-555bf3213f85\") " pod="openshift-image-registry/image-registry-59dccbbcb7-ql9ft"
Sep 18 13:45:18.244945 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:18.244923    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce0ee0f8-b52b-4ddc-86f9-41f650a590b2-metrics-tls\") pod \"dns-default-fcbr5\" (UID: \"ce0ee0f8-b52b-4ddc-86f9-41f650a590b2\") " pod="openshift-dns/dns-default-fcbr5"
Sep 18 13:45:18.245092 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:18.245074    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/65250207-5996-4ed6-b0f3-555bf3213f85-registry-tls\") pod \"image-registry-59dccbbcb7-ql9ft\" (UID: \"65250207-5996-4ed6-b0f3-555bf3213f85\") " pod="openshift-image-registry/image-registry-59dccbbcb7-ql9ft"
Sep 18 13:45:18.382252 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:18.382174    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:18.401450 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:18.401420    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bdwmn"]
Sep 18 13:45:18.435593 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:18.435566    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hj8lg"]
Sep 18 13:45:18.440760 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:18.440732    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:18.516991 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:18.516956    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rp2z9\""
Sep 18 13:45:18.525779 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:18.525750    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59dccbbcb7-ql9ft"
Sep 18 13:45:18.533148 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:18.533127    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5gmms\""
Sep 18 13:45:18.541638 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:18.541616    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fcbr5"
Sep 18 13:45:18.572762 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:18.572732    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:18.786232 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:18.786157    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:18.801864 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:45:18.801836    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod802e0a84_ff96_44d6_9aa2_db34555f46eb.slice/crio-537f5ab6a3ce3095aa107a071c192aa66e0e0d0b10a2317bb6e24bb98ca01571 WatchSource:0}: Error finding container 537f5ab6a3ce3095aa107a071c192aa66e0e0d0b10a2317bb6e24bb98ca01571: Status 404 returned error can't find the container with id 537f5ab6a3ce3095aa107a071c192aa66e0e0d0b10a2317bb6e24bb98ca01571
Sep 18 13:45:18.803054 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:45:18.803026    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f382898_0370_425b_8c01_6587a985e3f0.slice/crio-be749abd7145dc825465455de5adb375d1fd0f6cf1e8e14b1b5b46b2dece8b75 WatchSource:0}: Error finding container be749abd7145dc825465455de5adb375d1fd0f6cf1e8e14b1b5b46b2dece8b75: Status 404 returned error can't find the container with id be749abd7145dc825465455de5adb375d1fd0f6cf1e8e14b1b5b46b2dece8b75
Sep 18 13:45:19.081546 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:19.081478    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fcbr5"]
Sep 18 13:45:19.084356 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:19.084332    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-59dccbbcb7-ql9ft"]
Sep 18 13:45:19.084882 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:45:19.084857    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce0ee0f8_b52b_4ddc_86f9_41f650a590b2.slice/crio-00c06e83c71537ff8d9790759ececd0e653571d15128e47100249f577c6f7c8b WatchSource:0}: Error finding container 00c06e83c71537ff8d9790759ececd0e653571d15128e47100249f577c6f7c8b: Status 404 returned error can't find the container with id 00c06e83c71537ff8d9790759ececd0e653571d15128e47100249f577c6f7c8b
Sep 18 13:45:19.087399 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:45:19.087300    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65250207_5996_4ed6_b0f3_555bf3213f85.slice/crio-9dcf87b4b7face6b73b84840a3d7771ec8da39f56243ae9f3182737dd8aaa754 WatchSource:0}: Error finding container 9dcf87b4b7face6b73b84840a3d7771ec8da39f56243ae9f3182737dd8aaa754: Status 404 returned error can't find the container with id 9dcf87b4b7face6b73b84840a3d7771ec8da39f56243ae9f3182737dd8aaa754
Sep 18 13:45:19.395801 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:19.395758    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:19.433624 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:19.433546    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hj8lg" event={"ID":"6f382898-0370-425b-8c01-6587a985e3f0","Type":"ContainerStarted","Data":"e4cc1192f67344db9525aed09be3f68f44aa5a3af26b77d17f3a736020298ed7"}
Sep 18 13:45:19.433624 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:19.433604    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hj8lg" event={"ID":"6f382898-0370-425b-8c01-6587a985e3f0","Type":"ContainerStarted","Data":"be749abd7145dc825465455de5adb375d1fd0f6cf1e8e14b1b5b46b2dece8b75"}
Sep 18 13:45:19.433624 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:19.433633    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-hj8lg"
Sep 18 13:45:19.435456 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:19.435425    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bdwmn" event={"ID":"802e0a84-ff96-44d6-9aa2-db34555f46eb","Type":"ContainerStarted","Data":"537f5ab6a3ce3095aa107a071c192aa66e0e0d0b10a2317bb6e24bb98ca01571"}
Sep 18 13:45:19.438057 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:19.437580    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-4xlbs" event={"ID":"369561d5-6f05-4034-a7e1-24138261ca29","Type":"ContainerStarted","Data":"919186d5b4191187d45048f0e7d46e3d1c3a0ca500f78f8128d7aa70385860e3"}
Sep 18 13:45:19.438057 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:19.437632    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-4xlbs" event={"ID":"369561d5-6f05-4034-a7e1-24138261ca29","Type":"ContainerStarted","Data":"eb8be075a025c05012b29b626cead94b6544c3a03656aadd3e256827d94bf085"}
Sep 18 13:45:19.439492 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:19.439467    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59dccbbcb7-ql9ft" event={"ID":"65250207-5996-4ed6-b0f3-555bf3213f85","Type":"ContainerStarted","Data":"7fabaa707dd322219b3f9daca278a5ab4f1b4adc4a5b4cb6245bf9354f5451a4"}
Sep 18 13:45:19.439628 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:19.439514    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59dccbbcb7-ql9ft" event={"ID":"65250207-5996-4ed6-b0f3-555bf3213f85","Type":"ContainerStarted","Data":"9dcf87b4b7face6b73b84840a3d7771ec8da39f56243ae9f3182737dd8aaa754"}
Sep 18 13:45:19.439628 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:19.439542    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-59dccbbcb7-ql9ft"
Sep 18 13:45:19.441085 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:19.441051    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fcbr5" event={"ID":"ce0ee0f8-b52b-4ddc-86f9-41f650a590b2","Type":"ContainerStarted","Data":"00c06e83c71537ff8d9790759ececd0e653571d15128e47100249f577c6f7c8b"}
Sep 18 13:45:19.451554 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:19.451469    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-hj8lg" podStartSLOduration=66.451454173 podStartE2EDuration="1m6.451454173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:44:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:45:19.450519476 +0000 UTC m=+68.503178817" watchObservedRunningTime="2026-09-18 13:45:19.451454173 +0000 UTC m=+68.504113503"
Sep 18 13:45:19.474435 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:19.470516    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-4xlbs" podStartSLOduration=17.596448496 podStartE2EDuration="19.470498548s" totalImagesPullingTime="1.874050052s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:45:19.469731255 +0000 UTC m=+68.522390585" watchObservedRunningTime="2026-09-18 13:45:19.470498548 +0000 UTC m=+68.523157876"
Sep 18 13:45:19.476123 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:19.476092    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:19.495192 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:19.495148    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-59dccbbcb7-ql9ft" podStartSLOduration=65.495133514 podStartE2EDuration="1m5.495133514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:44:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:45:19.494127046 +0000 UTC m=+68.546786376" watchObservedRunningTime="2026-09-18 13:45:19.495133514 +0000 UTC m=+68.547792840"
Sep 18 13:45:19.783871 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:19.783769    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:20.185737 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:20.185700    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:20.390711 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:20.390653    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:20.999719 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:20.999687    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:21.066017 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:21.065980    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d9c9436-8b6c-4966-95f5-2cf258e95cc4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6lm8v\" (UID: \"9d9c9436-8b6c-4966-95f5-2cf258e95cc4\") " pod="openshift-insights/insights-runtime-extractor-6lm8v"
Sep 18 13:45:21.066189 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:21.066095    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9d9c9436-8b6c-4966-95f5-2cf258e95cc4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6lm8v\" (UID: \"9d9c9436-8b6c-4966-95f5-2cf258e95cc4\") " pod="openshift-insights/insights-runtime-extractor-6lm8v"
Sep 18 13:45:21.066189 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:45:21.066125    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 13:45:21.066295 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:45:21.066198    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d9c9436-8b6c-4966-95f5-2cf258e95cc4-insights-runtime-extractor-tls podName:9d9c9436-8b6c-4966-95f5-2cf258e95cc4 nodeName:}" failed. No retries permitted until 2026-09-18 13:45:37.066181851 +0000 UTC m=+86.118841161 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9d9c9436-8b6c-4966-95f5-2cf258e95cc4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6lm8v" (UID: "9d9c9436-8b6c-4966-95f5-2cf258e95cc4") : secret "insights-runtime-extractor-tls" not found
Sep 18 13:45:21.066701 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:21.066680    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9d9c9436-8b6c-4966-95f5-2cf258e95cc4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6lm8v\" (UID: \"9d9c9436-8b6c-4966-95f5-2cf258e95cc4\") " pod="openshift-insights/insights-runtime-extractor-6lm8v"
Sep 18 13:45:21.383714 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:21.383666    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:21.451530 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:21.451500    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fcbr5" event={"ID":"ce0ee0f8-b52b-4ddc-86f9-41f650a590b2","Type":"ContainerStarted","Data":"9d7a8f986b66754bb3d6accd11356fe6dffe143588c1c84b80d50053821c7651"}
Sep 18 13:45:21.451684 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:21.451556    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fcbr5" event={"ID":"ce0ee0f8-b52b-4ddc-86f9-41f650a590b2","Type":"ContainerStarted","Data":"aefa49a3400fcd97c4076fcb1244c4dd6d1e26f903acdb8da7c0828b8f476ca4"}
Sep 18 13:45:21.451684 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:21.451620    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-fcbr5"
Sep 18 13:45:21.468937 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:21.468882    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-fcbr5" podStartSLOduration=33.731203592 podStartE2EDuration="35.468861971s" totalImagesPullingTime="1.737658379s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:44:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:45:21.468377653 +0000 UTC m=+70.521036981" watchObservedRunningTime="2026-09-18 13:45:21.468861971 +0000 UTC m=+70.521521299"
Sep 18 13:45:21.581010 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:21.580969    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:22.190592 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:22.190557    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:22.587513 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:22.587466    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:22.801943 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:22.801884    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:22.999326 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:22.999301    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:23.460910 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:23.460873    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bdwmn" event={"ID":"802e0a84-ff96-44d6-9aa2-db34555f46eb","Type":"ContainerStarted","Data":"718e897d955c3beacb759259f0071fff6d31c1656f12bed62f00f9d7ab10e910"}
Sep 18 13:45:24.445090 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:24.445056    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:24.514357 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:24.514324    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:25.101990 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:25.101954    2577 scope.go:117] "RemoveContainer" containerID="0cac49ff1e95aaecdf9b815c5841cd55d51312c78d10834c1eab9174722fb097"
Sep 18 13:45:25.468991 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:25.468920    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-8tnxl_69d3178a-5769-4b5f-a243-30a1bedc5bd6/console-operator/1.log"
Sep 18 13:45:25.469403 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:25.468988    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-8tnxl" event={"ID":"69d3178a-5769-4b5f-a243-30a1bedc5bd6","Type":"ContainerStarted","Data":"12dd56e5cee81de04ca8760a34b2fc20e8abbead50a25708b36ac28ae0a947d0"}
Sep 18 13:45:25.469403 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:25.469268    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-8tnxl"
Sep 18 13:45:25.489686 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:25.489637    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-bdwmn" podStartSLOduration=68.397322101 podStartE2EDuration="1m12.489618972s" totalImagesPullingTime="4.092296871s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:44:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:45:23.481046499 +0000 UTC m=+72.533705827" watchObservedRunningTime="2026-09-18 13:45:25.489618972 +0000 UTC m=+74.542278302"
Sep 18 13:45:25.490264 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:25.490235    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-fd4bb46c-8tnxl" podStartSLOduration=21.786368268 podStartE2EDuration="25.490226005s" totalImagesPullingTime="3.703857737s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:45:25.48922772 +0000 UTC m=+74.541887047" watchObservedRunningTime="2026-09-18 13:45:25.490226005 +0000 UTC m=+74.542885332"
Sep 18 13:45:25.611312 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:25.611283    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-fd4bb46c-8tnxl"
Sep 18 13:45:25.813972 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:25.813890    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:25.898870 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:25.898838    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:25.963589 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:25.963555    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:26.110536 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:26.110505    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:26.290105 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:26.290073    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:26.480716 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:26.480632    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:26.552236 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:26.552211    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:26.889958 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:26.889920    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:26.991070 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:26.991038    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:27.685786 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:27.685755    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:27.885553 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:27.885518    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:28.014605 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:28.014504    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:28.115468 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:28.115435    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:28.246124 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:28.246083    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:29.545067 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:29.545039    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54868fd5f6-hcqwl"]
Sep 18 13:45:29.548122 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:29.548105    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54868fd5f6-hcqwl"
Sep 18 13:45:29.550766 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:29.550743    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 18 13:45:29.550766 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:29.550743    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 18 13:45:29.551122 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:29.551105    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-7j9gx\""
Sep 18 13:45:29.551395 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:29.551381    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 18 13:45:29.554898 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:29.554880    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 18 13:45:29.569038 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:29.569012    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54868fd5f6-hcqwl"]
Sep 18 13:45:29.633649 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:29.633617    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6594f4c75-xhzgm"]
Sep 18 13:45:29.636636 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:29.636615    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-xhzgm"
Sep 18 13:45:29.636785 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:29.636610    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4ea0b468-36e3-4264-80b8-08f60ca9b71d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-54868fd5f6-hcqwl\" (UID: \"4ea0b468-36e3-4264-80b8-08f60ca9b71d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54868fd5f6-hcqwl"
Sep 18 13:45:29.636785 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:29.636712    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pql2t\" (UniqueName: \"kubernetes.io/projected/4ea0b468-36e3-4264-80b8-08f60ca9b71d-kube-api-access-pql2t\") pod \"managed-serviceaccount-addon-agent-54868fd5f6-hcqwl\" (UID: \"4ea0b468-36e3-4264-80b8-08f60ca9b71d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54868fd5f6-hcqwl"
Sep 18 13:45:29.638642 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:29.638624    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 18 13:45:29.638724 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:29.638625    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-8s2sh\""
Sep 18 13:45:29.638777 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:29.638762    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 18 13:45:29.652267 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:29.652243    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-xhzgm"]
Sep 18 13:45:29.737798 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:29.737765    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4ea0b468-36e3-4264-80b8-08f60ca9b71d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-54868fd5f6-hcqwl\" (UID: \"4ea0b468-36e3-4264-80b8-08f60ca9b71d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54868fd5f6-hcqwl"
Sep 18 13:45:29.737798 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:29.737800    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pql2t\" (UniqueName: \"kubernetes.io/projected/4ea0b468-36e3-4264-80b8-08f60ca9b71d-kube-api-access-pql2t\") pod \"managed-serviceaccount-addon-agent-54868fd5f6-hcqwl\" (UID: \"4ea0b468-36e3-4264-80b8-08f60ca9b71d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54868fd5f6-hcqwl"
Sep 18 13:45:29.738036 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:29.737954    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4xgg\" (UniqueName: \"kubernetes.io/projected/e549fe9c-3866-403f-bbd4-f72d0ec6bd70-kube-api-access-n4xgg\") pod \"downloads-6594f4c75-xhzgm\" (UID: \"e549fe9c-3866-403f-bbd4-f72d0ec6bd70\") " pod="openshift-console/downloads-6594f4c75-xhzgm"
Sep 18 13:45:29.740658 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:29.740622    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4ea0b468-36e3-4264-80b8-08f60ca9b71d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-54868fd5f6-hcqwl\" (UID: \"4ea0b468-36e3-4264-80b8-08f60ca9b71d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54868fd5f6-hcqwl"
Sep 18 13:45:29.763570 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:29.763539    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pql2t\" (UniqueName: \"kubernetes.io/projected/4ea0b468-36e3-4264-80b8-08f60ca9b71d-kube-api-access-pql2t\") pod \"managed-serviceaccount-addon-agent-54868fd5f6-hcqwl\" (UID: \"4ea0b468-36e3-4264-80b8-08f60ca9b71d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54868fd5f6-hcqwl"
Sep 18 13:45:29.768160 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:29.768134    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:29.839129 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:29.839102    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4xgg\" (UniqueName: \"kubernetes.io/projected/e549fe9c-3866-403f-bbd4-f72d0ec6bd70-kube-api-access-n4xgg\") pod \"downloads-6594f4c75-xhzgm\" (UID: \"e549fe9c-3866-403f-bbd4-f72d0ec6bd70\") " pod="openshift-console/downloads-6594f4c75-xhzgm"
Sep 18 13:45:29.859739 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:29.859705    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4xgg\" (UniqueName: \"kubernetes.io/projected/e549fe9c-3866-403f-bbd4-f72d0ec6bd70-kube-api-access-n4xgg\") pod \"downloads-6594f4c75-xhzgm\" (UID: \"e549fe9c-3866-403f-bbd4-f72d0ec6bd70\") " pod="openshift-console/downloads-6594f4c75-xhzgm"
Sep 18 13:45:29.865490 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:29.865473    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54868fd5f6-hcqwl"
Sep 18 13:45:29.902497 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:29.902461    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:29.945887 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:29.945858    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-xhzgm"
Sep 18 13:45:30.006396 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:30.006370    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:30.104234 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:30.104154    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:30.147697 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:30.147663    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54868fd5f6-hcqwl"]
Sep 18 13:45:30.151211 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:45:30.151189    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4ea0b468_36e3_4264_80b8_08f60ca9b71d.slice/crio-788a2138eacd4468ea6e49d1e72cad93ea8e49baa44445de90095a392f821d89 WatchSource:0}: Error finding container 788a2138eacd4468ea6e49d1e72cad93ea8e49baa44445de90095a392f821d89: Status 404 returned error can't find the container with id 788a2138eacd4468ea6e49d1e72cad93ea8e49baa44445de90095a392f821d89
Sep 18 13:45:30.216110 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:30.216081    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-xhzgm"]
Sep 18 13:45:30.220148 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:45:30.220115    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode549fe9c_3866_403f_bbd4_f72d0ec6bd70.slice/crio-cca97ad12f385b3f2989132540372a664b7a21d55f6d5e9a9afc7f3ada32b6cb WatchSource:0}: Error finding container cca97ad12f385b3f2989132540372a664b7a21d55f6d5e9a9afc7f3ada32b6cb: Status 404 returned error can't find the container with id cca97ad12f385b3f2989132540372a664b7a21d55f6d5e9a9afc7f3ada32b6cb
Sep 18 13:45:30.491775 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:30.491687    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-xhzgm" event={"ID":"e549fe9c-3866-403f-bbd4-f72d0ec6bd70","Type":"ContainerStarted","Data":"cca97ad12f385b3f2989132540372a664b7a21d55f6d5e9a9afc7f3ada32b6cb"}
Sep 18 13:45:30.492738 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:30.492715    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54868fd5f6-hcqwl" event={"ID":"4ea0b468-36e3-4264-80b8-08f60ca9b71d","Type":"ContainerStarted","Data":"788a2138eacd4468ea6e49d1e72cad93ea8e49baa44445de90095a392f821d89"}
Sep 18 13:45:31.458174 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:31.458139    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-fcbr5"
Sep 18 13:45:32.261011 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:32.260973    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/38fce007-dfca-4566-bea9-3e8c2ea32974-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-7g9jh\" (UID: \"38fce007-dfca-4566-bea9-3e8c2ea32974\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-7g9jh"
Sep 18 13:45:32.263874 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:32.263814    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/38fce007-dfca-4566-bea9-3e8c2ea32974-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-7g9jh\" (UID: \"38fce007-dfca-4566-bea9-3e8c2ea32974\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-7g9jh"
Sep 18 13:45:32.435385 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:32.435294    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-85jmd\""
Sep 18 13:45:32.444120 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:32.444058    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-7g9jh"
Sep 18 13:45:33.648695 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:33.648626    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-7g9jh"]
Sep 18 13:45:33.652086 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:45:33.652046    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38fce007_dfca_4566_bea9_3e8c2ea32974.slice/crio-85fad7d2eb12959f0f17716b9f44a5275f2075b6026426cf67c7e805064ff8aa WatchSource:0}: Error finding container 85fad7d2eb12959f0f17716b9f44a5275f2075b6026426cf67c7e805064ff8aa: Status 404 returned error can't find the container with id 85fad7d2eb12959f0f17716b9f44a5275f2075b6026426cf67c7e805064ff8aa
Sep 18 13:45:34.511630 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:34.511563    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-7g9jh" event={"ID":"38fce007-dfca-4566-bea9-3e8c2ea32974","Type":"ContainerStarted","Data":"85fad7d2eb12959f0f17716b9f44a5275f2075b6026426cf67c7e805064ff8aa"}
Sep 18 13:45:34.513973 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:34.513461    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54868fd5f6-hcqwl" event={"ID":"4ea0b468-36e3-4264-80b8-08f60ca9b71d","Type":"ContainerStarted","Data":"58a548afa7dab00abc14b367723f9eb389f3611252dd57931928388df4f9b2ab"}
Sep 18 13:45:34.528754 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:34.528702    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54868fd5f6-hcqwl" podStartSLOduration=2.23746714 podStartE2EDuration="5.528681095s" totalImagesPullingTime="3.291213955s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:45:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:45:34.528319363 +0000 UTC m=+83.580978699" watchObservedRunningTime="2026-09-18 13:45:34.528681095 +0000 UTC m=+83.581340415"
Sep 18 13:45:35.705209 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:35.705164    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:35.773078 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:35.773046    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:35.833832 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:35.833788    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:36.260937 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:36.260892    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:36.360962 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:36.360923    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:36.457669 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:36.457626    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:36.526836 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:36.526731    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-7g9jh" event={"ID":"38fce007-dfca-4566-bea9-3e8c2ea32974","Type":"ContainerStarted","Data":"7ae34283fc20196b68db6ec0b42f3a5e020b3baaf2e631d83b0605f1f91bc376"}
Sep 18 13:45:36.529305 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:36.529231    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:36.546569 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:36.546510    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-7g9jh" podStartSLOduration=34.335645087 podStartE2EDuration="36.546491724s" totalImagesPullingTime="2.210846637s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:45:36.545757563 +0000 UTC m=+85.598416891" watchObservedRunningTime="2026-09-18 13:45:36.546491724 +0000 UTC m=+85.599151052"
Sep 18 13:45:36.645163 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:36.645130    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:37.104166 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:37.104129    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d9c9436-8b6c-4966-95f5-2cf258e95cc4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6lm8v\" (UID: \"9d9c9436-8b6c-4966-95f5-2cf258e95cc4\") " pod="openshift-insights/insights-runtime-extractor-6lm8v"
Sep 18 13:45:37.106932 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:37.106905    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d9c9436-8b6c-4966-95f5-2cf258e95cc4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6lm8v\" (UID: \"9d9c9436-8b6c-4966-95f5-2cf258e95cc4\") " pod="openshift-insights/insights-runtime-extractor-6lm8v"
Sep 18 13:45:37.353552 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:37.353519    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-gfxkd\""
Sep 18 13:45:37.361708 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:37.361637    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6lm8v"
Sep 18 13:45:37.658106 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:37.658033    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6lm8v"]
Sep 18 13:45:37.662006 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:45:37.661974    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d9c9436_8b6c_4966_95f5_2cf258e95cc4.slice/crio-c0d4f202c3d73f8e3f62f30572a3f4369f82763a814609f3547af0c0e3ad45c0 WatchSource:0}: Error finding container c0d4f202c3d73f8e3f62f30572a3f4369f82763a814609f3547af0c0e3ad45c0: Status 404 returned error can't find the container with id c0d4f202c3d73f8e3f62f30572a3f4369f82763a814609f3547af0c0e3ad45c0
Sep 18 13:45:38.080778 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:38.080696    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:38.143468 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:38.143433    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:38.531841 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:38.531787    2577 patch_prober.go:28] interesting pod/image-registry-59dccbbcb7-ql9ft container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 18 13:45:38.532023 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:38.531889    2577 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-59dccbbcb7-ql9ft" podUID="65250207-5996-4ed6-b0f3-555bf3213f85" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 13:45:38.537503 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:38.537462    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6lm8v" event={"ID":"9d9c9436-8b6c-4966-95f5-2cf258e95cc4","Type":"ContainerStarted","Data":"203012cba4ead3ea84012ecfa7ef16a357c6dea77d0f061ec1606e4f016e3dd3"}
Sep 18 13:45:38.537652 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:38.537532    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6lm8v" event={"ID":"9d9c9436-8b6c-4966-95f5-2cf258e95cc4","Type":"ContainerStarted","Data":"c0d4f202c3d73f8e3f62f30572a3f4369f82763a814609f3547af0c0e3ad45c0"}
Sep 18 13:45:39.478722 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:39.478686    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:39.543312 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:39.543272    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6lm8v" event={"ID":"9d9c9436-8b6c-4966-95f5-2cf258e95cc4","Type":"ContainerStarted","Data":"99673170952af7eb15d6f01cff69a6b76437b1a5d130ed707060b08b3c8c5052"}
Sep 18 13:45:39.578505 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:39.578473    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:39.662854 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:39.662793    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:40.452811 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:40.452781    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-59dccbbcb7-ql9ft"
Sep 18 13:45:40.592673 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:40.592612    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:40.697375 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:40.697338    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:40.794575 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:40.794493    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:41.910552 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:41.910517    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:42.236349 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:42.236246    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:42.298002 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:42.297967    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:43.758505 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:43.758462    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:43.840784 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:43.840749    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:45.681125 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:45.681092    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:45.946413 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:45.946326    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:46.085051 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:46.084984    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:46.093506 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:46.093426    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-jpl99"]
Sep 18 13:45:46.097081 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:46.097056    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jpl99"
Sep 18 13:45:46.101391 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:46.101370    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 18 13:45:46.101750 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:46.101716    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vn8xp\""
Sep 18 13:45:46.102287 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:46.102264    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 18 13:45:46.102430 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:46.102407    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 18 13:45:46.103457 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:46.102869    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 18 13:45:46.181606 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:46.181568    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:46.186767 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:46.186681    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f8742812-3cdb-4df7-801b-443e5178cb24-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jpl99\" (UID: \"f8742812-3cdb-4df7-801b-443e5178cb24\") " pod="openshift-monitoring/node-exporter-jpl99"
Sep 18 13:45:46.187047 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:46.186786    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs2ps\" (UniqueName: \"kubernetes.io/projected/f8742812-3cdb-4df7-801b-443e5178cb24-kube-api-access-vs2ps\") pod \"node-exporter-jpl99\" (UID: \"f8742812-3cdb-4df7-801b-443e5178cb24\") " pod="openshift-monitoring/node-exporter-jpl99"
Sep 18 13:45:46.187047 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:46.186855    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f8742812-3cdb-4df7-801b-443e5178cb24-root\") pod \"node-exporter-jpl99\" (UID: \"f8742812-3cdb-4df7-801b-443e5178cb24\") " pod="openshift-monitoring/node-exporter-jpl99"
Sep 18 13:45:46.187047 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:46.186897    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f8742812-3cdb-4df7-801b-443e5178cb24-node-exporter-accelerators-collector-config\") pod \"node-exporter-jpl99\" (UID: \"f8742812-3cdb-4df7-801b-443e5178cb24\") " pod="openshift-monitoring/node-exporter-jpl99"
Sep 18 13:45:46.187047 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:46.186944    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f8742812-3cdb-4df7-801b-443e5178cb24-sys\") pod \"node-exporter-jpl99\" (UID: \"f8742812-3cdb-4df7-801b-443e5178cb24\") " pod="openshift-monitoring/node-exporter-jpl99"
Sep 18 13:45:46.187047 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:46.187004    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f8742812-3cdb-4df7-801b-443e5178cb24-node-exporter-textfile\") pod \"node-exporter-jpl99\" (UID: \"f8742812-3cdb-4df7-801b-443e5178cb24\") " pod="openshift-monitoring/node-exporter-jpl99"
Sep 18 13:45:46.187315 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:46.187128    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8742812-3cdb-4df7-801b-443e5178cb24-metrics-client-ca\") pod \"node-exporter-jpl99\" (UID: \"f8742812-3cdb-4df7-801b-443e5178cb24\") " pod="openshift-monitoring/node-exporter-jpl99"
Sep 18 13:45:46.187315 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:46.187197    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f8742812-3cdb-4df7-801b-443e5178cb24-node-exporter-tls\") pod \"node-exporter-jpl99\" (UID: \"f8742812-3cdb-4df7-801b-443e5178cb24\") " pod="openshift-monitoring/node-exporter-jpl99"
Sep 18 13:45:46.187315 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:46.187243    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f8742812-3cdb-4df7-801b-443e5178cb24-node-exporter-wtmp\") pod \"node-exporter-jpl99\" (UID: \"f8742812-3cdb-4df7-801b-443e5178cb24\") " pod="openshift-monitoring/node-exporter-jpl99"
Sep 18 13:45:46.287424 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:46.287308    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:46.287678 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:46.287647    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f8742812-3cdb-4df7-801b-443e5178cb24-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jpl99\" (UID: \"f8742812-3cdb-4df7-801b-443e5178cb24\") " pod="openshift-monitoring/node-exporter-jpl99"
Sep 18 13:45:46.287832 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:46.287694    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs2ps\" (UniqueName: \"kubernetes.io/projected/f8742812-3cdb-4df7-801b-443e5178cb24-kube-api-access-vs2ps\") pod \"node-exporter-jpl99\" (UID: \"f8742812-3cdb-4df7-801b-443e5178cb24\") " pod="openshift-monitoring/node-exporter-jpl99"
Sep 18 13:45:46.287832 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:46.287725    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f8742812-3cdb-4df7-801b-443e5178cb24-root\") pod \"node-exporter-jpl99\" (UID: \"f8742812-3cdb-4df7-801b-443e5178cb24\") " pod="openshift-monitoring/node-exporter-jpl99"
Sep 18 13:45:46.287832 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:46.287761    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f8742812-3cdb-4df7-801b-443e5178cb24-node-exporter-accelerators-collector-config\") pod \"node-exporter-jpl99\" (UID: \"f8742812-3cdb-4df7-801b-443e5178cb24\") " pod="openshift-monitoring/node-exporter-jpl99"
Sep 18 13:45:46.288017 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:46.287797    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f8742812-3cdb-4df7-801b-443e5178cb24-sys\") pod \"node-exporter-jpl99\" (UID: \"f8742812-3cdb-4df7-801b-443e5178cb24\") " pod="openshift-monitoring/node-exporter-jpl99"
Sep 18 13:45:46.288017 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:46.287879    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f8742812-3cdb-4df7-801b-443e5178cb24-node-exporter-textfile\") pod \"node-exporter-jpl99\" (UID: \"f8742812-3cdb-4df7-801b-443e5178cb24\") " pod="openshift-monitoring/node-exporter-jpl99"
Sep 18 13:45:46.288017 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:46.287887    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f8742812-3cdb-4df7-801b-443e5178cb24-root\") pod \"node-exporter-jpl99\" (UID: \"f8742812-3cdb-4df7-801b-443e5178cb24\") " pod="openshift-monitoring/node-exporter-jpl99"
Sep 18 13:45:46.288017 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:46.287904    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8742812-3cdb-4df7-801b-443e5178cb24-metrics-client-ca\") pod \"node-exporter-jpl99\" (UID: \"f8742812-3cdb-4df7-801b-443e5178cb24\") " pod="openshift-monitoring/node-exporter-jpl99"
Sep 18 13:45:46.288017 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:46.287961    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f8742812-3cdb-4df7-801b-443e5178cb24-node-exporter-tls\") pod \"node-exporter-jpl99\" (UID: \"f8742812-3cdb-4df7-801b-443e5178cb24\") " pod="openshift-monitoring/node-exporter-jpl99"
Sep 18 13:45:46.288017 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:46.287987    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f8742812-3cdb-4df7-801b-443e5178cb24-sys\") pod \"node-exporter-jpl99\" (UID: \"f8742812-3cdb-4df7-801b-443e5178cb24\") " pod="openshift-monitoring/node-exporter-jpl99"
Sep 18 13:45:46.288017 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:46.288007    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f8742812-3cdb-4df7-801b-443e5178cb24-node-exporter-wtmp\") pod \"node-exporter-jpl99\" (UID: \"f8742812-3cdb-4df7-801b-443e5178cb24\") " pod="openshift-monitoring/node-exporter-jpl99"
Sep 18 13:45:46.288335 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:45:46.288144    2577 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 18 13:45:46.288335 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:46.288164    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f8742812-3cdb-4df7-801b-443e5178cb24-node-exporter-wtmp\") pod \"node-exporter-jpl99\" (UID: \"f8742812-3cdb-4df7-801b-443e5178cb24\") " pod="openshift-monitoring/node-exporter-jpl99"
Sep 18 13:45:46.288335 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:45:46.288226    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8742812-3cdb-4df7-801b-443e5178cb24-node-exporter-tls podName:f8742812-3cdb-4df7-801b-443e5178cb24 nodeName:}" failed. No retries permitted until 2026-09-18 13:45:46.788205036 +0000 UTC m=+95.840864353 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/f8742812-3cdb-4df7-801b-443e5178cb24-node-exporter-tls") pod "node-exporter-jpl99" (UID: "f8742812-3cdb-4df7-801b-443e5178cb24") : secret "node-exporter-tls" not found
Sep 18 13:45:46.288497 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:46.288375    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f8742812-3cdb-4df7-801b-443e5178cb24-node-exporter-textfile\") pod \"node-exporter-jpl99\" (UID: \"f8742812-3cdb-4df7-801b-443e5178cb24\") " pod="openshift-monitoring/node-exporter-jpl99"
Sep 18 13:45:46.288554 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:46.288505    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8742812-3cdb-4df7-801b-443e5178cb24-metrics-client-ca\") pod \"node-exporter-jpl99\" (UID: \"f8742812-3cdb-4df7-801b-443e5178cb24\") " pod="openshift-monitoring/node-exporter-jpl99"
Sep 18 13:45:46.288608 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:46.288587    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f8742812-3cdb-4df7-801b-443e5178cb24-node-exporter-accelerators-collector-config\") pod \"node-exporter-jpl99\" (UID: \"f8742812-3cdb-4df7-801b-443e5178cb24\") " pod="openshift-monitoring/node-exporter-jpl99"
Sep 18 13:45:46.290784 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:46.290762    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f8742812-3cdb-4df7-801b-443e5178cb24-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jpl99\" (UID: \"f8742812-3cdb-4df7-801b-443e5178cb24\") " pod="openshift-monitoring/node-exporter-jpl99"
Sep 18 13:45:46.298538 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:46.298506    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs2ps\" (UniqueName: \"kubernetes.io/projected/f8742812-3cdb-4df7-801b-443e5178cb24-kube-api-access-vs2ps\") pod \"node-exporter-jpl99\" (UID: \"f8742812-3cdb-4df7-801b-443e5178cb24\") " pod="openshift-monitoring/node-exporter-jpl99"
Sep 18 13:45:46.469181 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:46.469148    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:46.571062 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:46.570970    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:46.669619 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:46.669585    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:46.792916 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:46.792878    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f8742812-3cdb-4df7-801b-443e5178cb24-node-exporter-tls\") pod \"node-exporter-jpl99\" (UID: \"f8742812-3cdb-4df7-801b-443e5178cb24\") " pod="openshift-monitoring/node-exporter-jpl99"
Sep 18 13:45:46.795166 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:46.795139    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f8742812-3cdb-4df7-801b-443e5178cb24-node-exporter-tls\") pod \"node-exporter-jpl99\" (UID: \"f8742812-3cdb-4df7-801b-443e5178cb24\") " pod="openshift-monitoring/node-exporter-jpl99"
Sep 18 13:45:46.922921 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:46.922889    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:47.013266 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:47.013231    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jpl99"
Sep 18 13:45:47.021477 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:47.021454    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:47.023589 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:45:47.023565    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8742812_3cdb_4df7_801b_443e5178cb24.slice/crio-fe9530c5aa5577d0582c7c6e941ee43d045fc1358dc19cd616d172bad87782fc WatchSource:0}: Error finding container fe9530c5aa5577d0582c7c6e941ee43d045fc1358dc19cd616d172bad87782fc: Status 404 returned error can't find the container with id fe9530c5aa5577d0582c7c6e941ee43d045fc1358dc19cd616d172bad87782fc
Sep 18 13:45:47.248434 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:47.248338    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:47.580959 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:47.579713    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-xhzgm" event={"ID":"e549fe9c-3866-403f-bbd4-f72d0ec6bd70","Type":"ContainerStarted","Data":"91740f76293f0b951bd4e65cdd991076058f7b0c921380a6d9d1201864d6a9f7"}
Sep 18 13:45:47.580959 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:47.580521    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6594f4c75-xhzgm"
Sep 18 13:45:47.582988 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:47.582948    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6lm8v" event={"ID":"9d9c9436-8b6c-4966-95f5-2cf258e95cc4","Type":"ContainerStarted","Data":"93c01b7fad2ce4859b43a8a6a262eddc37551c023350004815c2a8f668624d32"}
Sep 18 13:45:47.585557 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:47.585529    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jpl99" event={"ID":"f8742812-3cdb-4df7-801b-443e5178cb24","Type":"ContainerStarted","Data":"fe9530c5aa5577d0582c7c6e941ee43d045fc1358dc19cd616d172bad87782fc"}
Sep 18 13:45:47.597232 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:47.597208    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6594f4c75-xhzgm"
Sep 18 13:45:47.624873 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:47.624124    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-6594f4c75-xhzgm" podStartSLOduration=1.937743717 podStartE2EDuration="18.624104028s" totalImagesPullingTime="16.686360311s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:45:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:45:47.604661676 +0000 UTC m=+96.657321004" watchObservedRunningTime="2026-09-18 13:45:47.624104028 +0000 UTC m=+96.676763350"
Sep 18 13:45:48.592414 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:48.592368    2577 generic.go:356] "Generic (PLEG): container finished" podID="f8742812-3cdb-4df7-801b-443e5178cb24" containerID="51fb487f1f81c665889e28978d15bd4e49ef1846319b796e38050fd8b0b8bace" exitCode=0
Sep 18 13:45:48.592954 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:48.592463    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jpl99" event={"ID":"f8742812-3cdb-4df7-801b-443e5178cb24","Type":"ContainerDied","Data":"51fb487f1f81c665889e28978d15bd4e49ef1846319b796e38050fd8b0b8bace"}
Sep 18 13:45:48.616870 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:48.616793    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-6lm8v" podStartSLOduration=34.631359471 podStartE2EDuration="43.616771284s" totalImagesPullingTime="8.985411813s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:45:05 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:45:47.647164415 +0000 UTC m=+96.699823742" watchObservedRunningTime="2026-09-18 13:45:48.616771284 +0000 UTC m=+97.669430611"
Sep 18 13:45:48.945378 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:48.945328    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:49.029537 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:49.029495    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:49.229225 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:49.229133    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:49.450231 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:49.450197    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:49.599325 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:49.599289    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jpl99" event={"ID":"f8742812-3cdb-4df7-801b-443e5178cb24","Type":"ContainerStarted","Data":"0e8146add6404486a2180d89aa03a1224a80a650b1705d261a5abcd9e4c52b25"}
Sep 18 13:45:49.599770 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:49.599359    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jpl99" event={"ID":"f8742812-3cdb-4df7-801b-443e5178cb24","Type":"ContainerStarted","Data":"300ca703332451b0a870c2f81372acacbbb552c214c9d5091452cd6113c77fe9"}
Sep 18 13:45:49.627490 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:49.627397    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-jpl99" podStartSLOduration=2.530453176 podStartE2EDuration="3.627376649s" totalImagesPullingTime="1.013618555s" totalInitContainerRuntime="83.304918ms" isStatefulPod=true podCreationTimestamp="2026-09-18 13:45:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:45:49.626448722 +0000 UTC m=+98.679108048" watchObservedRunningTime="2026-09-18 13:45:49.627376649 +0000 UTC m=+98.680035980"
Sep 18 13:45:49.670366 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:49.670333    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:49.750959 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:49.750925    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:50.450573 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:50.450539    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-hj8lg"
Sep 18 13:45:50.747810 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:50.747716    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:50.862812 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:50.862774    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:50.978192 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:50.978163    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:51.142794 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:51.142753    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:51.248908 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:51.248877    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:51.351790 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:51.351745    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:51.483928 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:51.483782    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:51.994857 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:51.994810    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-59dccbbcb7-ql9ft"]
Sep 18 13:45:52.116177 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:52.116136    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:52.210265 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:52.210229    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:52.297183 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:52.297098    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:54.447550 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:54.447513    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:54.554636 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:54.554551    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:56.078516 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:56.078478    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-759448d5f6-phtkc"]
Sep 18 13:45:56.103137 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:56.103102    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-759448d5f6-phtkc"]
Sep 18 13:45:56.103316 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:56.103227    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-759448d5f6-phtkc"
Sep 18 13:45:56.105695 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:56.105667    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 18 13:45:56.105855 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:56.105812    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 18 13:45:56.105972 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:56.105956    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 18 13:45:56.106118 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:56.106097    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-wkjzx\""
Sep 18 13:45:56.106197 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:56.106163    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 18 13:45:56.106259 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:56.106109    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 18 13:45:56.111963 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:56.111784    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 18 13:45:56.170278 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:56.170246    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b5614670-ed87-4cb5-bbe4-8cc37c051162-console-oauth-config\") pod \"console-759448d5f6-phtkc\" (UID: \"b5614670-ed87-4cb5-bbe4-8cc37c051162\") " pod="openshift-console/console-759448d5f6-phtkc"
Sep 18 13:45:56.170278 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:56.170277    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b5614670-ed87-4cb5-bbe4-8cc37c051162-service-ca\") pod \"console-759448d5f6-phtkc\" (UID: \"b5614670-ed87-4cb5-bbe4-8cc37c051162\") " pod="openshift-console/console-759448d5f6-phtkc"
Sep 18 13:45:56.170499 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:56.170308    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b5614670-ed87-4cb5-bbe4-8cc37c051162-oauth-serving-cert\") pod \"console-759448d5f6-phtkc\" (UID: \"b5614670-ed87-4cb5-bbe4-8cc37c051162\") " pod="openshift-console/console-759448d5f6-phtkc"
Sep 18 13:45:56.170499 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:56.170363    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56rf6\" (UniqueName: \"kubernetes.io/projected/b5614670-ed87-4cb5-bbe4-8cc37c051162-kube-api-access-56rf6\") pod \"console-759448d5f6-phtkc\" (UID: \"b5614670-ed87-4cb5-bbe4-8cc37c051162\") " pod="openshift-console/console-759448d5f6-phtkc"
Sep 18 13:45:56.170499 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:56.170408    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5614670-ed87-4cb5-bbe4-8cc37c051162-console-serving-cert\") pod \"console-759448d5f6-phtkc\" (UID: \"b5614670-ed87-4cb5-bbe4-8cc37c051162\") " pod="openshift-console/console-759448d5f6-phtkc"
Sep 18 13:45:56.170499 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:56.170426    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5614670-ed87-4cb5-bbe4-8cc37c051162-trusted-ca-bundle\") pod \"console-759448d5f6-phtkc\" (UID: \"b5614670-ed87-4cb5-bbe4-8cc37c051162\") " pod="openshift-console/console-759448d5f6-phtkc"
Sep 18 13:45:56.170499 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:56.170458    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b5614670-ed87-4cb5-bbe4-8cc37c051162-console-config\") pod \"console-759448d5f6-phtkc\" (UID: \"b5614670-ed87-4cb5-bbe4-8cc37c051162\") " pod="openshift-console/console-759448d5f6-phtkc"
Sep 18 13:45:56.271005 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:56.270967    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56rf6\" (UniqueName: \"kubernetes.io/projected/b5614670-ed87-4cb5-bbe4-8cc37c051162-kube-api-access-56rf6\") pod \"console-759448d5f6-phtkc\" (UID: \"b5614670-ed87-4cb5-bbe4-8cc37c051162\") " pod="openshift-console/console-759448d5f6-phtkc"
Sep 18 13:45:56.271005 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:56.271006    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5614670-ed87-4cb5-bbe4-8cc37c051162-console-serving-cert\") pod \"console-759448d5f6-phtkc\" (UID: \"b5614670-ed87-4cb5-bbe4-8cc37c051162\") " pod="openshift-console/console-759448d5f6-phtkc"
Sep 18 13:45:56.271211 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:56.271030    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5614670-ed87-4cb5-bbe4-8cc37c051162-trusted-ca-bundle\") pod \"console-759448d5f6-phtkc\" (UID: \"b5614670-ed87-4cb5-bbe4-8cc37c051162\") " pod="openshift-console/console-759448d5f6-phtkc"
Sep 18 13:45:56.271267 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:56.271218    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b5614670-ed87-4cb5-bbe4-8cc37c051162-console-config\") pod \"console-759448d5f6-phtkc\" (UID: \"b5614670-ed87-4cb5-bbe4-8cc37c051162\") " pod="openshift-console/console-759448d5f6-phtkc"
Sep 18 13:45:56.271322 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:56.271299    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b5614670-ed87-4cb5-bbe4-8cc37c051162-console-oauth-config\") pod \"console-759448d5f6-phtkc\" (UID: \"b5614670-ed87-4cb5-bbe4-8cc37c051162\") " pod="openshift-console/console-759448d5f6-phtkc"
Sep 18 13:45:56.271377 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:56.271327    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b5614670-ed87-4cb5-bbe4-8cc37c051162-service-ca\") pod \"console-759448d5f6-phtkc\" (UID: \"b5614670-ed87-4cb5-bbe4-8cc37c051162\") " pod="openshift-console/console-759448d5f6-phtkc"
Sep 18 13:45:56.271441 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:56.271400    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b5614670-ed87-4cb5-bbe4-8cc37c051162-oauth-serving-cert\") pod \"console-759448d5f6-phtkc\" (UID: \"b5614670-ed87-4cb5-bbe4-8cc37c051162\") " pod="openshift-console/console-759448d5f6-phtkc"
Sep 18 13:45:56.271874 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:56.271848    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b5614670-ed87-4cb5-bbe4-8cc37c051162-console-config\") pod \"console-759448d5f6-phtkc\" (UID: \"b5614670-ed87-4cb5-bbe4-8cc37c051162\") " pod="openshift-console/console-759448d5f6-phtkc"
Sep 18 13:45:56.272008 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:56.271949    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5614670-ed87-4cb5-bbe4-8cc37c051162-trusted-ca-bundle\") pod \"console-759448d5f6-phtkc\" (UID: \"b5614670-ed87-4cb5-bbe4-8cc37c051162\") " pod="openshift-console/console-759448d5f6-phtkc"
Sep 18 13:45:56.272178 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:56.272152    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b5614670-ed87-4cb5-bbe4-8cc37c051162-oauth-serving-cert\") pod \"console-759448d5f6-phtkc\" (UID: \"b5614670-ed87-4cb5-bbe4-8cc37c051162\") " pod="openshift-console/console-759448d5f6-phtkc"
Sep 18 13:45:56.272282 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:56.272187    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b5614670-ed87-4cb5-bbe4-8cc37c051162-service-ca\") pod \"console-759448d5f6-phtkc\" (UID: \"b5614670-ed87-4cb5-bbe4-8cc37c051162\") " pod="openshift-console/console-759448d5f6-phtkc"
Sep 18 13:45:56.273771 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:56.273744    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b5614670-ed87-4cb5-bbe4-8cc37c051162-console-oauth-config\") pod \"console-759448d5f6-phtkc\" (UID: \"b5614670-ed87-4cb5-bbe4-8cc37c051162\") " pod="openshift-console/console-759448d5f6-phtkc"
Sep 18 13:45:56.274070 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:56.274054    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5614670-ed87-4cb5-bbe4-8cc37c051162-console-serving-cert\") pod \"console-759448d5f6-phtkc\" (UID: \"b5614670-ed87-4cb5-bbe4-8cc37c051162\") " pod="openshift-console/console-759448d5f6-phtkc"
Sep 18 13:45:56.279048 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:56.279013    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56rf6\" (UniqueName: \"kubernetes.io/projected/b5614670-ed87-4cb5-bbe4-8cc37c051162-kube-api-access-56rf6\") pod \"console-759448d5f6-phtkc\" (UID: \"b5614670-ed87-4cb5-bbe4-8cc37c051162\") " pod="openshift-console/console-759448d5f6-phtkc"
Sep 18 13:45:56.415036 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:56.414997    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-759448d5f6-phtkc"
Sep 18 13:45:56.708528 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:56.708448    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-759448d5f6-phtkc"]
Sep 18 13:45:56.711276 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:45:56.711236    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5614670_ed87_4cb5_bbe4_8cc37c051162.slice/crio-e214579bb4bb423e850644ee814b40f1ee05155af414ee2dbc7485c5de9f757d WatchSource:0}: Error finding container e214579bb4bb423e850644ee814b40f1ee05155af414ee2dbc7485c5de9f757d: Status 404 returned error can't find the container with id e214579bb4bb423e850644ee814b40f1ee05155af414ee2dbc7485c5de9f757d
Sep 18 13:45:57.634436 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:57.634387    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-759448d5f6-phtkc" event={"ID":"b5614670-ed87-4cb5-bbe4-8cc37c051162","Type":"ContainerStarted","Data":"e214579bb4bb423e850644ee814b40f1ee05155af414ee2dbc7485c5de9f757d"}
Sep 18 13:45:58.631570 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:58.631531    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:45:58.703337 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:45:58.703286    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:46:02.655050 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:46:02.655012    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-759448d5f6-phtkc" event={"ID":"b5614670-ed87-4cb5-bbe4-8cc37c051162","Type":"ContainerStarted","Data":"136d128f4cf2ccc034b5f7a15d57b5f3a4e2356e0fc88f74ba3beb1abbaffea0"}
Sep 18 13:46:02.672113 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:46:02.672056    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-759448d5f6-phtkc" podStartSLOduration=1.7979870199999999 podStartE2EDuration="6.672042543s" totalImagesPullingTime="4.874055523s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:45:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:46:02.671691482 +0000 UTC m=+111.724350810" watchObservedRunningTime="2026-09-18 13:46:02.672042543 +0000 UTC m=+111.724701869"
Sep 18 13:46:06.415887 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:46:06.415855    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-759448d5f6-phtkc"
Sep 18 13:46:06.416274 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:46:06.415908    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-759448d5f6-phtkc"
Sep 18 13:46:06.420628 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:46:06.420604    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-759448d5f6-phtkc"
Sep 18 13:46:06.672766 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:46:06.672692    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-759448d5f6-phtkc"
Sep 18 13:46:06.844758 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:46:06.844716    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:46:06.945389 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:46:06.945297    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:46:07.046084 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:46:07.046050    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:46:07.197551 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:46:07.197464    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:46:07.271061 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:46:07.271028    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:46:09.454030 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:46:09.453994    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:46:09.554118 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:46:09.554089    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:46:17.020965 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:46:17.020920    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-59dccbbcb7-ql9ft" podUID="65250207-5996-4ed6-b0f3-555bf3213f85" containerName="registry" containerID="cri-o://7fabaa707dd322219b3f9daca278a5ab4f1b4adc4a5b4cb6245bf9354f5451a4" gracePeriod=30
Sep 18 13:46:17.257250 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:46:17.257227    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59dccbbcb7-ql9ft"
Sep 18 13:46:17.321318 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:46:17.321229    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/65250207-5996-4ed6-b0f3-555bf3213f85-ca-trust-extracted\") pod \"65250207-5996-4ed6-b0f3-555bf3213f85\" (UID: \"65250207-5996-4ed6-b0f3-555bf3213f85\") "
Sep 18 13:46:17.321318 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:46:17.321276    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/65250207-5996-4ed6-b0f3-555bf3213f85-registry-certificates\") pod \"65250207-5996-4ed6-b0f3-555bf3213f85\" (UID: \"65250207-5996-4ed6-b0f3-555bf3213f85\") "
Sep 18 13:46:17.321318 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:46:17.321306    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/65250207-5996-4ed6-b0f3-555bf3213f85-bound-sa-token\") pod \"65250207-5996-4ed6-b0f3-555bf3213f85\" (UID: \"65250207-5996-4ed6-b0f3-555bf3213f85\") "
Sep 18 13:46:17.321318 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:46:17.321326    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8r8cf\" (UniqueName: \"kubernetes.io/projected/65250207-5996-4ed6-b0f3-555bf3213f85-kube-api-access-8r8cf\") pod \"65250207-5996-4ed6-b0f3-555bf3213f85\" (UID: \"65250207-5996-4ed6-b0f3-555bf3213f85\") "
Sep 18 13:46:17.321641 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:46:17.321346    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/65250207-5996-4ed6-b0f3-555bf3213f85-trusted-ca\") pod \"65250207-5996-4ed6-b0f3-555bf3213f85\" (UID: \"65250207-5996-4ed6-b0f3-555bf3213f85\") "
Sep 18 13:46:17.321641 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:46:17.321364    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/65250207-5996-4ed6-b0f3-555bf3213f85-image-registry-private-configuration\") pod \"65250207-5996-4ed6-b0f3-555bf3213f85\" (UID: \"65250207-5996-4ed6-b0f3-555bf3213f85\") "
Sep 18 13:46:17.321641 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:46:17.321395    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/65250207-5996-4ed6-b0f3-555bf3213f85-installation-pull-secrets\") pod \"65250207-5996-4ed6-b0f3-555bf3213f85\" (UID: \"65250207-5996-4ed6-b0f3-555bf3213f85\") "
Sep 18 13:46:17.321641 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:46:17.321410    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/65250207-5996-4ed6-b0f3-555bf3213f85-registry-tls\") pod \"65250207-5996-4ed6-b0f3-555bf3213f85\" (UID: \"65250207-5996-4ed6-b0f3-555bf3213f85\") "
Sep 18 13:46:17.321870 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:46:17.321733    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65250207-5996-4ed6-b0f3-555bf3213f85-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "65250207-5996-4ed6-b0f3-555bf3213f85" (UID: "65250207-5996-4ed6-b0f3-555bf3213f85"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:46:17.321870 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:46:17.321790    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65250207-5996-4ed6-b0f3-555bf3213f85-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "65250207-5996-4ed6-b0f3-555bf3213f85" (UID: "65250207-5996-4ed6-b0f3-555bf3213f85"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:46:17.324094 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:46:17.324060    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65250207-5996-4ed6-b0f3-555bf3213f85-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "65250207-5996-4ed6-b0f3-555bf3213f85" (UID: "65250207-5996-4ed6-b0f3-555bf3213f85"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:46:17.324263 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:46:17.324207    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65250207-5996-4ed6-b0f3-555bf3213f85-kube-api-access-8r8cf" (OuterVolumeSpecName: "kube-api-access-8r8cf") pod "65250207-5996-4ed6-b0f3-555bf3213f85" (UID: "65250207-5996-4ed6-b0f3-555bf3213f85"). InnerVolumeSpecName "kube-api-access-8r8cf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:46:17.324376 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:46:17.324290    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65250207-5996-4ed6-b0f3-555bf3213f85-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "65250207-5996-4ed6-b0f3-555bf3213f85" (UID: "65250207-5996-4ed6-b0f3-555bf3213f85"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:46:17.324376 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:46:17.324324    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65250207-5996-4ed6-b0f3-555bf3213f85-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "65250207-5996-4ed6-b0f3-555bf3213f85" (UID: "65250207-5996-4ed6-b0f3-555bf3213f85"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:46:17.324376 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:46:17.324354    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65250207-5996-4ed6-b0f3-555bf3213f85-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "65250207-5996-4ed6-b0f3-555bf3213f85" (UID: "65250207-5996-4ed6-b0f3-555bf3213f85"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:46:17.331873 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:46:17.331789    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65250207-5996-4ed6-b0f3-555bf3213f85-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "65250207-5996-4ed6-b0f3-555bf3213f85" (UID: "65250207-5996-4ed6-b0f3-555bf3213f85"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:46:17.386661 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:46:17.386630    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:46:17.422949 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:46:17.422914    2577 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/65250207-5996-4ed6-b0f3-555bf3213f85-ca-trust-extracted\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:46:17.422949 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:46:17.422942    2577 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/65250207-5996-4ed6-b0f3-555bf3213f85-registry-certificates\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:46:17.423118 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:46:17.422959    2577 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/65250207-5996-4ed6-b0f3-555bf3213f85-bound-sa-token\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:46:17.423118 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:46:17.422971    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8r8cf\" (UniqueName: \"kubernetes.io/projected/65250207-5996-4ed6-b0f3-555bf3213f85-kube-api-access-8r8cf\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:46:17.423118 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:46:17.422984    2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/65250207-5996-4ed6-b0f3-555bf3213f85-trusted-ca\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:46:17.423118 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:46:17.422997    2577 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/65250207-5996-4ed6-b0f3-555bf3213f85-image-registry-private-configuration\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:46:17.423118 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:46:17.423010    2577 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/65250207-5996-4ed6-b0f3-555bf3213f85-installation-pull-secrets\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:46:17.423118 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:46:17.423022    2577 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/65250207-5996-4ed6-b0f3-555bf3213f85-registry-tls\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:46:17.476525 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:46:17.476487    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:46:17.709108 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:46:17.709077    2577 generic.go:356] "Generic (PLEG): container finished" podID="65250207-5996-4ed6-b0f3-555bf3213f85" containerID="7fabaa707dd322219b3f9daca278a5ab4f1b4adc4a5b4cb6245bf9354f5451a4" exitCode=0
Sep 18 13:46:17.709285 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:46:17.709135    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59dccbbcb7-ql9ft"
Sep 18 13:46:17.709285 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:46:17.709161    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59dccbbcb7-ql9ft" event={"ID":"65250207-5996-4ed6-b0f3-555bf3213f85","Type":"ContainerDied","Data":"7fabaa707dd322219b3f9daca278a5ab4f1b4adc4a5b4cb6245bf9354f5451a4"}
Sep 18 13:46:17.709285 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:46:17.709233    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59dccbbcb7-ql9ft" event={"ID":"65250207-5996-4ed6-b0f3-555bf3213f85","Type":"ContainerDied","Data":"9dcf87b4b7face6b73b84840a3d7771ec8da39f56243ae9f3182737dd8aaa754"}
Sep 18 13:46:17.709285 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:46:17.709253    2577 scope.go:117] "RemoveContainer" containerID="7fabaa707dd322219b3f9daca278a5ab4f1b4adc4a5b4cb6245bf9354f5451a4"
Sep 18 13:46:17.717757 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:46:17.717742    2577 scope.go:117] "RemoveContainer" containerID="7fabaa707dd322219b3f9daca278a5ab4f1b4adc4a5b4cb6245bf9354f5451a4"
Sep 18 13:46:17.718038 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:46:17.718017    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fabaa707dd322219b3f9daca278a5ab4f1b4adc4a5b4cb6245bf9354f5451a4\": container with ID starting with 7fabaa707dd322219b3f9daca278a5ab4f1b4adc4a5b4cb6245bf9354f5451a4 not found: ID does not exist" containerID="7fabaa707dd322219b3f9daca278a5ab4f1b4adc4a5b4cb6245bf9354f5451a4"
Sep 18 13:46:17.718106 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:46:17.718046    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fabaa707dd322219b3f9daca278a5ab4f1b4adc4a5b4cb6245bf9354f5451a4"} err="failed to get container status \"7fabaa707dd322219b3f9daca278a5ab4f1b4adc4a5b4cb6245bf9354f5451a4\": rpc error: code = NotFound desc = could not find container \"7fabaa707dd322219b3f9daca278a5ab4f1b4adc4a5b4cb6245bf9354f5451a4\": container with ID starting with 7fabaa707dd322219b3f9daca278a5ab4f1b4adc4a5b4cb6245bf9354f5451a4 not found: ID does not exist"
Sep 18 13:46:17.731443 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:46:17.731417    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-59dccbbcb7-ql9ft"]
Sep 18 13:46:17.736351 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:46:17.736321    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-59dccbbcb7-ql9ft"]
Sep 18 13:46:19.105695 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:46:19.105661    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65250207-5996-4ed6-b0f3-555bf3213f85" path="/var/lib/kubelet/pods/65250207-5996-4ed6-b0f3-555bf3213f85/volumes"
Sep 18 13:46:21.857072 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:46:21.857034    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87eba103-cfec-4000-a10c-fd4fd512ac90-metrics-certs\") pod \"network-metrics-daemon-nb9bg\" (UID: \"87eba103-cfec-4000-a10c-fd4fd512ac90\") " pod="openshift-multus/network-metrics-daemon-nb9bg"
Sep 18 13:46:21.859296 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:46:21.859276    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87eba103-cfec-4000-a10c-fd4fd512ac90-metrics-certs\") pod \"network-metrics-daemon-nb9bg\" (UID: \"87eba103-cfec-4000-a10c-fd4fd512ac90\") " pod="openshift-multus/network-metrics-daemon-nb9bg"
Sep 18 13:46:22.028211 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:46:22.028181    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tqxkk\""
Sep 18 13:46:22.036914 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:46:22.036888    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nb9bg"
Sep 18 13:46:22.396320 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:46:22.396249    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nb9bg"]
Sep 18 13:46:22.399270 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:46:22.399239    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87eba103_cfec_4000_a10c_fd4fd512ac90.slice/crio-3ae1dd474026d6a5b2eb4497aa9e2f961b3b31cc0329415124b136ee32c538bd WatchSource:0}: Error finding container 3ae1dd474026d6a5b2eb4497aa9e2f961b3b31cc0329415124b136ee32c538bd: Status 404 returned error can't find the container with id 3ae1dd474026d6a5b2eb4497aa9e2f961b3b31cc0329415124b136ee32c538bd
Sep 18 13:46:22.728592 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:46:22.728516    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nb9bg" event={"ID":"87eba103-cfec-4000-a10c-fd4fd512ac90","Type":"ContainerStarted","Data":"3ae1dd474026d6a5b2eb4497aa9e2f961b3b31cc0329415124b136ee32c538bd"}
Sep 18 13:46:24.447049 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:46:24.447007    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:46:24.509963 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:46:24.509926    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:46:24.737344 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:46:24.737260    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nb9bg" event={"ID":"87eba103-cfec-4000-a10c-fd4fd512ac90","Type":"ContainerStarted","Data":"16181682b6805d575be59fd528798be4e0e29a6b7da491460cd4a2c05a0ddf5b"}
Sep 18 13:46:24.737488 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:46:24.737364    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nb9bg" event={"ID":"87eba103-cfec-4000-a10c-fd4fd512ac90","Type":"ContainerStarted","Data":"56966ee6adf407dc17b897458f2a8e6e98b70b7a2e74b53cecbfd392ddf4ae63"}
Sep 18 13:46:24.755697 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:46:24.755638    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-nb9bg" podStartSLOduration=130.37747872 podStartE2EDuration="2m11.755623833s" totalImagesPullingTime="1.378145113s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:44:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:46:24.755132409 +0000 UTC m=+133.807791735" watchObservedRunningTime="2026-09-18 13:46:24.755623833 +0000 UTC m=+133.808283160"
Sep 18 13:46:24.878300 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:46:24.878268    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:46:24.944447 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:46:24.944411    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:46:25.743568 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:46:25.743537    2577 generic.go:356] "Generic (PLEG): container finished" podID="a20817cc-9558-434d-9061-86be074f9ad2" containerID="46cc6d4d3c4d13fba11a8a4ed55daf5ab54d53e5e6f49b28594947015610a579" exitCode=0
Sep 18 13:46:25.744079 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:46:25.743609    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-v99sv" event={"ID":"a20817cc-9558-434d-9061-86be074f9ad2","Type":"ContainerDied","Data":"46cc6d4d3c4d13fba11a8a4ed55daf5ab54d53e5e6f49b28594947015610a579"}
Sep 18 13:46:25.744151 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:46:25.744078    2577 scope.go:117] "RemoveContainer" containerID="46cc6d4d3c4d13fba11a8a4ed55daf5ab54d53e5e6f49b28594947015610a579"
Sep 18 13:46:26.749815 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:46:26.749781    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-v99sv" event={"ID":"a20817cc-9558-434d-9061-86be074f9ad2","Type":"ContainerStarted","Data":"e4182f4bc0ebe2cb871f62e06a60fb1c207846c3fdf8af88a956f2e716614a48"}
Sep 18 13:46:34.049473 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:46:34.049434    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:46:34.160955 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:46:34.160906    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:46:34.264531 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:46:34.264501    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:46:35.415924 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:46:35.415888    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:46:35.517008 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:46:35.516976    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:46:40.669147 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:46:40.669088    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:46:40.729075 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:46:40.729039    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:47:02.133014 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:02.132969    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:47:02.188804 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:02.188756    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:47:05.723710 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:05.723677    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:47:14.026246 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:14.026208    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:47:14.083874 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:14.083842    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:47:17.023353 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:17.023319    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:47:17.137383 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:17.137346    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:47:18.999725 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:18.999686    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-77b97c7f9b-q6nzl"]
Sep 18 13:47:19.000200 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:18.999999    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65250207-5996-4ed6-b0f3-555bf3213f85" containerName="registry"
Sep 18 13:47:19.000200 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:19.000013    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="65250207-5996-4ed6-b0f3-555bf3213f85" containerName="registry"
Sep 18 13:47:19.000200 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:19.000079    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="65250207-5996-4ed6-b0f3-555bf3213f85" containerName="registry"
Sep 18 13:47:19.002931 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:19.002903    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77b97c7f9b-q6nzl"
Sep 18 13:47:19.018881 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:19.018853    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77b97c7f9b-q6nzl"]
Sep 18 13:47:19.057996 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:19.057958    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:47:19.083097 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:19.083071    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8a8f3c05-66db-42a8-85e7-aaf007bb50fe-oauth-serving-cert\") pod \"console-77b97c7f9b-q6nzl\" (UID: \"8a8f3c05-66db-42a8-85e7-aaf007bb50fe\") " pod="openshift-console/console-77b97c7f9b-q6nzl"
Sep 18 13:47:19.083232 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:19.083119    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8a8f3c05-66db-42a8-85e7-aaf007bb50fe-service-ca\") pod \"console-77b97c7f9b-q6nzl\" (UID: \"8a8f3c05-66db-42a8-85e7-aaf007bb50fe\") " pod="openshift-console/console-77b97c7f9b-q6nzl"
Sep 18 13:47:19.083232 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:19.083152    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8a8f3c05-66db-42a8-85e7-aaf007bb50fe-console-config\") pod \"console-77b97c7f9b-q6nzl\" (UID: \"8a8f3c05-66db-42a8-85e7-aaf007bb50fe\") " pod="openshift-console/console-77b97c7f9b-q6nzl"
Sep 18 13:47:19.083232 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:19.083180    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8a8f3c05-66db-42a8-85e7-aaf007bb50fe-console-serving-cert\") pod \"console-77b97c7f9b-q6nzl\" (UID: \"8a8f3c05-66db-42a8-85e7-aaf007bb50fe\") " pod="openshift-console/console-77b97c7f9b-q6nzl"
Sep 18 13:47:19.083232 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:19.083203    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v299j\" (UniqueName: \"kubernetes.io/projected/8a8f3c05-66db-42a8-85e7-aaf007bb50fe-kube-api-access-v299j\") pod \"console-77b97c7f9b-q6nzl\" (UID: \"8a8f3c05-66db-42a8-85e7-aaf007bb50fe\") " pod="openshift-console/console-77b97c7f9b-q6nzl"
Sep 18 13:47:19.083424 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:19.083258    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8a8f3c05-66db-42a8-85e7-aaf007bb50fe-console-oauth-config\") pod \"console-77b97c7f9b-q6nzl\" (UID: \"8a8f3c05-66db-42a8-85e7-aaf007bb50fe\") " pod="openshift-console/console-77b97c7f9b-q6nzl"
Sep 18 13:47:19.083424 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:19.083297    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a8f3c05-66db-42a8-85e7-aaf007bb50fe-trusted-ca-bundle\") pod \"console-77b97c7f9b-q6nzl\" (UID: \"8a8f3c05-66db-42a8-85e7-aaf007bb50fe\") " pod="openshift-console/console-77b97c7f9b-q6nzl"
Sep 18 13:47:19.127135 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:19.127103    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:47:19.184357 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:19.184323    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a8f3c05-66db-42a8-85e7-aaf007bb50fe-trusted-ca-bundle\") pod \"console-77b97c7f9b-q6nzl\" (UID: \"8a8f3c05-66db-42a8-85e7-aaf007bb50fe\") " pod="openshift-console/console-77b97c7f9b-q6nzl"
Sep 18 13:47:19.184517 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:19.184391    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8a8f3c05-66db-42a8-85e7-aaf007bb50fe-oauth-serving-cert\") pod \"console-77b97c7f9b-q6nzl\" (UID: \"8a8f3c05-66db-42a8-85e7-aaf007bb50fe\") " pod="openshift-console/console-77b97c7f9b-q6nzl"
Sep 18 13:47:19.184517 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:19.184448    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8a8f3c05-66db-42a8-85e7-aaf007bb50fe-service-ca\") pod \"console-77b97c7f9b-q6nzl\" (UID: \"8a8f3c05-66db-42a8-85e7-aaf007bb50fe\") " pod="openshift-console/console-77b97c7f9b-q6nzl"
Sep 18 13:47:19.184517 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:19.184476    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8a8f3c05-66db-42a8-85e7-aaf007bb50fe-console-config\") pod \"console-77b97c7f9b-q6nzl\" (UID: \"8a8f3c05-66db-42a8-85e7-aaf007bb50fe\") " pod="openshift-console/console-77b97c7f9b-q6nzl"
Sep 18 13:47:19.184517 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:19.184507    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8a8f3c05-66db-42a8-85e7-aaf007bb50fe-console-serving-cert\") pod \"console-77b97c7f9b-q6nzl\" (UID: \"8a8f3c05-66db-42a8-85e7-aaf007bb50fe\") " pod="openshift-console/console-77b97c7f9b-q6nzl"
Sep 18 13:47:19.184721 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:19.184535    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v299j\" (UniqueName: \"kubernetes.io/projected/8a8f3c05-66db-42a8-85e7-aaf007bb50fe-kube-api-access-v299j\") pod \"console-77b97c7f9b-q6nzl\" (UID: \"8a8f3c05-66db-42a8-85e7-aaf007bb50fe\") " pod="openshift-console/console-77b97c7f9b-q6nzl"
Sep 18 13:47:19.184721 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:19.184606    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8a8f3c05-66db-42a8-85e7-aaf007bb50fe-console-oauth-config\") pod \"console-77b97c7f9b-q6nzl\" (UID: \"8a8f3c05-66db-42a8-85e7-aaf007bb50fe\") " pod="openshift-console/console-77b97c7f9b-q6nzl"
Sep 18 13:47:19.185298 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:19.185268    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8a8f3c05-66db-42a8-85e7-aaf007bb50fe-console-config\") pod \"console-77b97c7f9b-q6nzl\" (UID: \"8a8f3c05-66db-42a8-85e7-aaf007bb50fe\") " pod="openshift-console/console-77b97c7f9b-q6nzl"
Sep 18 13:47:19.185416 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:19.185344    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8a8f3c05-66db-42a8-85e7-aaf007bb50fe-service-ca\") pod \"console-77b97c7f9b-q6nzl\" (UID: \"8a8f3c05-66db-42a8-85e7-aaf007bb50fe\") " pod="openshift-console/console-77b97c7f9b-q6nzl"
Sep 18 13:47:19.185580 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:19.185558    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8a8f3c05-66db-42a8-85e7-aaf007bb50fe-oauth-serving-cert\") pod \"console-77b97c7f9b-q6nzl\" (UID: \"8a8f3c05-66db-42a8-85e7-aaf007bb50fe\") " pod="openshift-console/console-77b97c7f9b-q6nzl"
Sep 18 13:47:19.185673 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:19.185595    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a8f3c05-66db-42a8-85e7-aaf007bb50fe-trusted-ca-bundle\") pod \"console-77b97c7f9b-q6nzl\" (UID: \"8a8f3c05-66db-42a8-85e7-aaf007bb50fe\") " pod="openshift-console/console-77b97c7f9b-q6nzl"
Sep 18 13:47:19.187120 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:19.187101    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8a8f3c05-66db-42a8-85e7-aaf007bb50fe-console-serving-cert\") pod \"console-77b97c7f9b-q6nzl\" (UID: \"8a8f3c05-66db-42a8-85e7-aaf007bb50fe\") " pod="openshift-console/console-77b97c7f9b-q6nzl"
Sep 18 13:47:19.187120 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:19.187118    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8a8f3c05-66db-42a8-85e7-aaf007bb50fe-console-oauth-config\") pod \"console-77b97c7f9b-q6nzl\" (UID: \"8a8f3c05-66db-42a8-85e7-aaf007bb50fe\") " pod="openshift-console/console-77b97c7f9b-q6nzl"
Sep 18 13:47:19.194543 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:19.194519    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v299j\" (UniqueName: \"kubernetes.io/projected/8a8f3c05-66db-42a8-85e7-aaf007bb50fe-kube-api-access-v299j\") pod \"console-77b97c7f9b-q6nzl\" (UID: \"8a8f3c05-66db-42a8-85e7-aaf007bb50fe\") " pod="openshift-console/console-77b97c7f9b-q6nzl"
Sep 18 13:47:19.312470 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:19.312382    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77b97c7f9b-q6nzl"
Sep 18 13:47:19.583451 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:19.583375    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77b97c7f9b-q6nzl"]
Sep 18 13:47:19.586399 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:47:19.586373    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a8f3c05_66db_42a8_85e7_aaf007bb50fe.slice/crio-9ff454ca39ad1164387e3a4e08862485ecf051fddcbb04e3705a812c09339f14 WatchSource:0}: Error finding container 9ff454ca39ad1164387e3a4e08862485ecf051fddcbb04e3705a812c09339f14: Status 404 returned error can't find the container with id 9ff454ca39ad1164387e3a4e08862485ecf051fddcbb04e3705a812c09339f14
Sep 18 13:47:19.942644 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:19.942602    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77b97c7f9b-q6nzl" event={"ID":"8a8f3c05-66db-42a8-85e7-aaf007bb50fe","Type":"ContainerStarted","Data":"86a72f358933feffd0385edfd0c5b39b47800cc53eebb1d7e3f108ea2246002c"}
Sep 18 13:47:19.942842 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:19.942665    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77b97c7f9b-q6nzl" event={"ID":"8a8f3c05-66db-42a8-85e7-aaf007bb50fe","Type":"ContainerStarted","Data":"9ff454ca39ad1164387e3a4e08862485ecf051fddcbb04e3705a812c09339f14"}
Sep 18 13:47:19.961236 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:19.961190    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-77b97c7f9b-q6nzl" podStartSLOduration=1.9611743879999999 podStartE2EDuration="1.961174388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:47:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:47:19.960168753 +0000 UTC m=+189.012828079" watchObservedRunningTime="2026-09-18 13:47:19.961174388 +0000 UTC m=+189.013833715"
Sep 18 13:47:24.460768 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:24.460731    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:47:24.568390 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:24.568353    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:47:24.669092 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:24.669054    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:47:29.313408 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:29.313378    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-77b97c7f9b-q6nzl"
Sep 18 13:47:29.313791 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:29.313435    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-77b97c7f9b-q6nzl"
Sep 18 13:47:29.318142 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:29.318122    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-77b97c7f9b-q6nzl"
Sep 18 13:47:29.981719 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:29.981687    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-77b97c7f9b-q6nzl"
Sep 18 13:47:30.028906 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:30.028869    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-759448d5f6-phtkc"]
Sep 18 13:47:30.150792 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:30.150763    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:47:30.250534 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:30.250456    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:47:30.314991 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:30.314959    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:47:41.924325 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:41.924288    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:47:55.054407 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:55.054342    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-759448d5f6-phtkc" podUID="b5614670-ed87-4cb5-bbe4-8cc37c051162" containerName="console" containerID="cri-o://136d128f4cf2ccc034b5f7a15d57b5f3a4e2356e0fc88f74ba3beb1abbaffea0" gracePeriod=15
Sep 18 13:47:55.282689 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:55.282665    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-759448d5f6-phtkc"
Sep 18 13:47:55.358503 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:55.358474    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b5614670-ed87-4cb5-bbe4-8cc37c051162-console-oauth-config\") pod \"b5614670-ed87-4cb5-bbe4-8cc37c051162\" (UID: \"b5614670-ed87-4cb5-bbe4-8cc37c051162\") "
Sep 18 13:47:55.358762 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:55.358523    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-56rf6\" (UniqueName: \"kubernetes.io/projected/b5614670-ed87-4cb5-bbe4-8cc37c051162-kube-api-access-56rf6\") pod \"b5614670-ed87-4cb5-bbe4-8cc37c051162\" (UID: \"b5614670-ed87-4cb5-bbe4-8cc37c051162\") "
Sep 18 13:47:55.358762 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:55.358540    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b5614670-ed87-4cb5-bbe4-8cc37c051162-console-config\") pod \"b5614670-ed87-4cb5-bbe4-8cc37c051162\" (UID: \"b5614670-ed87-4cb5-bbe4-8cc37c051162\") "
Sep 18 13:47:55.358762 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:55.358579    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5614670-ed87-4cb5-bbe4-8cc37c051162-console-serving-cert\") pod \"b5614670-ed87-4cb5-bbe4-8cc37c051162\" (UID: \"b5614670-ed87-4cb5-bbe4-8cc37c051162\") "
Sep 18 13:47:55.358762 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:55.358603    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5614670-ed87-4cb5-bbe4-8cc37c051162-trusted-ca-bundle\") pod \"b5614670-ed87-4cb5-bbe4-8cc37c051162\" (UID: \"b5614670-ed87-4cb5-bbe4-8cc37c051162\") "
Sep 18 13:47:55.358762 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:55.358631    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b5614670-ed87-4cb5-bbe4-8cc37c051162-oauth-serving-cert\") pod \"b5614670-ed87-4cb5-bbe4-8cc37c051162\" (UID: \"b5614670-ed87-4cb5-bbe4-8cc37c051162\") "
Sep 18 13:47:55.358762 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:55.358678    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b5614670-ed87-4cb5-bbe4-8cc37c051162-service-ca\") pod \"b5614670-ed87-4cb5-bbe4-8cc37c051162\" (UID: \"b5614670-ed87-4cb5-bbe4-8cc37c051162\") "
Sep 18 13:47:55.359103 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:55.359010    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5614670-ed87-4cb5-bbe4-8cc37c051162-console-config" (OuterVolumeSpecName: "console-config") pod "b5614670-ed87-4cb5-bbe4-8cc37c051162" (UID: "b5614670-ed87-4cb5-bbe4-8cc37c051162"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:47:55.359160 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:55.359097    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5614670-ed87-4cb5-bbe4-8cc37c051162-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b5614670-ed87-4cb5-bbe4-8cc37c051162" (UID: "b5614670-ed87-4cb5-bbe4-8cc37c051162"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:47:55.359229 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:55.359159    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5614670-ed87-4cb5-bbe4-8cc37c051162-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b5614670-ed87-4cb5-bbe4-8cc37c051162" (UID: "b5614670-ed87-4cb5-bbe4-8cc37c051162"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:47:55.359273 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:55.359230    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5614670-ed87-4cb5-bbe4-8cc37c051162-service-ca" (OuterVolumeSpecName: "service-ca") pod "b5614670-ed87-4cb5-bbe4-8cc37c051162" (UID: "b5614670-ed87-4cb5-bbe4-8cc37c051162"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:47:55.360879 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:55.360854    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5614670-ed87-4cb5-bbe4-8cc37c051162-kube-api-access-56rf6" (OuterVolumeSpecName: "kube-api-access-56rf6") pod "b5614670-ed87-4cb5-bbe4-8cc37c051162" (UID: "b5614670-ed87-4cb5-bbe4-8cc37c051162"). InnerVolumeSpecName "kube-api-access-56rf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:47:55.360993 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:55.360886    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5614670-ed87-4cb5-bbe4-8cc37c051162-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b5614670-ed87-4cb5-bbe4-8cc37c051162" (UID: "b5614670-ed87-4cb5-bbe4-8cc37c051162"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:47:55.360993 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:55.360905    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5614670-ed87-4cb5-bbe4-8cc37c051162-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b5614670-ed87-4cb5-bbe4-8cc37c051162" (UID: "b5614670-ed87-4cb5-bbe4-8cc37c051162"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:47:55.459992 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:55.459962    2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b5614670-ed87-4cb5-bbe4-8cc37c051162-service-ca\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:47:55.459992 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:55.459988    2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b5614670-ed87-4cb5-bbe4-8cc37c051162-console-oauth-config\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:47:55.459992 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:55.459999    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56rf6\" (UniqueName: \"kubernetes.io/projected/b5614670-ed87-4cb5-bbe4-8cc37c051162-kube-api-access-56rf6\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:47:55.460211 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:55.460008    2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b5614670-ed87-4cb5-bbe4-8cc37c051162-console-config\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:47:55.460211 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:55.460017    2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5614670-ed87-4cb5-bbe4-8cc37c051162-console-serving-cert\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:47:55.460211 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:55.460025    2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5614670-ed87-4cb5-bbe4-8cc37c051162-trusted-ca-bundle\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:47:55.460211 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:55.460033    2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b5614670-ed87-4cb5-bbe4-8cc37c051162-oauth-serving-cert\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:47:56.069019 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:56.068980    2577 generic.go:356] "Generic (PLEG): container finished" podID="b5614670-ed87-4cb5-bbe4-8cc37c051162" containerID="136d128f4cf2ccc034b5f7a15d57b5f3a4e2356e0fc88f74ba3beb1abbaffea0" exitCode=0
Sep 18 13:47:56.069019 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:56.069023    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-759448d5f6-phtkc"
Sep 18 13:47:56.069519 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:56.069027    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-759448d5f6-phtkc" event={"ID":"b5614670-ed87-4cb5-bbe4-8cc37c051162","Type":"ContainerDied","Data":"136d128f4cf2ccc034b5f7a15d57b5f3a4e2356e0fc88f74ba3beb1abbaffea0"}
Sep 18 13:47:56.069519 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:56.069080    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-759448d5f6-phtkc" event={"ID":"b5614670-ed87-4cb5-bbe4-8cc37c051162","Type":"ContainerDied","Data":"e214579bb4bb423e850644ee814b40f1ee05155af414ee2dbc7485c5de9f757d"}
Sep 18 13:47:56.069519 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:56.069111    2577 scope.go:117] "RemoveContainer" containerID="136d128f4cf2ccc034b5f7a15d57b5f3a4e2356e0fc88f74ba3beb1abbaffea0"
Sep 18 13:47:56.077470 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:56.077450    2577 scope.go:117] "RemoveContainer" containerID="136d128f4cf2ccc034b5f7a15d57b5f3a4e2356e0fc88f74ba3beb1abbaffea0"
Sep 18 13:47:56.077758 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:47:56.077738    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"136d128f4cf2ccc034b5f7a15d57b5f3a4e2356e0fc88f74ba3beb1abbaffea0\": container with ID starting with 136d128f4cf2ccc034b5f7a15d57b5f3a4e2356e0fc88f74ba3beb1abbaffea0 not found: ID does not exist" containerID="136d128f4cf2ccc034b5f7a15d57b5f3a4e2356e0fc88f74ba3beb1abbaffea0"
Sep 18 13:47:56.077802 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:56.077768    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"136d128f4cf2ccc034b5f7a15d57b5f3a4e2356e0fc88f74ba3beb1abbaffea0"} err="failed to get container status \"136d128f4cf2ccc034b5f7a15d57b5f3a4e2356e0fc88f74ba3beb1abbaffea0\": rpc error: code = NotFound desc = could not find container \"136d128f4cf2ccc034b5f7a15d57b5f3a4e2356e0fc88f74ba3beb1abbaffea0\": container with ID starting with 136d128f4cf2ccc034b5f7a15d57b5f3a4e2356e0fc88f74ba3beb1abbaffea0 not found: ID does not exist"
Sep 18 13:47:56.089087 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:56.089061    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-759448d5f6-phtkc"]
Sep 18 13:47:56.090912 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:56.090892    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-759448d5f6-phtkc"]
Sep 18 13:47:57.105267 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:47:57.105231    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5614670-ed87-4cb5-bbe4-8cc37c051162" path="/var/lib/kubelet/pods/b5614670-ed87-4cb5-bbe4-8cc37c051162/volumes"
Sep 18 13:48:10.948740 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:10.948702    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:48:11.050766 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:11.050716    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:48:17.741167 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:17.741126    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:48:17.843257 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:17.843225    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:48:18.554059 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:18.554024    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754079ksx"]
Sep 18 13:48:18.554304 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:18.554282    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5614670-ed87-4cb5-bbe4-8cc37c051162" containerName="console"
Sep 18 13:48:18.554348 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:18.554306    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5614670-ed87-4cb5-bbe4-8cc37c051162" containerName="console"
Sep 18 13:48:18.554382 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:18.554353    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5614670-ed87-4cb5-bbe4-8cc37c051162" containerName="console"
Sep 18 13:48:18.557035 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:18.557018    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754079ksx"
Sep 18 13:48:18.559302 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:18.559279    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 18 13:48:18.559809 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:18.559795    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 18 13:48:18.559900 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:18.559800    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-l2rh5\""
Sep 18 13:48:18.569339 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:18.569318    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754079ksx"]
Sep 18 13:48:18.634938 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:18.634901    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpmzh\" (UniqueName: \"kubernetes.io/projected/2078d813-8506-4581-af08-29e90b6cbd8f-kube-api-access-lpmzh\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754079ksx\" (UID: \"2078d813-8506-4581-af08-29e90b6cbd8f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754079ksx"
Sep 18 13:48:18.635103 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:18.634942    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2078d813-8506-4581-af08-29e90b6cbd8f-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754079ksx\" (UID: \"2078d813-8506-4581-af08-29e90b6cbd8f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754079ksx"
Sep 18 13:48:18.635103 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:18.634963    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2078d813-8506-4581-af08-29e90b6cbd8f-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754079ksx\" (UID: \"2078d813-8506-4581-af08-29e90b6cbd8f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754079ksx"
Sep 18 13:48:18.735838 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:18.735795    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpmzh\" (UniqueName: \"kubernetes.io/projected/2078d813-8506-4581-af08-29e90b6cbd8f-kube-api-access-lpmzh\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754079ksx\" (UID: \"2078d813-8506-4581-af08-29e90b6cbd8f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754079ksx"
Sep 18 13:48:18.736008 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:18.735852    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2078d813-8506-4581-af08-29e90b6cbd8f-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754079ksx\" (UID: \"2078d813-8506-4581-af08-29e90b6cbd8f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754079ksx"
Sep 18 13:48:18.736008 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:18.735880    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2078d813-8506-4581-af08-29e90b6cbd8f-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754079ksx\" (UID: \"2078d813-8506-4581-af08-29e90b6cbd8f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754079ksx"
Sep 18 13:48:18.736292 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:18.736272    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2078d813-8506-4581-af08-29e90b6cbd8f-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754079ksx\" (UID: \"2078d813-8506-4581-af08-29e90b6cbd8f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754079ksx"
Sep 18 13:48:18.736292 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:18.736285    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2078d813-8506-4581-af08-29e90b6cbd8f-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754079ksx\" (UID: \"2078d813-8506-4581-af08-29e90b6cbd8f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754079ksx"
Sep 18 13:48:18.745501 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:18.745478    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpmzh\" (UniqueName: \"kubernetes.io/projected/2078d813-8506-4581-af08-29e90b6cbd8f-kube-api-access-lpmzh\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754079ksx\" (UID: \"2078d813-8506-4581-af08-29e90b6cbd8f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754079ksx"
Sep 18 13:48:18.866688 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:18.866656    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754079ksx"
Sep 18 13:48:19.143775 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:19.143746    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754079ksx"]
Sep 18 13:48:19.146051 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:48:19.146025    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2078d813_8506_4581_af08_29e90b6cbd8f.slice/crio-45d6357f631cc6f1048a70b115b5dfc02bcf8f357754cebc6631432be7fc52f8 WatchSource:0}: Error finding container 45d6357f631cc6f1048a70b115b5dfc02bcf8f357754cebc6631432be7fc52f8: Status 404 returned error can't find the container with id 45d6357f631cc6f1048a70b115b5dfc02bcf8f357754cebc6631432be7fc52f8
Sep 18 13:48:20.153292 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:20.153253    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754079ksx" event={"ID":"2078d813-8506-4581-af08-29e90b6cbd8f","Type":"ContainerStarted","Data":"45d6357f631cc6f1048a70b115b5dfc02bcf8f357754cebc6631432be7fc52f8"}
Sep 18 13:48:25.173611 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:25.173501    2577 generic.go:356] "Generic (PLEG): container finished" podID="2078d813-8506-4581-af08-29e90b6cbd8f" containerID="0a376a18636adbd58befee4f78458b11dda03b6ad39a6f21172a0b9a59b5cf36" exitCode=0
Sep 18 13:48:25.173611 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:25.173552    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754079ksx" event={"ID":"2078d813-8506-4581-af08-29e90b6cbd8f","Type":"ContainerDied","Data":"0a376a18636adbd58befee4f78458b11dda03b6ad39a6f21172a0b9a59b5cf36"}
Sep 18 13:48:28.186981 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:28.186948    2577 generic.go:356] "Generic (PLEG): container finished" podID="2078d813-8506-4581-af08-29e90b6cbd8f" containerID="28dfb313fa6dcd29028c9ca84104d22d2fb5ba0ac753acfbc1f3ae7afbddb60a" exitCode=0
Sep 18 13:48:28.187486 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:28.187007    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754079ksx" event={"ID":"2078d813-8506-4581-af08-29e90b6cbd8f","Type":"ContainerDied","Data":"28dfb313fa6dcd29028c9ca84104d22d2fb5ba0ac753acfbc1f3ae7afbddb60a"}
Sep 18 13:48:36.220195 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:36.220159    2577 generic.go:356] "Generic (PLEG): container finished" podID="2078d813-8506-4581-af08-29e90b6cbd8f" containerID="412c38c23aabf9cb25d22bc226ec3e9f2d2fd4a25f44942824729ab539f4a3b3" exitCode=0
Sep 18 13:48:36.220575 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:36.220214    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754079ksx" event={"ID":"2078d813-8506-4581-af08-29e90b6cbd8f","Type":"ContainerDied","Data":"412c38c23aabf9cb25d22bc226ec3e9f2d2fd4a25f44942824729ab539f4a3b3"}
Sep 18 13:48:37.338343 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:37.338319    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754079ksx"
Sep 18 13:48:37.388868 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:37.388837    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lpmzh\" (UniqueName: \"kubernetes.io/projected/2078d813-8506-4581-af08-29e90b6cbd8f-kube-api-access-lpmzh\") pod \"2078d813-8506-4581-af08-29e90b6cbd8f\" (UID: \"2078d813-8506-4581-af08-29e90b6cbd8f\") "
Sep 18 13:48:37.389035 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:37.388902    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2078d813-8506-4581-af08-29e90b6cbd8f-util\") pod \"2078d813-8506-4581-af08-29e90b6cbd8f\" (UID: \"2078d813-8506-4581-af08-29e90b6cbd8f\") "
Sep 18 13:48:37.389035 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:37.388953    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2078d813-8506-4581-af08-29e90b6cbd8f-bundle\") pod \"2078d813-8506-4581-af08-29e90b6cbd8f\" (UID: \"2078d813-8506-4581-af08-29e90b6cbd8f\") "
Sep 18 13:48:37.389608 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:37.389586    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2078d813-8506-4581-af08-29e90b6cbd8f-bundle" (OuterVolumeSpecName: "bundle") pod "2078d813-8506-4581-af08-29e90b6cbd8f" (UID: "2078d813-8506-4581-af08-29e90b6cbd8f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:48:37.390967 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:37.390944    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2078d813-8506-4581-af08-29e90b6cbd8f-kube-api-access-lpmzh" (OuterVolumeSpecName: "kube-api-access-lpmzh") pod "2078d813-8506-4581-af08-29e90b6cbd8f" (UID: "2078d813-8506-4581-af08-29e90b6cbd8f"). InnerVolumeSpecName "kube-api-access-lpmzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:48:37.393086 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:37.393056    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2078d813-8506-4581-af08-29e90b6cbd8f-util" (OuterVolumeSpecName: "util") pod "2078d813-8506-4581-af08-29e90b6cbd8f" (UID: "2078d813-8506-4581-af08-29e90b6cbd8f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:48:37.489468 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:37.489393    2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2078d813-8506-4581-af08-29e90b6cbd8f-util\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:48:37.489468 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:37.489419    2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2078d813-8506-4581-af08-29e90b6cbd8f-bundle\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:48:37.489468 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:37.489429    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpmzh\" (UniqueName: \"kubernetes.io/projected/2078d813-8506-4581-af08-29e90b6cbd8f-kube-api-access-lpmzh\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:48:38.229069 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:38.229026    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754079ksx" event={"ID":"2078d813-8506-4581-af08-29e90b6cbd8f","Type":"ContainerDied","Data":"45d6357f631cc6f1048a70b115b5dfc02bcf8f357754cebc6631432be7fc52f8"}
Sep 18 13:48:38.229069 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:38.229063    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754079ksx"
Sep 18 13:48:38.229265 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:38.229087    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45d6357f631cc6f1048a70b115b5dfc02bcf8f357754cebc6631432be7fc52f8"
Sep 18 13:48:39.566732 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:39.566628    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:48:40.362684 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:40.362646    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:48:41.526488 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:41.526456    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2gk5w"]
Sep 18 13:48:41.526868 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:41.526724    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2078d813-8506-4581-af08-29e90b6cbd8f" containerName="pull"
Sep 18 13:48:41.526868 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:41.526734    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2078d813-8506-4581-af08-29e90b6cbd8f" containerName="pull"
Sep 18 13:48:41.526868 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:41.526745    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2078d813-8506-4581-af08-29e90b6cbd8f" containerName="util"
Sep 18 13:48:41.526868 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:41.526750    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2078d813-8506-4581-af08-29e90b6cbd8f" containerName="util"
Sep 18 13:48:41.526868 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:41.526761    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2078d813-8506-4581-af08-29e90b6cbd8f" containerName="extract"
Sep 18 13:48:41.526868 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:41.526767    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2078d813-8506-4581-af08-29e90b6cbd8f" containerName="extract"
Sep 18 13:48:41.526868 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:41.526808    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2078d813-8506-4581-af08-29e90b6cbd8f" containerName="extract"
Sep 18 13:48:41.528643 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:41.528628    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2gk5w"
Sep 18 13:48:41.530764 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:41.530743    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 18 13:48:41.530899 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:41.530785    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 18 13:48:41.530899 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:41.530856    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-57fjb\""
Sep 18 13:48:41.541995 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:41.541972    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2gk5w"]
Sep 18 13:48:41.618811 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:41.618771    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfw7g\" (UniqueName: \"kubernetes.io/projected/e9a5b16e-c81a-4569-8f0b-7da35af829e6-kube-api-access-bfw7g\") pod \"cert-manager-operator-controller-manager-55c444bf86-2gk5w\" (UID: \"e9a5b16e-c81a-4569-8f0b-7da35af829e6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2gk5w"
Sep 18 13:48:41.619003 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:41.618844    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e9a5b16e-c81a-4569-8f0b-7da35af829e6-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-2gk5w\" (UID: \"e9a5b16e-c81a-4569-8f0b-7da35af829e6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2gk5w"
Sep 18 13:48:41.720090 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:41.720057    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfw7g\" (UniqueName: \"kubernetes.io/projected/e9a5b16e-c81a-4569-8f0b-7da35af829e6-kube-api-access-bfw7g\") pod \"cert-manager-operator-controller-manager-55c444bf86-2gk5w\" (UID: \"e9a5b16e-c81a-4569-8f0b-7da35af829e6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2gk5w"
Sep 18 13:48:41.720189 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:41.720104    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e9a5b16e-c81a-4569-8f0b-7da35af829e6-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-2gk5w\" (UID: \"e9a5b16e-c81a-4569-8f0b-7da35af829e6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2gk5w"
Sep 18 13:48:41.720492 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:41.720476    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e9a5b16e-c81a-4569-8f0b-7da35af829e6-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-2gk5w\" (UID: \"e9a5b16e-c81a-4569-8f0b-7da35af829e6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2gk5w"
Sep 18 13:48:41.728658 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:41.728623    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfw7g\" (UniqueName: \"kubernetes.io/projected/e9a5b16e-c81a-4569-8f0b-7da35af829e6-kube-api-access-bfw7g\") pod \"cert-manager-operator-controller-manager-55c444bf86-2gk5w\" (UID: \"e9a5b16e-c81a-4569-8f0b-7da35af829e6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2gk5w"
Sep 18 13:48:41.837692 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:41.837659    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2gk5w"
Sep 18 13:48:42.121357 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:42.121280    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2gk5w"]
Sep 18 13:48:42.124582 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:48:42.124554    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9a5b16e_c81a_4569_8f0b_7da35af829e6.slice/crio-1b5cc399aa62ad0636e41c900530aeb896c0a61a4520f881c7f5a8427e7f23db WatchSource:0}: Error finding container 1b5cc399aa62ad0636e41c900530aeb896c0a61a4520f881c7f5a8427e7f23db: Status 404 returned error can't find the container with id 1b5cc399aa62ad0636e41c900530aeb896c0a61a4520f881c7f5a8427e7f23db
Sep 18 13:48:42.244242 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:42.244205    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2gk5w" event={"ID":"e9a5b16e-c81a-4569-8f0b-7da35af829e6","Type":"ContainerStarted","Data":"1b5cc399aa62ad0636e41c900530aeb896c0a61a4520f881c7f5a8427e7f23db"}
Sep 18 13:48:43.365160 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:43.365122    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:48:43.465783 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:43.465753    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:48:45.060997 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:45.060966    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:48:45.155622 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:45.155589    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:48:45.265696 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:45.265665    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2gk5w" event={"ID":"e9a5b16e-c81a-4569-8f0b-7da35af829e6","Type":"ContainerStarted","Data":"774f7257d8a758be2ea82e10dce383086e14eff0a0cb62992458187d3e7bc1e0"}
Sep 18 13:48:45.285366 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:45.285317    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2gk5w" podStartSLOduration=2.072941171 podStartE2EDuration="4.285300823s" totalImagesPullingTime="2.212359652s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:48:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:48:45.283229607 +0000 UTC m=+274.335888935" watchObservedRunningTime="2026-09-18 13:48:45.285300823 +0000 UTC m=+274.337960150"
Sep 18 13:48:45.427761 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:45.427730    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-rgdqm"]
Sep 18 13:48:45.430725 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:45.430707    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-rgdqm"
Sep 18 13:48:45.432706 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:45.432678    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 18 13:48:45.432817 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:45.432800    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 18 13:48:45.432895 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:45.432882    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-j7mmg\""
Sep 18 13:48:45.440322 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:45.440299    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-rgdqm"]
Sep 18 13:48:45.552914 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:45.552877    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ee1a5ab2-d919-4c00-a35b-fb3cd73e0d63-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-rgdqm\" (UID: \"ee1a5ab2-d919-4c00-a35b-fb3cd73e0d63\") " pod="cert-manager/cert-manager-webhook-6486ffd997-rgdqm"
Sep 18 13:48:45.553113 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:45.552931    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nxm8\" (UniqueName: \"kubernetes.io/projected/ee1a5ab2-d919-4c00-a35b-fb3cd73e0d63-kube-api-access-6nxm8\") pod \"cert-manager-webhook-6486ffd997-rgdqm\" (UID: \"ee1a5ab2-d919-4c00-a35b-fb3cd73e0d63\") " pod="cert-manager/cert-manager-webhook-6486ffd997-rgdqm"
Sep 18 13:48:45.653952 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:45.653912    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ee1a5ab2-d919-4c00-a35b-fb3cd73e0d63-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-rgdqm\" (UID: \"ee1a5ab2-d919-4c00-a35b-fb3cd73e0d63\") " pod="cert-manager/cert-manager-webhook-6486ffd997-rgdqm"
Sep 18 13:48:45.653952 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:45.653957    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nxm8\" (UniqueName: \"kubernetes.io/projected/ee1a5ab2-d919-4c00-a35b-fb3cd73e0d63-kube-api-access-6nxm8\") pod \"cert-manager-webhook-6486ffd997-rgdqm\" (UID: \"ee1a5ab2-d919-4c00-a35b-fb3cd73e0d63\") " pod="cert-manager/cert-manager-webhook-6486ffd997-rgdqm"
Sep 18 13:48:45.669611 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:45.669580    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ee1a5ab2-d919-4c00-a35b-fb3cd73e0d63-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-rgdqm\" (UID: \"ee1a5ab2-d919-4c00-a35b-fb3cd73e0d63\") " pod="cert-manager/cert-manager-webhook-6486ffd997-rgdqm"
Sep 18 13:48:45.669751 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:45.669657    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nxm8\" (UniqueName: \"kubernetes.io/projected/ee1a5ab2-d919-4c00-a35b-fb3cd73e0d63-kube-api-access-6nxm8\") pod \"cert-manager-webhook-6486ffd997-rgdqm\" (UID: \"ee1a5ab2-d919-4c00-a35b-fb3cd73e0d63\") " pod="cert-manager/cert-manager-webhook-6486ffd997-rgdqm"
Sep 18 13:48:45.712974 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:45.712900    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:48:45.739947 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:45.739915    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-rgdqm"
Sep 18 13:48:45.893107 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:45.893072    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:48:46.051472 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:46.051312    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-rgdqm"]
Sep 18 13:48:46.054029 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:48:46.053997    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee1a5ab2_d919_4c00_a35b_fb3cd73e0d63.slice/crio-39a8dda6a0fd9e12cb0440c309d961c3b7353ffe9cb9affb284306d1b31ce54e WatchSource:0}: Error finding container 39a8dda6a0fd9e12cb0440c309d961c3b7353ffe9cb9affb284306d1b31ce54e: Status 404 returned error can't find the container with id 39a8dda6a0fd9e12cb0440c309d961c3b7353ffe9cb9affb284306d1b31ce54e
Sep 18 13:48:46.270926 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:46.270887    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-rgdqm" event={"ID":"ee1a5ab2-d919-4c00-a35b-fb3cd73e0d63","Type":"ContainerStarted","Data":"39a8dda6a0fd9e12cb0440c309d961c3b7353ffe9cb9affb284306d1b31ce54e"}
Sep 18 13:48:46.328882 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:46.328788    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9fxjb"]
Sep 18 13:48:46.333013 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:46.332993    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9fxjb"
Sep 18 13:48:46.335179 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:46.335157    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-l2rh5\""
Sep 18 13:48:46.335296 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:46.335205    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 18 13:48:46.335565 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:46.335551    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 18 13:48:46.349136 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:46.349116    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9fxjb"]
Sep 18 13:48:46.359934 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:46.359913    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ec93af33-0338-4e31-889a-2e63af568371-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9fxjb\" (UID: \"ec93af33-0338-4e31-889a-2e63af568371\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9fxjb"
Sep 18 13:48:46.360027 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:46.359955    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtdgq\" (UniqueName: \"kubernetes.io/projected/ec93af33-0338-4e31-889a-2e63af568371-kube-api-access-mtdgq\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9fxjb\" (UID: \"ec93af33-0338-4e31-889a-2e63af568371\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9fxjb"
Sep 18 13:48:46.360086 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:46.360025    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ec93af33-0338-4e31-889a-2e63af568371-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9fxjb\" (UID: \"ec93af33-0338-4e31-889a-2e63af568371\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9fxjb"
Sep 18 13:48:46.461288 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:46.461261    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ec93af33-0338-4e31-889a-2e63af568371-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9fxjb\" (UID: \"ec93af33-0338-4e31-889a-2e63af568371\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9fxjb"
Sep 18 13:48:46.461485 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:46.461300    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ec93af33-0338-4e31-889a-2e63af568371-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9fxjb\" (UID: \"ec93af33-0338-4e31-889a-2e63af568371\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9fxjb"
Sep 18 13:48:46.461485 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:46.461331    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtdgq\" (UniqueName: \"kubernetes.io/projected/ec93af33-0338-4e31-889a-2e63af568371-kube-api-access-mtdgq\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9fxjb\" (UID: \"ec93af33-0338-4e31-889a-2e63af568371\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9fxjb"
Sep 18 13:48:46.461715 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:46.461693    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ec93af33-0338-4e31-889a-2e63af568371-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9fxjb\" (UID: \"ec93af33-0338-4e31-889a-2e63af568371\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9fxjb"
Sep 18 13:48:46.461715 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:46.461706    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ec93af33-0338-4e31-889a-2e63af568371-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9fxjb\" (UID: \"ec93af33-0338-4e31-889a-2e63af568371\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9fxjb"
Sep 18 13:48:46.469542 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:46.469515    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtdgq\" (UniqueName: \"kubernetes.io/projected/ec93af33-0338-4e31-889a-2e63af568371-kube-api-access-mtdgq\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9fxjb\" (UID: \"ec93af33-0338-4e31-889a-2e63af568371\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9fxjb"
Sep 18 13:48:46.641755 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:46.641725    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9fxjb"
Sep 18 13:48:46.917195 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:46.917172    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9fxjb"]
Sep 18 13:48:46.919380 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:48:46.919344    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec93af33_0338_4e31_889a_2e63af568371.slice/crio-ce319cb7bde2f3abbab3f4280f58d69dcab7525816565c13c6c41db99790a873 WatchSource:0}: Error finding container ce319cb7bde2f3abbab3f4280f58d69dcab7525816565c13c6c41db99790a873: Status 404 returned error can't find the container with id ce319cb7bde2f3abbab3f4280f58d69dcab7525816565c13c6c41db99790a873
Sep 18 13:48:47.117380 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:47.117347    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:48:47.277223 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:47.277123    2577 generic.go:356] "Generic (PLEG): container finished" podID="ec93af33-0338-4e31-889a-2e63af568371" containerID="e56c2a4f9f01714a9e9af41e0466b2604ead51b0d900c4ec5da2590b45aa3bee" exitCode=0
Sep 18 13:48:47.277223 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:47.277206    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9fxjb" event={"ID":"ec93af33-0338-4e31-889a-2e63af568371","Type":"ContainerDied","Data":"e56c2a4f9f01714a9e9af41e0466b2604ead51b0d900c4ec5da2590b45aa3bee"}
Sep 18 13:48:47.277669 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:47.277256    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9fxjb" event={"ID":"ec93af33-0338-4e31-889a-2e63af568371","Type":"ContainerStarted","Data":"ce319cb7bde2f3abbab3f4280f58d69dcab7525816565c13c6c41db99790a873"}
Sep 18 13:48:47.453641 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:47.453611    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:48:49.293448 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:49.293414    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9fxjb" event={"ID":"ec93af33-0338-4e31-889a-2e63af568371","Type":"ContainerStarted","Data":"6bbeec004a449a888958e1c856ad6f4d7c1ebdd558858ee96cceafeb9a2e16fb"}
Sep 18 13:48:50.299521 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:50.299487    2577 generic.go:356] "Generic (PLEG): container finished" podID="ec93af33-0338-4e31-889a-2e63af568371" containerID="6bbeec004a449a888958e1c856ad6f4d7c1ebdd558858ee96cceafeb9a2e16fb" exitCode=0
Sep 18 13:48:50.300032 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:50.299571    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9fxjb" event={"ID":"ec93af33-0338-4e31-889a-2e63af568371","Type":"ContainerDied","Data":"6bbeec004a449a888958e1c856ad6f4d7c1ebdd558858ee96cceafeb9a2e16fb"}
Sep 18 13:48:50.372413 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:50.372386    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:48:50.473439 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:50.473403    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:48:51.306272 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:51.306240    2577 generic.go:356] "Generic (PLEG): container finished" podID="ec93af33-0338-4e31-889a-2e63af568371" containerID="c90019e22f3f0caf43ac463c6299cc91f8198e124da0aee20ecefc00ab3c918a" exitCode=0
Sep 18 13:48:51.306697 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:51.306326    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9fxjb" event={"ID":"ec93af33-0338-4e31-889a-2e63af568371","Type":"ContainerDied","Data":"c90019e22f3f0caf43ac463c6299cc91f8198e124da0aee20ecefc00ab3c918a"}
Sep 18 13:48:52.432980 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:52.432958    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9fxjb"
Sep 18 13:48:52.512128 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:52.512089    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ec93af33-0338-4e31-889a-2e63af568371-bundle\") pod \"ec93af33-0338-4e31-889a-2e63af568371\" (UID: \"ec93af33-0338-4e31-889a-2e63af568371\") "
Sep 18 13:48:52.512271 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:52.512155    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mtdgq\" (UniqueName: \"kubernetes.io/projected/ec93af33-0338-4e31-889a-2e63af568371-kube-api-access-mtdgq\") pod \"ec93af33-0338-4e31-889a-2e63af568371\" (UID: \"ec93af33-0338-4e31-889a-2e63af568371\") "
Sep 18 13:48:52.512271 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:52.512199    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ec93af33-0338-4e31-889a-2e63af568371-util\") pod \"ec93af33-0338-4e31-889a-2e63af568371\" (UID: \"ec93af33-0338-4e31-889a-2e63af568371\") "
Sep 18 13:48:52.512587 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:52.512547    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec93af33-0338-4e31-889a-2e63af568371-bundle" (OuterVolumeSpecName: "bundle") pod "ec93af33-0338-4e31-889a-2e63af568371" (UID: "ec93af33-0338-4e31-889a-2e63af568371"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:48:52.514383 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:52.514354    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec93af33-0338-4e31-889a-2e63af568371-kube-api-access-mtdgq" (OuterVolumeSpecName: "kube-api-access-mtdgq") pod "ec93af33-0338-4e31-889a-2e63af568371" (UID: "ec93af33-0338-4e31-889a-2e63af568371"). InnerVolumeSpecName "kube-api-access-mtdgq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:48:52.516935 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:52.516912    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec93af33-0338-4e31-889a-2e63af568371-util" (OuterVolumeSpecName: "util") pod "ec93af33-0338-4e31-889a-2e63af568371" (UID: "ec93af33-0338-4e31-889a-2e63af568371"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:48:52.612876 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:52.612838    2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ec93af33-0338-4e31-889a-2e63af568371-bundle\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:48:52.612876 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:52.612874    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtdgq\" (UniqueName: \"kubernetes.io/projected/ec93af33-0338-4e31-889a-2e63af568371-kube-api-access-mtdgq\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:48:52.613066 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:52.612887    2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ec93af33-0338-4e31-889a-2e63af568371-util\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:48:53.316716 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:53.316683    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9fxjb"
Sep 18 13:48:53.316716 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:53.316703    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9fxjb" event={"ID":"ec93af33-0338-4e31-889a-2e63af568371","Type":"ContainerDied","Data":"ce319cb7bde2f3abbab3f4280f58d69dcab7525816565c13c6c41db99790a873"}
Sep 18 13:48:53.316924 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:53.316751    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce319cb7bde2f3abbab3f4280f58d69dcab7525816565c13c6c41db99790a873"
Sep 18 13:48:55.664937 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:55.664907    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:48:56.989712 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:56.989669    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:48:57.088781 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:57.088738    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:48:57.992721 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:57.992685    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-hln9n"]
Sep 18 13:48:57.993183 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:57.993012    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec93af33-0338-4e31-889a-2e63af568371" containerName="pull"
Sep 18 13:48:57.993183 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:57.993029    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec93af33-0338-4e31-889a-2e63af568371" containerName="pull"
Sep 18 13:48:57.993183 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:57.993043    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec93af33-0338-4e31-889a-2e63af568371" containerName="util"
Sep 18 13:48:57.993183 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:57.993048    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec93af33-0338-4e31-889a-2e63af568371" containerName="util"
Sep 18 13:48:57.993183 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:57.993053    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec93af33-0338-4e31-889a-2e63af568371" containerName="extract"
Sep 18 13:48:57.993183 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:57.993060    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec93af33-0338-4e31-889a-2e63af568371" containerName="extract"
Sep 18 13:48:57.993183 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:57.993103    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec93af33-0338-4e31-889a-2e63af568371" containerName="extract"
Sep 18 13:48:58.030333 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:58.030292    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-hln9n"]
Sep 18 13:48:58.030514 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:58.030398    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-hln9n"
Sep 18 13:48:58.032677 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:58.032650    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-gb56t\""
Sep 18 13:48:58.159501 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:58.159459    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b308d646-ab63-42e2-83fe-dd9a04eaebaf-bound-sa-token\") pod \"cert-manager-55b4d954d5-hln9n\" (UID: \"b308d646-ab63-42e2-83fe-dd9a04eaebaf\") " pod="cert-manager/cert-manager-55b4d954d5-hln9n"
Sep 18 13:48:58.159674 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:58.159606    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8ctz\" (UniqueName: \"kubernetes.io/projected/b308d646-ab63-42e2-83fe-dd9a04eaebaf-kube-api-access-r8ctz\") pod \"cert-manager-55b4d954d5-hln9n\" (UID: \"b308d646-ab63-42e2-83fe-dd9a04eaebaf\") " pod="cert-manager/cert-manager-55b4d954d5-hln9n"
Sep 18 13:48:58.260602 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:58.260520    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8ctz\" (UniqueName: \"kubernetes.io/projected/b308d646-ab63-42e2-83fe-dd9a04eaebaf-kube-api-access-r8ctz\") pod \"cert-manager-55b4d954d5-hln9n\" (UID: \"b308d646-ab63-42e2-83fe-dd9a04eaebaf\") " pod="cert-manager/cert-manager-55b4d954d5-hln9n"
Sep 18 13:48:58.260602 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:58.260560    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b308d646-ab63-42e2-83fe-dd9a04eaebaf-bound-sa-token\") pod \"cert-manager-55b4d954d5-hln9n\" (UID: \"b308d646-ab63-42e2-83fe-dd9a04eaebaf\") " pod="cert-manager/cert-manager-55b4d954d5-hln9n"
Sep 18 13:48:58.268611 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:58.268584    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b308d646-ab63-42e2-83fe-dd9a04eaebaf-bound-sa-token\") pod \"cert-manager-55b4d954d5-hln9n\" (UID: \"b308d646-ab63-42e2-83fe-dd9a04eaebaf\") " pod="cert-manager/cert-manager-55b4d954d5-hln9n"
Sep 18 13:48:58.268838 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:58.268804    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8ctz\" (UniqueName: \"kubernetes.io/projected/b308d646-ab63-42e2-83fe-dd9a04eaebaf-kube-api-access-r8ctz\") pod \"cert-manager-55b4d954d5-hln9n\" (UID: \"b308d646-ab63-42e2-83fe-dd9a04eaebaf\") " pod="cert-manager/cert-manager-55b4d954d5-hln9n"
Sep 18 13:48:58.341038 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:58.341002    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-hln9n"
Sep 18 13:48:58.567928 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:58.567846    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:48:58.620764 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:58.620669    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-hln9n"]
Sep 18 13:48:58.623298 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:48:58.623269    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb308d646_ab63_42e2_83fe_dd9a04eaebaf.slice/crio-a473decbf612036a4f4644d97e63b7a7733122bc5647e34242f5b74fe59fdf63 WatchSource:0}: Error finding container a473decbf612036a4f4644d97e63b7a7733122bc5647e34242f5b74fe59fdf63: Status 404 returned error can't find the container with id a473decbf612036a4f4644d97e63b7a7733122bc5647e34242f5b74fe59fdf63
Sep 18 13:48:58.668633 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:58.668608    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:48:59.049845 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:59.046666    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5m9bf"]
Sep 18 13:48:59.051089 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:59.051060    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5m9bf"
Sep 18 13:48:59.053511 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:59.053485    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 18 13:48:59.053640 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:59.053543    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-vxn29\""
Sep 18 13:48:59.053884 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:59.053870    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 18 13:48:59.060602 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:59.060574    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5m9bf"]
Sep 18 13:48:59.168123 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:59.168088    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9082649d-4dfb-4491-b8f6-854fb3ef7485-tmp\") pod \"openshift-lws-operator-fd8ccff4c-5m9bf\" (UID: \"9082649d-4dfb-4491-b8f6-854fb3ef7485\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5m9bf"
Sep 18 13:48:59.168297 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:59.168133    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbskb\" (UniqueName: \"kubernetes.io/projected/9082649d-4dfb-4491-b8f6-854fb3ef7485-kube-api-access-lbskb\") pod \"openshift-lws-operator-fd8ccff4c-5m9bf\" (UID: \"9082649d-4dfb-4491-b8f6-854fb3ef7485\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5m9bf"
Sep 18 13:48:59.268638 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:59.268597    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9082649d-4dfb-4491-b8f6-854fb3ef7485-tmp\") pod \"openshift-lws-operator-fd8ccff4c-5m9bf\" (UID: \"9082649d-4dfb-4491-b8f6-854fb3ef7485\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5m9bf"
Sep 18 13:48:59.268862 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:59.268661    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbskb\" (UniqueName: \"kubernetes.io/projected/9082649d-4dfb-4491-b8f6-854fb3ef7485-kube-api-access-lbskb\") pod \"openshift-lws-operator-fd8ccff4c-5m9bf\" (UID: \"9082649d-4dfb-4491-b8f6-854fb3ef7485\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5m9bf"
Sep 18 13:48:59.269109 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:59.269085    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9082649d-4dfb-4491-b8f6-854fb3ef7485-tmp\") pod \"openshift-lws-operator-fd8ccff4c-5m9bf\" (UID: \"9082649d-4dfb-4491-b8f6-854fb3ef7485\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5m9bf"
Sep 18 13:48:59.277122 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:59.277099    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbskb\" (UniqueName: \"kubernetes.io/projected/9082649d-4dfb-4491-b8f6-854fb3ef7485-kube-api-access-lbskb\") pod \"openshift-lws-operator-fd8ccff4c-5m9bf\" (UID: \"9082649d-4dfb-4491-b8f6-854fb3ef7485\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5m9bf"
Sep 18 13:48:59.351637 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:59.351601    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-hln9n" event={"ID":"b308d646-ab63-42e2-83fe-dd9a04eaebaf","Type":"ContainerStarted","Data":"92c6eafab7216bd22825a0266577470733b5b5e16745c7418d5598882662329a"}
Sep 18 13:48:59.351864 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:59.351648    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-hln9n" event={"ID":"b308d646-ab63-42e2-83fe-dd9a04eaebaf","Type":"ContainerStarted","Data":"a473decbf612036a4f4644d97e63b7a7733122bc5647e34242f5b74fe59fdf63"}
Sep 18 13:48:59.353239 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:59.353217    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-rgdqm" event={"ID":"ee1a5ab2-d919-4c00-a35b-fb3cd73e0d63","Type":"ContainerStarted","Data":"45ae1328ee331b6774e4026b24461c224c3be8a786a8a19a273ac0722bec3bfb"}
Sep 18 13:48:59.353360 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:59.353341    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-rgdqm"
Sep 18 13:48:59.361353 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:59.361336    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5m9bf"
Sep 18 13:48:59.372297 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:59.372137    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-hln9n" podStartSLOduration=2.372121748 podStartE2EDuration="2.372121748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:48:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:48:59.370535944 +0000 UTC m=+288.423195271" watchObservedRunningTime="2026-09-18 13:48:59.372121748 +0000 UTC m=+288.424781077"
Sep 18 13:48:59.397991 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:59.397938    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-rgdqm" podStartSLOduration=2.1413619 podStartE2EDuration="14.397918485s" totalImagesPullingTime="12.256556585s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:48:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:48:59.397602927 +0000 UTC m=+288.450262245" watchObservedRunningTime="2026-09-18 13:48:59.397918485 +0000 UTC m=+288.450577811"
Sep 18 13:48:59.636396 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:48:59.636373    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5m9bf"]
Sep 18 13:48:59.638965 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:48:59.638933    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9082649d_4dfb_4491_b8f6_854fb3ef7485.slice/crio-101d5d0a8e6d09d51dfdd73db3f22efe65e60aa3aebaa4d6b04c8835d2b1c218 WatchSource:0}: Error finding container 101d5d0a8e6d09d51dfdd73db3f22efe65e60aa3aebaa4d6b04c8835d2b1c218: Status 404 returned error can't find the container with id 101d5d0a8e6d09d51dfdd73db3f22efe65e60aa3aebaa4d6b04c8835d2b1c218
Sep 18 13:49:00.359082 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:00.359045    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5m9bf" event={"ID":"9082649d-4dfb-4491-b8f6-854fb3ef7485","Type":"ContainerStarted","Data":"101d5d0a8e6d09d51dfdd73db3f22efe65e60aa3aebaa4d6b04c8835d2b1c218"}
Sep 18 13:49:02.369642 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:02.369607    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5m9bf" event={"ID":"9082649d-4dfb-4491-b8f6-854fb3ef7485","Type":"ContainerStarted","Data":"a072698cceee86a7801616b71b9b77f698a00d636fe3b6a0c7b80e7fba5a24dc"}
Sep 18 13:49:02.387466 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:02.387414    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5m9bf" podStartSLOduration=1.5198650740000001 podStartE2EDuration="3.387397037s" totalImagesPullingTime="1.867531963s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:48:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:49:02.385884839 +0000 UTC m=+291.438544170" watchObservedRunningTime="2026-09-18 13:49:02.387397037 +0000 UTC m=+291.440056360"
Sep 18 13:49:03.150274 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:03.150235    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:49:03.254814 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:03.254778    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:49:04.847575 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:04.847531    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:49:05.361616 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:05.361586    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-rgdqm"
Sep 18 13:49:06.797548 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:06.797515    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af98rqh"]
Sep 18 13:49:06.801935 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:06.801915    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af98rqh"
Sep 18 13:49:06.804079 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:06.804049    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 18 13:49:06.804182 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:06.804060    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-l2rh5\""
Sep 18 13:49:06.804481 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:06.804465    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 18 13:49:06.808877 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:06.808817    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af98rqh"]
Sep 18 13:49:06.932201 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:06.932163    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2a19c901-19a1-4d78-82c2-1411730aba4c-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af98rqh\" (UID: \"2a19c901-19a1-4d78-82c2-1411730aba4c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af98rqh"
Sep 18 13:49:06.932350 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:06.932214    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2a19c901-19a1-4d78-82c2-1411730aba4c-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af98rqh\" (UID: \"2a19c901-19a1-4d78-82c2-1411730aba4c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af98rqh"
Sep 18 13:49:06.932350 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:06.932258    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn5mq\" (UniqueName: \"kubernetes.io/projected/2a19c901-19a1-4d78-82c2-1411730aba4c-kube-api-access-gn5mq\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af98rqh\" (UID: \"2a19c901-19a1-4d78-82c2-1411730aba4c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af98rqh"
Sep 18 13:49:07.033451 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:07.033411    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2a19c901-19a1-4d78-82c2-1411730aba4c-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af98rqh\" (UID: \"2a19c901-19a1-4d78-82c2-1411730aba4c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af98rqh"
Sep 18 13:49:07.033633 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:07.033465    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2a19c901-19a1-4d78-82c2-1411730aba4c-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af98rqh\" (UID: \"2a19c901-19a1-4d78-82c2-1411730aba4c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af98rqh"
Sep 18 13:49:07.033633 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:07.033498    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn5mq\" (UniqueName: \"kubernetes.io/projected/2a19c901-19a1-4d78-82c2-1411730aba4c-kube-api-access-gn5mq\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af98rqh\" (UID: \"2a19c901-19a1-4d78-82c2-1411730aba4c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af98rqh"
Sep 18 13:49:07.033948 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:07.033924    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2a19c901-19a1-4d78-82c2-1411730aba4c-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af98rqh\" (UID: \"2a19c901-19a1-4d78-82c2-1411730aba4c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af98rqh"
Sep 18 13:49:07.033999 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:07.033924    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2a19c901-19a1-4d78-82c2-1411730aba4c-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af98rqh\" (UID: \"2a19c901-19a1-4d78-82c2-1411730aba4c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af98rqh"
Sep 18 13:49:07.042047 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:07.042019    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn5mq\" (UniqueName: \"kubernetes.io/projected/2a19c901-19a1-4d78-82c2-1411730aba4c-kube-api-access-gn5mq\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af98rqh\" (UID: \"2a19c901-19a1-4d78-82c2-1411730aba4c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af98rqh"
Sep 18 13:49:07.112392 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:07.112359    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af98rqh"
Sep 18 13:49:07.384458 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:07.384395    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af98rqh"]
Sep 18 13:49:07.387456 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:49:07.387430    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a19c901_19a1_4d78_82c2_1411730aba4c.slice/crio-8d29844cdad4f43345c93258c4795395d10018ffac24b093a36fafc305effcc4 WatchSource:0}: Error finding container 8d29844cdad4f43345c93258c4795395d10018ffac24b093a36fafc305effcc4: Status 404 returned error can't find the container with id 8d29844cdad4f43345c93258c4795395d10018ffac24b093a36fafc305effcc4
Sep 18 13:49:07.394664 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:07.394638    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af98rqh" event={"ID":"2a19c901-19a1-4d78-82c2-1411730aba4c","Type":"ContainerStarted","Data":"8d29844cdad4f43345c93258c4795395d10018ffac24b093a36fafc305effcc4"}
Sep 18 13:49:08.400482 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:08.400447    2577 generic.go:356] "Generic (PLEG): container finished" podID="2a19c901-19a1-4d78-82c2-1411730aba4c" containerID="909a377de2bb7c0ce23135cb9e5941f37447e8d7409d31c7b021b81595dcaa37" exitCode=0
Sep 18 13:49:08.400893 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:08.400521    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af98rqh" event={"ID":"2a19c901-19a1-4d78-82c2-1411730aba4c","Type":"ContainerDied","Data":"909a377de2bb7c0ce23135cb9e5941f37447e8d7409d31c7b021b81595dcaa37"}
Sep 18 13:49:09.406473 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:09.406440    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af98rqh" event={"ID":"2a19c901-19a1-4d78-82c2-1411730aba4c","Type":"ContainerStarted","Data":"689ee1b52801c25ccf0a5fca8e7f62d68938f3429d3b957b16c71e76929b8a57"}
Sep 18 13:49:10.412670 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:10.412635    2577 generic.go:356] "Generic (PLEG): container finished" podID="2a19c901-19a1-4d78-82c2-1411730aba4c" containerID="689ee1b52801c25ccf0a5fca8e7f62d68938f3429d3b957b16c71e76929b8a57" exitCode=0
Sep 18 13:49:10.413070 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:10.412691    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af98rqh" event={"ID":"2a19c901-19a1-4d78-82c2-1411730aba4c","Type":"ContainerDied","Data":"689ee1b52801c25ccf0a5fca8e7f62d68938f3429d3b957b16c71e76929b8a57"}
Sep 18 13:49:11.419047 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:11.419010    2577 generic.go:356] "Generic (PLEG): container finished" podID="2a19c901-19a1-4d78-82c2-1411730aba4c" containerID="9af482e3de46d7ab65c50a0255f1ac671b2ca9bb247cf91b9630ee6b616b87bf" exitCode=0
Sep 18 13:49:11.419428 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:11.419086    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af98rqh" event={"ID":"2a19c901-19a1-4d78-82c2-1411730aba4c","Type":"ContainerDied","Data":"9af482e3de46d7ab65c50a0255f1ac671b2ca9bb247cf91b9630ee6b616b87bf"}
Sep 18 13:49:12.551493 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:12.551466    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af98rqh"
Sep 18 13:49:12.680093 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:12.679998    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gn5mq\" (UniqueName: \"kubernetes.io/projected/2a19c901-19a1-4d78-82c2-1411730aba4c-kube-api-access-gn5mq\") pod \"2a19c901-19a1-4d78-82c2-1411730aba4c\" (UID: \"2a19c901-19a1-4d78-82c2-1411730aba4c\") "
Sep 18 13:49:12.680093 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:12.680040    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2a19c901-19a1-4d78-82c2-1411730aba4c-util\") pod \"2a19c901-19a1-4d78-82c2-1411730aba4c\" (UID: \"2a19c901-19a1-4d78-82c2-1411730aba4c\") "
Sep 18 13:49:12.680093 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:12.680078    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2a19c901-19a1-4d78-82c2-1411730aba4c-bundle\") pod \"2a19c901-19a1-4d78-82c2-1411730aba4c\" (UID: \"2a19c901-19a1-4d78-82c2-1411730aba4c\") "
Sep 18 13:49:12.680879 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:12.680816    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a19c901-19a1-4d78-82c2-1411730aba4c-bundle" (OuterVolumeSpecName: "bundle") pod "2a19c901-19a1-4d78-82c2-1411730aba4c" (UID: "2a19c901-19a1-4d78-82c2-1411730aba4c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:49:12.682160 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:12.682135    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a19c901-19a1-4d78-82c2-1411730aba4c-kube-api-access-gn5mq" (OuterVolumeSpecName: "kube-api-access-gn5mq") pod "2a19c901-19a1-4d78-82c2-1411730aba4c" (UID: "2a19c901-19a1-4d78-82c2-1411730aba4c"). InnerVolumeSpecName "kube-api-access-gn5mq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:49:12.711239 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:12.711202    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a19c901-19a1-4d78-82c2-1411730aba4c-util" (OuterVolumeSpecName: "util") pod "2a19c901-19a1-4d78-82c2-1411730aba4c" (UID: "2a19c901-19a1-4d78-82c2-1411730aba4c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:49:12.781357 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:12.781322    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gn5mq\" (UniqueName: \"kubernetes.io/projected/2a19c901-19a1-4d78-82c2-1411730aba4c-kube-api-access-gn5mq\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:49:12.781357 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:12.781353    2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2a19c901-19a1-4d78-82c2-1411730aba4c-util\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:49:12.781357 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:12.781365    2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2a19c901-19a1-4d78-82c2-1411730aba4c-bundle\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:49:13.051073 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:13.050987    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-8tnxl_69d3178a-5769-4b5f-a243-30a1bedc5bd6/console-operator/1.log"
Sep 18 13:49:13.054194 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:13.054162    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-8tnxl_69d3178a-5769-4b5f-a243-30a1bedc5bd6/console-operator/1.log"
Sep 18 13:49:13.062626 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:13.062604    2577 kubelet.go:1648] "Image garbage collection succeeded"
Sep 18 13:49:13.429050 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:13.429016    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af98rqh" event={"ID":"2a19c901-19a1-4d78-82c2-1411730aba4c","Type":"ContainerDied","Data":"8d29844cdad4f43345c93258c4795395d10018ffac24b093a36fafc305effcc4"}
Sep 18 13:49:13.429050 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:13.429040    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af98rqh"
Sep 18 13:49:13.429301 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:13.429094    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d29844cdad4f43345c93258c4795395d10018ffac24b093a36fafc305effcc4"
Sep 18 13:49:22.232016 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:22.231961    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tsfm6"]
Sep 18 13:49:22.232501 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:22.232369    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a19c901-19a1-4d78-82c2-1411730aba4c" containerName="extract"
Sep 18 13:49:22.232501 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:22.232388    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a19c901-19a1-4d78-82c2-1411730aba4c" containerName="extract"
Sep 18 13:49:22.232501 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:22.232402    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a19c901-19a1-4d78-82c2-1411730aba4c" containerName="util"
Sep 18 13:49:22.232501 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:22.232410    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a19c901-19a1-4d78-82c2-1411730aba4c" containerName="util"
Sep 18 13:49:22.232501 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:22.232436    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a19c901-19a1-4d78-82c2-1411730aba4c" containerName="pull"
Sep 18 13:49:22.232501 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:22.232444    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a19c901-19a1-4d78-82c2-1411730aba4c" containerName="pull"
Sep 18 13:49:22.232808 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:22.232508    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a19c901-19a1-4d78-82c2-1411730aba4c" containerName="extract"
Sep 18 13:49:22.237052 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:22.237030    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tsfm6"
Sep 18 13:49:22.239350 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:22.239323    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 18 13:49:22.239491 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:22.239354    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-l2rh5\""
Sep 18 13:49:22.239491 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:22.239393    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 18 13:49:22.247841 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:22.247787    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tsfm6"]
Sep 18 13:49:22.358054 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:22.358010    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/75bc2527-be76-41c7-9df5-c9fcea6bd5f6-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tsfm6\" (UID: \"75bc2527-be76-41c7-9df5-c9fcea6bd5f6\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tsfm6"
Sep 18 13:49:22.358054 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:22.358052    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/75bc2527-be76-41c7-9df5-c9fcea6bd5f6-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tsfm6\" (UID: \"75bc2527-be76-41c7-9df5-c9fcea6bd5f6\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tsfm6"
Sep 18 13:49:22.358270 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:22.358100    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw4sl\" (UniqueName: \"kubernetes.io/projected/75bc2527-be76-41c7-9df5-c9fcea6bd5f6-kube-api-access-lw4sl\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tsfm6\" (UID: \"75bc2527-be76-41c7-9df5-c9fcea6bd5f6\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tsfm6"
Sep 18 13:49:22.458728 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:22.458693    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw4sl\" (UniqueName: \"kubernetes.io/projected/75bc2527-be76-41c7-9df5-c9fcea6bd5f6-kube-api-access-lw4sl\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tsfm6\" (UID: \"75bc2527-be76-41c7-9df5-c9fcea6bd5f6\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tsfm6"
Sep 18 13:49:22.458922 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:22.458755    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/75bc2527-be76-41c7-9df5-c9fcea6bd5f6-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tsfm6\" (UID: \"75bc2527-be76-41c7-9df5-c9fcea6bd5f6\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tsfm6"
Sep 18 13:49:22.458922 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:22.458774    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/75bc2527-be76-41c7-9df5-c9fcea6bd5f6-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tsfm6\" (UID: \"75bc2527-be76-41c7-9df5-c9fcea6bd5f6\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tsfm6"
Sep 18 13:49:22.459173 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:22.459158    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/75bc2527-be76-41c7-9df5-c9fcea6bd5f6-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tsfm6\" (UID: \"75bc2527-be76-41c7-9df5-c9fcea6bd5f6\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tsfm6"
Sep 18 13:49:22.459213 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:22.459181    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/75bc2527-be76-41c7-9df5-c9fcea6bd5f6-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tsfm6\" (UID: \"75bc2527-be76-41c7-9df5-c9fcea6bd5f6\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tsfm6"
Sep 18 13:49:22.467653 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:22.467626    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw4sl\" (UniqueName: \"kubernetes.io/projected/75bc2527-be76-41c7-9df5-c9fcea6bd5f6-kube-api-access-lw4sl\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tsfm6\" (UID: \"75bc2527-be76-41c7-9df5-c9fcea6bd5f6\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tsfm6"
Sep 18 13:49:22.547411 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:22.547324    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tsfm6"
Sep 18 13:49:22.821713 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:22.821617    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tsfm6"]
Sep 18 13:49:22.824491 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:49:22.824459    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75bc2527_be76_41c7_9df5_c9fcea6bd5f6.slice/crio-759aeefc6e4aa2f8c1bd75e9eb1238857b1a99a72c39bd15bc8cd32882f5ac42 WatchSource:0}: Error finding container 759aeefc6e4aa2f8c1bd75e9eb1238857b1a99a72c39bd15bc8cd32882f5ac42: Status 404 returned error can't find the container with id 759aeefc6e4aa2f8c1bd75e9eb1238857b1a99a72c39bd15bc8cd32882f5ac42
Sep 18 13:49:23.014742 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:23.014713    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:49:23.473476 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:23.473447    2577 generic.go:356] "Generic (PLEG): container finished" podID="75bc2527-be76-41c7-9df5-c9fcea6bd5f6" containerID="79afe5692ac5fbad6640854bf933c20107138e29c3972ddde8972da0b0ae5efd" exitCode=0
Sep 18 13:49:23.473854 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:23.473532    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tsfm6" event={"ID":"75bc2527-be76-41c7-9df5-c9fcea6bd5f6","Type":"ContainerDied","Data":"79afe5692ac5fbad6640854bf933c20107138e29c3972ddde8972da0b0ae5efd"}
Sep 18 13:49:23.473854 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:23.473584    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tsfm6" event={"ID":"75bc2527-be76-41c7-9df5-c9fcea6bd5f6","Type":"ContainerStarted","Data":"759aeefc6e4aa2f8c1bd75e9eb1238857b1a99a72c39bd15bc8cd32882f5ac42"}
Sep 18 13:49:23.474059 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:23.474043    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 13:49:23.980068 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:23.980030    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-978ccd68d-2l9f7"]
Sep 18 13:49:23.984063 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:23.984039    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-978ccd68d-2l9f7"
Sep 18 13:49:23.988290 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:23.988267    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 18 13:49:23.988810 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:23.988787    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-dlrjx\""
Sep 18 13:49:23.989093 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:23.988787    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 18 13:49:23.989093 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:23.989084    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 18 13:49:23.989944 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:23.989927    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 18 13:49:24.010957 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:24.010927    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-978ccd68d-2l9f7"]
Sep 18 13:49:24.173190 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:24.173155    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/278aa551-51a0-4564-af58-0d20184ab87a-webhook-cert\") pod \"opendatahub-operator-controller-manager-978ccd68d-2l9f7\" (UID: \"278aa551-51a0-4564-af58-0d20184ab87a\") " pod="opendatahub/opendatahub-operator-controller-manager-978ccd68d-2l9f7"
Sep 18 13:49:24.173396 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:24.173199    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/278aa551-51a0-4564-af58-0d20184ab87a-tmp\") pod \"opendatahub-operator-controller-manager-978ccd68d-2l9f7\" (UID: \"278aa551-51a0-4564-af58-0d20184ab87a\") " pod="opendatahub/opendatahub-operator-controller-manager-978ccd68d-2l9f7"
Sep 18 13:49:24.173396 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:24.173227    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/278aa551-51a0-4564-af58-0d20184ab87a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-978ccd68d-2l9f7\" (UID: \"278aa551-51a0-4564-af58-0d20184ab87a\") " pod="opendatahub/opendatahub-operator-controller-manager-978ccd68d-2l9f7"
Sep 18 13:49:24.173396 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:24.173283    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sdbl\" (UniqueName: \"kubernetes.io/projected/278aa551-51a0-4564-af58-0d20184ab87a-kube-api-access-6sdbl\") pod \"opendatahub-operator-controller-manager-978ccd68d-2l9f7\" (UID: \"278aa551-51a0-4564-af58-0d20184ab87a\") " pod="opendatahub/opendatahub-operator-controller-manager-978ccd68d-2l9f7"
Sep 18 13:49:24.173396 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:24.173316    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/278aa551-51a0-4564-af58-0d20184ab87a-manifests\") pod \"opendatahub-operator-controller-manager-978ccd68d-2l9f7\" (UID: \"278aa551-51a0-4564-af58-0d20184ab87a\") " pod="opendatahub/opendatahub-operator-controller-manager-978ccd68d-2l9f7"
Sep 18 13:49:24.274382 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:24.274282    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/278aa551-51a0-4564-af58-0d20184ab87a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-978ccd68d-2l9f7\" (UID: \"278aa551-51a0-4564-af58-0d20184ab87a\") " pod="opendatahub/opendatahub-operator-controller-manager-978ccd68d-2l9f7"
Sep 18 13:49:24.274382 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:24.274338    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sdbl\" (UniqueName: \"kubernetes.io/projected/278aa551-51a0-4564-af58-0d20184ab87a-kube-api-access-6sdbl\") pod \"opendatahub-operator-controller-manager-978ccd68d-2l9f7\" (UID: \"278aa551-51a0-4564-af58-0d20184ab87a\") " pod="opendatahub/opendatahub-operator-controller-manager-978ccd68d-2l9f7"
Sep 18 13:49:24.274382 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:24.274376    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/278aa551-51a0-4564-af58-0d20184ab87a-manifests\") pod \"opendatahub-operator-controller-manager-978ccd68d-2l9f7\" (UID: \"278aa551-51a0-4564-af58-0d20184ab87a\") " pod="opendatahub/opendatahub-operator-controller-manager-978ccd68d-2l9f7"
Sep 18 13:49:24.274669 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:24.274402    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/278aa551-51a0-4564-af58-0d20184ab87a-webhook-cert\") pod \"opendatahub-operator-controller-manager-978ccd68d-2l9f7\" (UID: \"278aa551-51a0-4564-af58-0d20184ab87a\") " pod="opendatahub/opendatahub-operator-controller-manager-978ccd68d-2l9f7"
Sep 18 13:49:24.274669 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:24.274429    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/278aa551-51a0-4564-af58-0d20184ab87a-tmp\") pod \"opendatahub-operator-controller-manager-978ccd68d-2l9f7\" (UID: \"278aa551-51a0-4564-af58-0d20184ab87a\") " pod="opendatahub/opendatahub-operator-controller-manager-978ccd68d-2l9f7"
Sep 18 13:49:24.274921 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:24.274890    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/278aa551-51a0-4564-af58-0d20184ab87a-manifests\") pod \"opendatahub-operator-controller-manager-978ccd68d-2l9f7\" (UID: \"278aa551-51a0-4564-af58-0d20184ab87a\") " pod="opendatahub/opendatahub-operator-controller-manager-978ccd68d-2l9f7"
Sep 18 13:49:24.276782 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:24.276748    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/278aa551-51a0-4564-af58-0d20184ab87a-tmp\") pod \"opendatahub-operator-controller-manager-978ccd68d-2l9f7\" (UID: \"278aa551-51a0-4564-af58-0d20184ab87a\") " pod="opendatahub/opendatahub-operator-controller-manager-978ccd68d-2l9f7"
Sep 18 13:49:24.276984 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:24.276966    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/278aa551-51a0-4564-af58-0d20184ab87a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-978ccd68d-2l9f7\" (UID: \"278aa551-51a0-4564-af58-0d20184ab87a\") " pod="opendatahub/opendatahub-operator-controller-manager-978ccd68d-2l9f7"
Sep 18 13:49:24.277051 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:24.276968    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/278aa551-51a0-4564-af58-0d20184ab87a-webhook-cert\") pod \"opendatahub-operator-controller-manager-978ccd68d-2l9f7\" (UID: \"278aa551-51a0-4564-af58-0d20184ab87a\") " pod="opendatahub/opendatahub-operator-controller-manager-978ccd68d-2l9f7"
Sep 18 13:49:24.283133 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:24.283113    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sdbl\" (UniqueName: \"kubernetes.io/projected/278aa551-51a0-4564-af58-0d20184ab87a-kube-api-access-6sdbl\") pod \"opendatahub-operator-controller-manager-978ccd68d-2l9f7\" (UID: \"278aa551-51a0-4564-af58-0d20184ab87a\") " pod="opendatahub/opendatahub-operator-controller-manager-978ccd68d-2l9f7"
Sep 18 13:49:24.295843 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:24.295803    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-978ccd68d-2l9f7"
Sep 18 13:49:24.331606 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:24.331566    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:49:24.431713 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:24.431678    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:49:24.480008 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:24.479975    2577 generic.go:356] "Generic (PLEG): container finished" podID="75bc2527-be76-41c7-9df5-c9fcea6bd5f6" containerID="513aa6ad7c8f94b15262fcd10cc3adf34c0a279509091525c48626bcb8efd79d" exitCode=0
Sep 18 13:49:24.480365 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:24.480027    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tsfm6" event={"ID":"75bc2527-be76-41c7-9df5-c9fcea6bd5f6","Type":"ContainerDied","Data":"513aa6ad7c8f94b15262fcd10cc3adf34c0a279509091525c48626bcb8efd79d"}
Sep 18 13:49:24.508276 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:24.508238    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:49:24.574126 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:24.574099    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-978ccd68d-2l9f7"]
Sep 18 13:49:24.576791 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:49:24.576760    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod278aa551_51a0_4564_af58_0d20184ab87a.slice/crio-042dab7b69b21f3d70f9eaa9e165a34ecf067d03ea883a795df9001b277c45c7 WatchSource:0}: Error finding container 042dab7b69b21f3d70f9eaa9e165a34ecf067d03ea883a795df9001b277c45c7: Status 404 returned error can't find the container with id 042dab7b69b21f3d70f9eaa9e165a34ecf067d03ea883a795df9001b277c45c7
Sep 18 13:49:24.584600 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:24.584563    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:49:24.685532 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:24.685491    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:49:24.806815 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:24.806730    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:49:25.487291 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:25.487253    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-978ccd68d-2l9f7" event={"ID":"278aa551-51a0-4564-af58-0d20184ab87a","Type":"ContainerStarted","Data":"042dab7b69b21f3d70f9eaa9e165a34ecf067d03ea883a795df9001b277c45c7"}
Sep 18 13:49:25.490145 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:25.490120    2577 generic.go:356] "Generic (PLEG): container finished" podID="75bc2527-be76-41c7-9df5-c9fcea6bd5f6" containerID="4551fc9d813bfae43f56e6ae23d056ecc1a9e3d65e578348958d4f96aecc85d7" exitCode=0
Sep 18 13:49:25.490279 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:25.490163    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tsfm6" event={"ID":"75bc2527-be76-41c7-9df5-c9fcea6bd5f6","Type":"ContainerDied","Data":"4551fc9d813bfae43f56e6ae23d056ecc1a9e3d65e578348958d4f96aecc85d7"}
Sep 18 13:49:27.542341 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:27.542313    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tsfm6"
Sep 18 13:49:27.603250 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:27.603228    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/75bc2527-be76-41c7-9df5-c9fcea6bd5f6-bundle\") pod \"75bc2527-be76-41c7-9df5-c9fcea6bd5f6\" (UID: \"75bc2527-be76-41c7-9df5-c9fcea6bd5f6\") "
Sep 18 13:49:27.603346 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:27.603271    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/75bc2527-be76-41c7-9df5-c9fcea6bd5f6-util\") pod \"75bc2527-be76-41c7-9df5-c9fcea6bd5f6\" (UID: \"75bc2527-be76-41c7-9df5-c9fcea6bd5f6\") "
Sep 18 13:49:27.603346 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:27.603336    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lw4sl\" (UniqueName: \"kubernetes.io/projected/75bc2527-be76-41c7-9df5-c9fcea6bd5f6-kube-api-access-lw4sl\") pod \"75bc2527-be76-41c7-9df5-c9fcea6bd5f6\" (UID: \"75bc2527-be76-41c7-9df5-c9fcea6bd5f6\") "
Sep 18 13:49:27.604108 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:27.604070    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75bc2527-be76-41c7-9df5-c9fcea6bd5f6-bundle" (OuterVolumeSpecName: "bundle") pod "75bc2527-be76-41c7-9df5-c9fcea6bd5f6" (UID: "75bc2527-be76-41c7-9df5-c9fcea6bd5f6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:49:27.605297 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:27.605275    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75bc2527-be76-41c7-9df5-c9fcea6bd5f6-kube-api-access-lw4sl" (OuterVolumeSpecName: "kube-api-access-lw4sl") pod "75bc2527-be76-41c7-9df5-c9fcea6bd5f6" (UID: "75bc2527-be76-41c7-9df5-c9fcea6bd5f6"). InnerVolumeSpecName "kube-api-access-lw4sl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:49:27.608792 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:27.608772    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75bc2527-be76-41c7-9df5-c9fcea6bd5f6-util" (OuterVolumeSpecName: "util") pod "75bc2527-be76-41c7-9df5-c9fcea6bd5f6" (UID: "75bc2527-be76-41c7-9df5-c9fcea6bd5f6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:49:27.704236 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:27.704201    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lw4sl\" (UniqueName: \"kubernetes.io/projected/75bc2527-be76-41c7-9df5-c9fcea6bd5f6-kube-api-access-lw4sl\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:49:27.704236 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:27.704230    2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/75bc2527-be76-41c7-9df5-c9fcea6bd5f6-bundle\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:49:27.704236 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:27.704240    2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/75bc2527-be76-41c7-9df5-c9fcea6bd5f6-util\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:49:28.382792 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:28.382763    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:49:28.490637 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:28.490602    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:49:28.507398 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:28.507361    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-978ccd68d-2l9f7" event={"ID":"278aa551-51a0-4564-af58-0d20184ab87a","Type":"ContainerStarted","Data":"773a73dff2fb9c732d2aefae2a072ffd5652b89d59a2c807c567b01817bb33d7"}
Sep 18 13:49:28.509281 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:28.509258    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tsfm6" event={"ID":"75bc2527-be76-41c7-9df5-c9fcea6bd5f6","Type":"ContainerDied","Data":"759aeefc6e4aa2f8c1bd75e9eb1238857b1a99a72c39bd15bc8cd32882f5ac42"}
Sep 18 13:49:28.509397 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:28.509296    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="759aeefc6e4aa2f8c1bd75e9eb1238857b1a99a72c39bd15bc8cd32882f5ac42"
Sep 18 13:49:28.509397 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:28.509299    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tsfm6"
Sep 18 13:49:29.515157 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:29.515121    2577 generic.go:356] "Generic (PLEG): container finished" podID="278aa551-51a0-4564-af58-0d20184ab87a" containerID="773a73dff2fb9c732d2aefae2a072ffd5652b89d59a2c807c567b01817bb33d7" exitCode=0
Sep 18 13:49:29.515710 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:29.515207    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-978ccd68d-2l9f7" event={"ID":"278aa551-51a0-4564-af58-0d20184ab87a","Type":"ContainerDied","Data":"773a73dff2fb9c732d2aefae2a072ffd5652b89d59a2c807c567b01817bb33d7"}
Sep 18 13:49:30.521891 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:30.521849    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-978ccd68d-2l9f7" event={"ID":"278aa551-51a0-4564-af58-0d20184ab87a","Type":"ContainerStarted","Data":"26b697b0204356dbb9ed230ae3164ff3b7f203b69babc2795971400dc8785395"}
Sep 18 13:49:30.522341 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:30.521969    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-978ccd68d-2l9f7"
Sep 18 13:49:30.544765 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:30.544717    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-978ccd68d-2l9f7" podStartSLOduration=3.182848638 podStartE2EDuration="7.544700237s" totalImagesPullingTime="3.31685637s" totalInitContainerRuntime="1.044995229s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:49:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:49:30.543806636 +0000 UTC m=+319.596465963" watchObservedRunningTime="2026-09-18 13:49:30.544700237 +0000 UTC m=+319.597359560"
Sep 18 13:49:40.699078 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:40.699046    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:49:40.774747 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:40.774713    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:49:41.529415 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:41.529371    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-978ccd68d-2l9f7"
Sep 18 13:49:43.219272 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:43.219240    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:49:43.325628 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:43.325593    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:49:43.444770 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:43.444738    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-d5c57f774-6wnxx"]
Sep 18 13:49:43.445050 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:43.445038    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75bc2527-be76-41c7-9df5-c9fcea6bd5f6" containerName="pull"
Sep 18 13:49:43.445097 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:43.445051    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="75bc2527-be76-41c7-9df5-c9fcea6bd5f6" containerName="pull"
Sep 18 13:49:43.445097 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:43.445064    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75bc2527-be76-41c7-9df5-c9fcea6bd5f6" containerName="util"
Sep 18 13:49:43.445097 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:43.445069    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="75bc2527-be76-41c7-9df5-c9fcea6bd5f6" containerName="util"
Sep 18 13:49:43.445097 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:43.445082    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75bc2527-be76-41c7-9df5-c9fcea6bd5f6" containerName="extract"
Sep 18 13:49:43.445097 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:43.445086    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="75bc2527-be76-41c7-9df5-c9fcea6bd5f6" containerName="extract"
Sep 18 13:49:43.445241 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:43.445137    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="75bc2527-be76-41c7-9df5-c9fcea6bd5f6" containerName="extract"
Sep 18 13:49:43.447334 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:43.447310    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:49:43.453544 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:43.453528    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-d5c57f774-6wnxx"
Sep 18 13:49:43.455518 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:43.455497    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 18 13:49:43.455641 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:43.455626    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 18 13:49:43.455728 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:43.455709    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 18 13:49:43.456010 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:43.455993    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 18 13:49:43.460417 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:43.460396    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-d5c57f774-6wnxx"]
Sep 18 13:49:43.519241 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:43.519157    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e49a69a-0b86-4a68-b0fa-09d7c271e040-tls-certs\") pod \"kube-auth-proxy-d5c57f774-6wnxx\" (UID: \"5e49a69a-0b86-4a68-b0fa-09d7c271e040\") " pod="openshift-ingress/kube-auth-proxy-d5c57f774-6wnxx"
Sep 18 13:49:43.519241 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:43.519195    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e49a69a-0b86-4a68-b0fa-09d7c271e040-tmp\") pod \"kube-auth-proxy-d5c57f774-6wnxx\" (UID: \"5e49a69a-0b86-4a68-b0fa-09d7c271e040\") " pod="openshift-ingress/kube-auth-proxy-d5c57f774-6wnxx"
Sep 18 13:49:43.519400 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:43.519246    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2zqk\" (UniqueName: \"kubernetes.io/projected/5e49a69a-0b86-4a68-b0fa-09d7c271e040-kube-api-access-v2zqk\") pod \"kube-auth-proxy-d5c57f774-6wnxx\" (UID: \"5e49a69a-0b86-4a68-b0fa-09d7c271e040\") " pod="openshift-ingress/kube-auth-proxy-d5c57f774-6wnxx"
Sep 18 13:49:43.620576 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:43.620541    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e49a69a-0b86-4a68-b0fa-09d7c271e040-tls-certs\") pod \"kube-auth-proxy-d5c57f774-6wnxx\" (UID: \"5e49a69a-0b86-4a68-b0fa-09d7c271e040\") " pod="openshift-ingress/kube-auth-proxy-d5c57f774-6wnxx"
Sep 18 13:49:43.620576 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:43.620579    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e49a69a-0b86-4a68-b0fa-09d7c271e040-tmp\") pod \"kube-auth-proxy-d5c57f774-6wnxx\" (UID: \"5e49a69a-0b86-4a68-b0fa-09d7c271e040\") " pod="openshift-ingress/kube-auth-proxy-d5c57f774-6wnxx"
Sep 18 13:49:43.620788 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:43.620618    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2zqk\" (UniqueName: \"kubernetes.io/projected/5e49a69a-0b86-4a68-b0fa-09d7c271e040-kube-api-access-v2zqk\") pod \"kube-auth-proxy-d5c57f774-6wnxx\" (UID: \"5e49a69a-0b86-4a68-b0fa-09d7c271e040\") " pod="openshift-ingress/kube-auth-proxy-d5c57f774-6wnxx"
Sep 18 13:49:43.620788 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:49:43.620681    2577 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 18 13:49:43.620788 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:49:43.620772    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e49a69a-0b86-4a68-b0fa-09d7c271e040-tls-certs podName:5e49a69a-0b86-4a68-b0fa-09d7c271e040 nodeName:}" failed. No retries permitted until 2026-09-18 13:49:44.120755398 +0000 UTC m=+333.173414708 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/5e49a69a-0b86-4a68-b0fa-09d7c271e040-tls-certs") pod "kube-auth-proxy-d5c57f774-6wnxx" (UID: "5e49a69a-0b86-4a68-b0fa-09d7c271e040") : secret "kube-auth-proxy-tls" not found
Sep 18 13:49:43.622906 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:43.622886    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e49a69a-0b86-4a68-b0fa-09d7c271e040-tmp\") pod \"kube-auth-proxy-d5c57f774-6wnxx\" (UID: \"5e49a69a-0b86-4a68-b0fa-09d7c271e040\") " pod="openshift-ingress/kube-auth-proxy-d5c57f774-6wnxx"
Sep 18 13:49:43.632978 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:43.632951    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2zqk\" (UniqueName: \"kubernetes.io/projected/5e49a69a-0b86-4a68-b0fa-09d7c271e040-kube-api-access-v2zqk\") pod \"kube-auth-proxy-d5c57f774-6wnxx\" (UID: \"5e49a69a-0b86-4a68-b0fa-09d7c271e040\") " pod="openshift-ingress/kube-auth-proxy-d5c57f774-6wnxx"
Sep 18 13:49:43.714061 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:43.714027    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:49:44.124785 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:44.124749    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e49a69a-0b86-4a68-b0fa-09d7c271e040-tls-certs\") pod \"kube-auth-proxy-d5c57f774-6wnxx\" (UID: \"5e49a69a-0b86-4a68-b0fa-09d7c271e040\") " pod="openshift-ingress/kube-auth-proxy-d5c57f774-6wnxx"
Sep 18 13:49:44.127117 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:44.127094    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e49a69a-0b86-4a68-b0fa-09d7c271e040-tls-certs\") pod \"kube-auth-proxy-d5c57f774-6wnxx\" (UID: \"5e49a69a-0b86-4a68-b0fa-09d7c271e040\") " pod="openshift-ingress/kube-auth-proxy-d5c57f774-6wnxx"
Sep 18 13:49:44.364175 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:44.364135    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-d5c57f774-6wnxx"
Sep 18 13:49:44.661704 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:44.661621    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-d5c57f774-6wnxx"]
Sep 18 13:49:44.664865 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:49:44.664839    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e49a69a_0b86_4a68_b0fa_09d7c271e040.slice/crio-d8beaf692a7ea755ce13e14a8a4d19ec71fa7afd9a822326cd524512d9defabb WatchSource:0}: Error finding container d8beaf692a7ea755ce13e14a8a4d19ec71fa7afd9a822326cd524512d9defabb: Status 404 returned error can't find the container with id d8beaf692a7ea755ce13e14a8a4d19ec71fa7afd9a822326cd524512d9defabb
Sep 18 13:49:44.880690 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:44.880661    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:49:44.888369 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:44.888345    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txwct"]
Sep 18 13:49:44.892972 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:44.892955    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txwct"
Sep 18 13:49:44.895091 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:44.895069    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 18 13:49:44.895091 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:44.895068    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 18 13:49:44.895262 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:44.895098    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-tls6z\""
Sep 18 13:49:44.895262 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:44.895103    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 18 13:49:44.895262 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:44.895071    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 18 13:49:44.903546 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:44.903525    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txwct"]
Sep 18 13:49:44.932967 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:44.932895    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cac37c59-c25b-477b-b4e6-70eaf279e409-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-txwct\" (UID: \"cac37c59-c25b-477b-b4e6-70eaf279e409\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txwct"
Sep 18 13:49:44.933084 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:44.932967    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cac37c59-c25b-477b-b4e6-70eaf279e409-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-txwct\" (UID: \"cac37c59-c25b-477b-b4e6-70eaf279e409\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txwct"
Sep 18 13:49:44.933084 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:44.933025    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cac37c59-c25b-477b-b4e6-70eaf279e409-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-txwct\" (UID: \"cac37c59-c25b-477b-b4e6-70eaf279e409\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txwct"
Sep 18 13:49:44.933084 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:44.933055    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cac37c59-c25b-477b-b4e6-70eaf279e409-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-txwct\" (UID: \"cac37c59-c25b-477b-b4e6-70eaf279e409\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txwct"
Sep 18 13:49:44.933211 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:44.933084    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x56vz\" (UniqueName: \"kubernetes.io/projected/cac37c59-c25b-477b-b4e6-70eaf279e409-kube-api-access-x56vz\") pod \"istiod-openshift-gateway-7b7995d5cd-txwct\" (UID: \"cac37c59-c25b-477b-b4e6-70eaf279e409\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txwct"
Sep 18 13:49:44.933211 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:44.933127    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cac37c59-c25b-477b-b4e6-70eaf279e409-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-txwct\" (UID: \"cac37c59-c25b-477b-b4e6-70eaf279e409\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txwct"
Sep 18 13:49:44.933211 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:44.933159    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cac37c59-c25b-477b-b4e6-70eaf279e409-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-txwct\" (UID: \"cac37c59-c25b-477b-b4e6-70eaf279e409\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txwct"
Sep 18 13:49:44.962605 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:44.962577    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:49:45.034172 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:45.034141    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cac37c59-c25b-477b-b4e6-70eaf279e409-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-txwct\" (UID: \"cac37c59-c25b-477b-b4e6-70eaf279e409\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txwct"
Sep 18 13:49:45.034358 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:45.034182    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cac37c59-c25b-477b-b4e6-70eaf279e409-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-txwct\" (UID: \"cac37c59-c25b-477b-b4e6-70eaf279e409\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txwct"
Sep 18 13:49:45.034358 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:45.034312    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x56vz\" (UniqueName: \"kubernetes.io/projected/cac37c59-c25b-477b-b4e6-70eaf279e409-kube-api-access-x56vz\") pod \"istiod-openshift-gateway-7b7995d5cd-txwct\" (UID: \"cac37c59-c25b-477b-b4e6-70eaf279e409\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txwct"
Sep 18 13:49:45.034483 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:45.034358    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cac37c59-c25b-477b-b4e6-70eaf279e409-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-txwct\" (UID: \"cac37c59-c25b-477b-b4e6-70eaf279e409\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txwct"
Sep 18 13:49:45.034483 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:45.034385    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cac37c59-c25b-477b-b4e6-70eaf279e409-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-txwct\" (UID: \"cac37c59-c25b-477b-b4e6-70eaf279e409\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txwct"
Sep 18 13:49:45.034483 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:45.034429    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cac37c59-c25b-477b-b4e6-70eaf279e409-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-txwct\" (UID: \"cac37c59-c25b-477b-b4e6-70eaf279e409\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txwct"
Sep 18 13:49:45.034635 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:45.034531    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cac37c59-c25b-477b-b4e6-70eaf279e409-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-txwct\" (UID: \"cac37c59-c25b-477b-b4e6-70eaf279e409\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txwct"
Sep 18 13:49:45.035074 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:45.035042    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cac37c59-c25b-477b-b4e6-70eaf279e409-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-txwct\" (UID: \"cac37c59-c25b-477b-b4e6-70eaf279e409\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txwct"
Sep 18 13:49:45.036783 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:45.036760    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cac37c59-c25b-477b-b4e6-70eaf279e409-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-txwct\" (UID: \"cac37c59-c25b-477b-b4e6-70eaf279e409\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txwct"
Sep 18 13:49:45.037013 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:45.036983    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cac37c59-c25b-477b-b4e6-70eaf279e409-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-txwct\" (UID: \"cac37c59-c25b-477b-b4e6-70eaf279e409\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txwct"
Sep 18 13:49:45.037013 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:45.037004    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cac37c59-c25b-477b-b4e6-70eaf279e409-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-txwct\" (UID: \"cac37c59-c25b-477b-b4e6-70eaf279e409\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txwct"
Sep 18 13:49:45.037173 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:45.037157    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cac37c59-c25b-477b-b4e6-70eaf279e409-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-txwct\" (UID: \"cac37c59-c25b-477b-b4e6-70eaf279e409\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txwct"
Sep 18 13:49:45.045203 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:45.045175    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cac37c59-c25b-477b-b4e6-70eaf279e409-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-txwct\" (UID: \"cac37c59-c25b-477b-b4e6-70eaf279e409\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txwct"
Sep 18 13:49:45.045301 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:45.045264    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x56vz\" (UniqueName: \"kubernetes.io/projected/cac37c59-c25b-477b-b4e6-70eaf279e409-kube-api-access-x56vz\") pod \"istiod-openshift-gateway-7b7995d5cd-txwct\" (UID: \"cac37c59-c25b-477b-b4e6-70eaf279e409\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txwct"
Sep 18 13:49:45.202443 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:45.202350    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txwct"
Sep 18 13:49:45.495280 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:45.494999    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txwct"]
Sep 18 13:49:45.498301 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:49:45.498268    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcac37c59_c25b_477b_b4e6_70eaf279e409.slice/crio-bce848c1a4840849a52eafc25e876e67a915ef3c8e95b58c4a7a240ec630269e WatchSource:0}: Error finding container bce848c1a4840849a52eafc25e876e67a915ef3c8e95b58c4a7a240ec630269e: Status 404 returned error can't find the container with id bce848c1a4840849a52eafc25e876e67a915ef3c8e95b58c4a7a240ec630269e
Sep 18 13:49:45.594135 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:45.594090    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txwct" event={"ID":"cac37c59-c25b-477b-b4e6-70eaf279e409","Type":"ContainerStarted","Data":"bce848c1a4840849a52eafc25e876e67a915ef3c8e95b58c4a7a240ec630269e"}
Sep 18 13:49:45.595592 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:45.595562    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-d5c57f774-6wnxx" event={"ID":"5e49a69a-0b86-4a68-b0fa-09d7c271e040","Type":"ContainerStarted","Data":"d8beaf692a7ea755ce13e14a8a4d19ec71fa7afd9a822326cd524512d9defabb"}
Sep 18 13:49:45.873446 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:45.873382    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:49:47.609039 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:47.608998    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-d5c57f774-6wnxx" event={"ID":"5e49a69a-0b86-4a68-b0fa-09d7c271e040","Type":"ContainerStarted","Data":"77ccc55ef815552a056f585c2ea0cf7d90b694ae9ef0ec3cc384359fad145799"}
Sep 18 13:49:47.625762 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:47.625712    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-d5c57f774-6wnxx" podStartSLOduration=2.074598935 podStartE2EDuration="4.625692616s" totalImagesPullingTime="2.551093681s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:49:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:49:47.625568179 +0000 UTC m=+336.678227506" watchObservedRunningTime="2026-09-18 13:49:47.625692616 +0000 UTC m=+336.678351960"
Sep 18 13:49:48.949270 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:48.949221    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 13:49:48.949539 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:48.949305    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 13:49:49.621082 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:49.621046    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txwct" event={"ID":"cac37c59-c25b-477b-b4e6-70eaf279e409","Type":"ContainerStarted","Data":"dbb99437b35dbe6dcf955ae61659be8013d5234bae9593ee8bab49d49c85dc4b"}
Sep 18 13:49:49.621290 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:49.621158    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txwct"
Sep 18 13:49:49.652489 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:49.652439    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txwct" podStartSLOduration=2.204700009 podStartE2EDuration="5.652422194s" totalImagesPullingTime="3.447722185s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:49:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:49:49.651582852 +0000 UTC m=+338.704242180" watchObservedRunningTime="2026-09-18 13:49:49.652422194 +0000 UTC m=+338.705081521"
Sep 18 13:49:50.231411 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:50.231372    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:49:50.326589 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:50.326557    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:49:50.628083 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:50.628051    2577 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-txwct container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 18 13:49:50.628239 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:50.628102    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txwct" podUID="cac37c59-c25b-477b-b4e6-70eaf279e409" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 13:49:51.095932 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:51.095892    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:49:51.285186 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:51.285156    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:49:51.387066 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:51.386984    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:49:53.627358 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:53.627320    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txwct"
Sep 18 13:49:54.412501 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:54.412462    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:49:54.523786 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:54.523750    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:49:54.650553 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:54.650524    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:49:55.025413 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:55.025380    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:49:55.127732 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:55.127657    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:49:55.307424 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:49:55.307341    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:50:01.911988 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:01.911953    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6cdc5b77c9-hm24r"]
Sep 18 13:50:01.917952 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:01.917930    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6cdc5b77c9-hm24r"
Sep 18 13:50:01.920391 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:01.920366    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 18 13:50:01.920738 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:01.920720    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 18 13:50:01.920886 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:01.920720    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 18 13:50:01.921233 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:01.921215    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-q6vxx\""
Sep 18 13:50:01.929321 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:01.929294    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6cdc5b77c9-hm24r"]
Sep 18 13:50:01.981507 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:01.981474    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8a2f0da4-3361-4b70-985b-c7ecadb8c4a8-manager-config\") pod \"lws-controller-manager-6cdc5b77c9-hm24r\" (UID: \"8a2f0da4-3361-4b70-985b-c7ecadb8c4a8\") " pod="openshift-lws-operator/lws-controller-manager-6cdc5b77c9-hm24r"
Sep 18 13:50:01.981711 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:01.981542    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8a2f0da4-3361-4b70-985b-c7ecadb8c4a8-cert\") pod \"lws-controller-manager-6cdc5b77c9-hm24r\" (UID: \"8a2f0da4-3361-4b70-985b-c7ecadb8c4a8\") " pod="openshift-lws-operator/lws-controller-manager-6cdc5b77c9-hm24r"
Sep 18 13:50:01.981711 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:01.981606    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8a2f0da4-3361-4b70-985b-c7ecadb8c4a8-metrics-cert\") pod \"lws-controller-manager-6cdc5b77c9-hm24r\" (UID: \"8a2f0da4-3361-4b70-985b-c7ecadb8c4a8\") " pod="openshift-lws-operator/lws-controller-manager-6cdc5b77c9-hm24r"
Sep 18 13:50:01.981711 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:01.981649    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ppb6\" (UniqueName: \"kubernetes.io/projected/8a2f0da4-3361-4b70-985b-c7ecadb8c4a8-kube-api-access-4ppb6\") pod \"lws-controller-manager-6cdc5b77c9-hm24r\" (UID: \"8a2f0da4-3361-4b70-985b-c7ecadb8c4a8\") " pod="openshift-lws-operator/lws-controller-manager-6cdc5b77c9-hm24r"
Sep 18 13:50:02.082868 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:02.082798    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8a2f0da4-3361-4b70-985b-c7ecadb8c4a8-cert\") pod \"lws-controller-manager-6cdc5b77c9-hm24r\" (UID: \"8a2f0da4-3361-4b70-985b-c7ecadb8c4a8\") " pod="openshift-lws-operator/lws-controller-manager-6cdc5b77c9-hm24r"
Sep 18 13:50:02.082868 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:02.082876    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8a2f0da4-3361-4b70-985b-c7ecadb8c4a8-metrics-cert\") pod \"lws-controller-manager-6cdc5b77c9-hm24r\" (UID: \"8a2f0da4-3361-4b70-985b-c7ecadb8c4a8\") " pod="openshift-lws-operator/lws-controller-manager-6cdc5b77c9-hm24r"
Sep 18 13:50:02.083135 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:02.082902    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ppb6\" (UniqueName: \"kubernetes.io/projected/8a2f0da4-3361-4b70-985b-c7ecadb8c4a8-kube-api-access-4ppb6\") pod \"lws-controller-manager-6cdc5b77c9-hm24r\" (UID: \"8a2f0da4-3361-4b70-985b-c7ecadb8c4a8\") " pod="openshift-lws-operator/lws-controller-manager-6cdc5b77c9-hm24r"
Sep 18 13:50:02.083135 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:02.082950    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8a2f0da4-3361-4b70-985b-c7ecadb8c4a8-manager-config\") pod \"lws-controller-manager-6cdc5b77c9-hm24r\" (UID: \"8a2f0da4-3361-4b70-985b-c7ecadb8c4a8\") " pod="openshift-lws-operator/lws-controller-manager-6cdc5b77c9-hm24r"
Sep 18 13:50:02.083652 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:02.083632    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8a2f0da4-3361-4b70-985b-c7ecadb8c4a8-manager-config\") pod \"lws-controller-manager-6cdc5b77c9-hm24r\" (UID: \"8a2f0da4-3361-4b70-985b-c7ecadb8c4a8\") " pod="openshift-lws-operator/lws-controller-manager-6cdc5b77c9-hm24r"
Sep 18 13:50:02.085378 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:02.085351    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8a2f0da4-3361-4b70-985b-c7ecadb8c4a8-metrics-cert\") pod \"lws-controller-manager-6cdc5b77c9-hm24r\" (UID: \"8a2f0da4-3361-4b70-985b-c7ecadb8c4a8\") " pod="openshift-lws-operator/lws-controller-manager-6cdc5b77c9-hm24r"
Sep 18 13:50:02.085471 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:02.085411    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8a2f0da4-3361-4b70-985b-c7ecadb8c4a8-cert\") pod \"lws-controller-manager-6cdc5b77c9-hm24r\" (UID: \"8a2f0da4-3361-4b70-985b-c7ecadb8c4a8\") " pod="openshift-lws-operator/lws-controller-manager-6cdc5b77c9-hm24r"
Sep 18 13:50:02.097245 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:02.097224    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ppb6\" (UniqueName: \"kubernetes.io/projected/8a2f0da4-3361-4b70-985b-c7ecadb8c4a8-kube-api-access-4ppb6\") pod \"lws-controller-manager-6cdc5b77c9-hm24r\" (UID: \"8a2f0da4-3361-4b70-985b-c7ecadb8c4a8\") " pod="openshift-lws-operator/lws-controller-manager-6cdc5b77c9-hm24r"
Sep 18 13:50:02.227343 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:02.227260    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6cdc5b77c9-hm24r"
Sep 18 13:50:02.521027 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:02.520948    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6cdc5b77c9-hm24r"]
Sep 18 13:50:02.524569 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:50:02.524546    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a2f0da4_3361_4b70_985b_c7ecadb8c4a8.slice/crio-a86e4686ecb15424787eb99592546ad9ef45b21ba3556b6076b7234f300b9041 WatchSource:0}: Error finding container a86e4686ecb15424787eb99592546ad9ef45b21ba3556b6076b7234f300b9041: Status 404 returned error can't find the container with id a86e4686ecb15424787eb99592546ad9ef45b21ba3556b6076b7234f300b9041
Sep 18 13:50:02.684638 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:02.684602    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6cdc5b77c9-hm24r" event={"ID":"8a2f0da4-3361-4b70-985b-c7ecadb8c4a8","Type":"ContainerStarted","Data":"a86e4686ecb15424787eb99592546ad9ef45b21ba3556b6076b7234f300b9041"}
Sep 18 13:50:04.630132 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:04.630092    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:50:04.696663 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:04.696630    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6cdc5b77c9-hm24r" event={"ID":"8a2f0da4-3361-4b70-985b-c7ecadb8c4a8","Type":"ContainerStarted","Data":"1c548a14ff3fd6aabafb502575edb26969b7925b36b2a7eb023e5bf00910223e"}
Sep 18 13:50:04.696856 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:04.696741    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6cdc5b77c9-hm24r"
Sep 18 13:50:04.731312 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:04.731256    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6cdc5b77c9-hm24r" podStartSLOduration=2.470709101 podStartE2EDuration="3.731237452s" totalImagesPullingTime="1.260528351s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:50:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:50:04.72925101 +0000 UTC m=+353.781910337" watchObservedRunningTime="2026-09-18 13:50:04.731237452 +0000 UTC m=+353.783896778"
Sep 18 13:50:04.764315 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:04.764290    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:50:04.850133 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:04.850096    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:50:04.962679 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:04.962596    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:50:05.125597 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:05.125567    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:50:05.550739 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:05.550705    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:50:06.318718 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:06.318664    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:50:06.433755 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:06.433725    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:50:06.970372 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:06.970331    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txwct"]
Sep 18 13:50:06.970591 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:06.970566    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txwct" podUID="cac37c59-c25b-477b-b4e6-70eaf279e409" containerName="discovery" containerID="cri-o://dbb99437b35dbe6dcf955ae61659be8013d5234bae9593ee8bab49d49c85dc4b" gracePeriod=30
Sep 18 13:50:07.213815 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:07.213791    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txwct"
Sep 18 13:50:07.325978 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:07.325893    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cac37c59-c25b-477b-b4e6-70eaf279e409-istio-kubeconfig\") pod \"cac37c59-c25b-477b-b4e6-70eaf279e409\" (UID: \"cac37c59-c25b-477b-b4e6-70eaf279e409\") "
Sep 18 13:50:07.325978 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:07.325932    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cac37c59-c25b-477b-b4e6-70eaf279e409-istio-csr-dns-cert\") pod \"cac37c59-c25b-477b-b4e6-70eaf279e409\" (UID: \"cac37c59-c25b-477b-b4e6-70eaf279e409\") "
Sep 18 13:50:07.325978 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:07.325972    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x56vz\" (UniqueName: \"kubernetes.io/projected/cac37c59-c25b-477b-b4e6-70eaf279e409-kube-api-access-x56vz\") pod \"cac37c59-c25b-477b-b4e6-70eaf279e409\" (UID: \"cac37c59-c25b-477b-b4e6-70eaf279e409\") "
Sep 18 13:50:07.326567 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:07.325992    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cac37c59-c25b-477b-b4e6-70eaf279e409-local-certs\") pod \"cac37c59-c25b-477b-b4e6-70eaf279e409\" (UID: \"cac37c59-c25b-477b-b4e6-70eaf279e409\") "
Sep 18 13:50:07.326567 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:07.326013    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cac37c59-c25b-477b-b4e6-70eaf279e409-istio-csr-ca-configmap\") pod \"cac37c59-c25b-477b-b4e6-70eaf279e409\" (UID: \"cac37c59-c25b-477b-b4e6-70eaf279e409\") "
Sep 18 13:50:07.326567 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:07.326036    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cac37c59-c25b-477b-b4e6-70eaf279e409-cacerts\") pod \"cac37c59-c25b-477b-b4e6-70eaf279e409\" (UID: \"cac37c59-c25b-477b-b4e6-70eaf279e409\") "
Sep 18 13:50:07.326567 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:07.326076    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cac37c59-c25b-477b-b4e6-70eaf279e409-istio-token\") pod \"cac37c59-c25b-477b-b4e6-70eaf279e409\" (UID: \"cac37c59-c25b-477b-b4e6-70eaf279e409\") "
Sep 18 13:50:07.326567 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:07.326482    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cac37c59-c25b-477b-b4e6-70eaf279e409-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "cac37c59-c25b-477b-b4e6-70eaf279e409" (UID: "cac37c59-c25b-477b-b4e6-70eaf279e409"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:50:07.328441 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:07.328410    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cac37c59-c25b-477b-b4e6-70eaf279e409-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "cac37c59-c25b-477b-b4e6-70eaf279e409" (UID: "cac37c59-c25b-477b-b4e6-70eaf279e409"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:50:07.328550 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:07.328465    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cac37c59-c25b-477b-b4e6-70eaf279e409-kube-api-access-x56vz" (OuterVolumeSpecName: "kube-api-access-x56vz") pod "cac37c59-c25b-477b-b4e6-70eaf279e409" (UID: "cac37c59-c25b-477b-b4e6-70eaf279e409"). InnerVolumeSpecName "kube-api-access-x56vz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:50:07.328550 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:07.328491    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cac37c59-c25b-477b-b4e6-70eaf279e409-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "cac37c59-c25b-477b-b4e6-70eaf279e409" (UID: "cac37c59-c25b-477b-b4e6-70eaf279e409"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:50:07.328628 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:07.328586    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cac37c59-c25b-477b-b4e6-70eaf279e409-cacerts" (OuterVolumeSpecName: "cacerts") pod "cac37c59-c25b-477b-b4e6-70eaf279e409" (UID: "cac37c59-c25b-477b-b4e6-70eaf279e409"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:50:07.328859 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:07.328839    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cac37c59-c25b-477b-b4e6-70eaf279e409-local-certs" (OuterVolumeSpecName: "local-certs") pod "cac37c59-c25b-477b-b4e6-70eaf279e409" (UID: "cac37c59-c25b-477b-b4e6-70eaf279e409"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:50:07.328916 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:07.328892    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cac37c59-c25b-477b-b4e6-70eaf279e409-istio-token" (OuterVolumeSpecName: "istio-token") pod "cac37c59-c25b-477b-b4e6-70eaf279e409" (UID: "cac37c59-c25b-477b-b4e6-70eaf279e409"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:50:07.429346 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:07.429309    2577 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cac37c59-c25b-477b-b4e6-70eaf279e409-istio-csr-ca-configmap\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:50:07.429346 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:07.429339    2577 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cac37c59-c25b-477b-b4e6-70eaf279e409-cacerts\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:50:07.429346 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:07.429350    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cac37c59-c25b-477b-b4e6-70eaf279e409-istio-token\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:50:07.429628 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:07.429359    2577 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cac37c59-c25b-477b-b4e6-70eaf279e409-istio-kubeconfig\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:50:07.429628 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:07.429372    2577 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cac37c59-c25b-477b-b4e6-70eaf279e409-istio-csr-dns-cert\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:50:07.429628 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:07.429383    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x56vz\" (UniqueName: \"kubernetes.io/projected/cac37c59-c25b-477b-b4e6-70eaf279e409-kube-api-access-x56vz\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:50:07.429628 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:07.429393    2577 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cac37c59-c25b-477b-b4e6-70eaf279e409-local-certs\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:50:07.712971 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:07.712932    2577 generic.go:356] "Generic (PLEG): container finished" podID="cac37c59-c25b-477b-b4e6-70eaf279e409" containerID="dbb99437b35dbe6dcf955ae61659be8013d5234bae9593ee8bab49d49c85dc4b" exitCode=0
Sep 18 13:50:07.712971 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:07.712976    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txwct"
Sep 18 13:50:07.713195 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:07.713015    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txwct" event={"ID":"cac37c59-c25b-477b-b4e6-70eaf279e409","Type":"ContainerDied","Data":"dbb99437b35dbe6dcf955ae61659be8013d5234bae9593ee8bab49d49c85dc4b"}
Sep 18 13:50:07.713195 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:07.713067    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txwct" event={"ID":"cac37c59-c25b-477b-b4e6-70eaf279e409","Type":"ContainerDied","Data":"bce848c1a4840849a52eafc25e876e67a915ef3c8e95b58c4a7a240ec630269e"}
Sep 18 13:50:07.713195 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:07.713090    2577 scope.go:117] "RemoveContainer" containerID="dbb99437b35dbe6dcf955ae61659be8013d5234bae9593ee8bab49d49c85dc4b"
Sep 18 13:50:07.722193 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:07.722178    2577 scope.go:117] "RemoveContainer" containerID="dbb99437b35dbe6dcf955ae61659be8013d5234bae9593ee8bab49d49c85dc4b"
Sep 18 13:50:07.722489 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:50:07.722469    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbb99437b35dbe6dcf955ae61659be8013d5234bae9593ee8bab49d49c85dc4b\": container with ID starting with dbb99437b35dbe6dcf955ae61659be8013d5234bae9593ee8bab49d49c85dc4b not found: ID does not exist" containerID="dbb99437b35dbe6dcf955ae61659be8013d5234bae9593ee8bab49d49c85dc4b"
Sep 18 13:50:07.722539 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:07.722498    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbb99437b35dbe6dcf955ae61659be8013d5234bae9593ee8bab49d49c85dc4b"} err="failed to get container status \"dbb99437b35dbe6dcf955ae61659be8013d5234bae9593ee8bab49d49c85dc4b\": rpc error: code = NotFound desc = could not find container \"dbb99437b35dbe6dcf955ae61659be8013d5234bae9593ee8bab49d49c85dc4b\": container with ID starting with dbb99437b35dbe6dcf955ae61659be8013d5234bae9593ee8bab49d49c85dc4b not found: ID does not exist"
Sep 18 13:50:07.740618 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:07.740592    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txwct"]
Sep 18 13:50:07.753618 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:07.753591    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-txwct"]
Sep 18 13:50:09.106856 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:09.106806    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cac37c59-c25b-477b-b4e6-70eaf279e409" path="/var/lib/kubelet/pods/cac37c59-c25b-477b-b4e6-70eaf279e409/volumes"
Sep 18 13:50:09.534793 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:09.534703    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:50:09.644149 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:09.644110    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:50:15.704693 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:15.704661    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6cdc5b77c9-hm24r"
Sep 18 13:50:43.902872 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:43.902839    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:50:43.967988 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:43.967958    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:50:54.382591 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:54.382548    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77b97c7f9b-q6nzl"]
Sep 18 13:50:54.592224 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:54.592192    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:50:54.698592 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:54.698505    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:50:55.671982 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:55.671947    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:50:55.821388 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:50:55.821355    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:51:00.120968 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:00.120938    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ablnsh"]
Sep 18 13:51:00.121330 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:00.121224    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cac37c59-c25b-477b-b4e6-70eaf279e409" containerName="discovery"
Sep 18 13:51:00.121330 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:00.121235    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cac37c59-c25b-477b-b4e6-70eaf279e409" containerName="discovery"
Sep 18 13:51:00.121330 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:00.121288    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="cac37c59-c25b-477b-b4e6-70eaf279e409" containerName="discovery"
Sep 18 13:51:00.124112 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:00.124095    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ablnsh"
Sep 18 13:51:00.126714 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:00.126695    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 18 13:51:00.126817 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:00.126696    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-l2rh5\""
Sep 18 13:51:00.127084 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:00.127069    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 18 13:51:00.133438 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:00.133417    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ablnsh"]
Sep 18 13:51:00.251415 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:00.251382    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6td5\" (UniqueName: \"kubernetes.io/projected/74f6f854-679e-4427-af63-622ffcc4d74b-kube-api-access-j6td5\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ablnsh\" (UID: \"74f6f854-679e-4427-af63-622ffcc4d74b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ablnsh"
Sep 18 13:51:00.251593 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:00.251433    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/74f6f854-679e-4427-af63-622ffcc4d74b-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ablnsh\" (UID: \"74f6f854-679e-4427-af63-622ffcc4d74b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ablnsh"
Sep 18 13:51:00.251593 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:00.251507    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/74f6f854-679e-4427-af63-622ffcc4d74b-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ablnsh\" (UID: \"74f6f854-679e-4427-af63-622ffcc4d74b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ablnsh"
Sep 18 13:51:00.280532 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:00.280503    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65l5sr"]
Sep 18 13:51:00.283707 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:00.283690    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65l5sr"
Sep 18 13:51:00.310420 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:00.310383    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65l5sr"]
Sep 18 13:51:00.352142 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:00.352111    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/74f6f854-679e-4427-af63-622ffcc4d74b-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ablnsh\" (UID: \"74f6f854-679e-4427-af63-622ffcc4d74b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ablnsh"
Sep 18 13:51:00.352313 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:00.352151    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfxkl\" (UniqueName: \"kubernetes.io/projected/6ed3c114-22de-45a3-abee-7e663d48f87d-kube-api-access-sfxkl\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65l5sr\" (UID: \"6ed3c114-22de-45a3-abee-7e663d48f87d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65l5sr"
Sep 18 13:51:00.352313 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:00.352177    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6ed3c114-22de-45a3-abee-7e663d48f87d-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65l5sr\" (UID: \"6ed3c114-22de-45a3-abee-7e663d48f87d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65l5sr"
Sep 18 13:51:00.352313 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:00.352214    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6td5\" (UniqueName: \"kubernetes.io/projected/74f6f854-679e-4427-af63-622ffcc4d74b-kube-api-access-j6td5\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ablnsh\" (UID: \"74f6f854-679e-4427-af63-622ffcc4d74b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ablnsh"
Sep 18 13:51:00.352486 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:00.352366    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6ed3c114-22de-45a3-abee-7e663d48f87d-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65l5sr\" (UID: \"6ed3c114-22de-45a3-abee-7e663d48f87d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65l5sr"
Sep 18 13:51:00.352486 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:00.352445    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/74f6f854-679e-4427-af63-622ffcc4d74b-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ablnsh\" (UID: \"74f6f854-679e-4427-af63-622ffcc4d74b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ablnsh"
Sep 18 13:51:00.352584 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:00.352559    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/74f6f854-679e-4427-af63-622ffcc4d74b-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ablnsh\" (UID: \"74f6f854-679e-4427-af63-622ffcc4d74b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ablnsh"
Sep 18 13:51:00.352762 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:00.352743    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/74f6f854-679e-4427-af63-622ffcc4d74b-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ablnsh\" (UID: \"74f6f854-679e-4427-af63-622ffcc4d74b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ablnsh"
Sep 18 13:51:00.362008 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:00.361986    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6td5\" (UniqueName: \"kubernetes.io/projected/74f6f854-679e-4427-af63-622ffcc4d74b-kube-api-access-j6td5\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ablnsh\" (UID: \"74f6f854-679e-4427-af63-622ffcc4d74b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ablnsh"
Sep 18 13:51:00.381216 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:00.381154    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lfrn"]
Sep 18 13:51:00.384514 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:00.384498    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lfrn"
Sep 18 13:51:00.397329 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:00.397305    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lfrn"]
Sep 18 13:51:00.434198 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:00.434174    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ablnsh"
Sep 18 13:51:00.452174 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:00.452146    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75mghv"]
Sep 18 13:51:00.453138 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:00.453118    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/497a77e0-e6f0-4d88-aadc-66b7c31adc82-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lfrn\" (UID: \"497a77e0-e6f0-4d88-aadc-66b7c31adc82\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lfrn"
Sep 18 13:51:00.453213 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:00.453151    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28gj4\" (UniqueName: \"kubernetes.io/projected/497a77e0-e6f0-4d88-aadc-66b7c31adc82-kube-api-access-28gj4\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lfrn\" (UID: \"497a77e0-e6f0-4d88-aadc-66b7c31adc82\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lfrn"
Sep 18 13:51:00.453213 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:00.453194    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfxkl\" (UniqueName: \"kubernetes.io/projected/6ed3c114-22de-45a3-abee-7e663d48f87d-kube-api-access-sfxkl\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65l5sr\" (UID: \"6ed3c114-22de-45a3-abee-7e663d48f87d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65l5sr"
Sep 18 13:51:00.453302 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:00.453262    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6ed3c114-22de-45a3-abee-7e663d48f87d-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65l5sr\" (UID: \"6ed3c114-22de-45a3-abee-7e663d48f87d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65l5sr"
Sep 18 13:51:00.453369 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:00.453351    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6ed3c114-22de-45a3-abee-7e663d48f87d-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65l5sr\" (UID: \"6ed3c114-22de-45a3-abee-7e663d48f87d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65l5sr"
Sep 18 13:51:00.453418 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:00.453405    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/497a77e0-e6f0-4d88-aadc-66b7c31adc82-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lfrn\" (UID: \"497a77e0-e6f0-4d88-aadc-66b7c31adc82\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lfrn"
Sep 18 13:51:00.453638 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:00.453617    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6ed3c114-22de-45a3-abee-7e663d48f87d-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65l5sr\" (UID: \"6ed3c114-22de-45a3-abee-7e663d48f87d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65l5sr"
Sep 18 13:51:00.453702 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:00.453661    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6ed3c114-22de-45a3-abee-7e663d48f87d-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65l5sr\" (UID: \"6ed3c114-22de-45a3-abee-7e663d48f87d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65l5sr"
Sep 18 13:51:00.456202 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:00.456185    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75mghv"
Sep 18 13:51:00.461664 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:00.461640    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfxkl\" (UniqueName: \"kubernetes.io/projected/6ed3c114-22de-45a3-abee-7e663d48f87d-kube-api-access-sfxkl\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65l5sr\" (UID: \"6ed3c114-22de-45a3-abee-7e663d48f87d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65l5sr"
Sep 18 13:51:00.465590 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:00.465568    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75mghv"]
Sep 18 13:51:00.554486 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:00.554450    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/617f1c94-bd5b-499e-9ba9-d079ab543794-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75mghv\" (UID: \"617f1c94-bd5b-499e-9ba9-d079ab543794\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75mghv"
Sep 18 13:51:00.554486 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:00.554485    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/617f1c94-bd5b-499e-9ba9-d079ab543794-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75mghv\" (UID: \"617f1c94-bd5b-499e-9ba9-d079ab543794\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75mghv"
Sep 18 13:51:00.554722 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:00.554509    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/497a77e0-e6f0-4d88-aadc-66b7c31adc82-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lfrn\" (UID: \"497a77e0-e6f0-4d88-aadc-66b7c31adc82\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lfrn"
Sep 18 13:51:00.554722 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:00.554663    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/497a77e0-e6f0-4d88-aadc-66b7c31adc82-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lfrn\" (UID: \"497a77e0-e6f0-4d88-aadc-66b7c31adc82\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lfrn"
Sep 18 13:51:00.554722 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:00.554683    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs2fz\" (UniqueName: \"kubernetes.io/projected/617f1c94-bd5b-499e-9ba9-d079ab543794-kube-api-access-zs2fz\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75mghv\" (UID: \"617f1c94-bd5b-499e-9ba9-d079ab543794\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75mghv"
Sep 18 13:51:00.554924 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:00.554710    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28gj4\" (UniqueName: \"kubernetes.io/projected/497a77e0-e6f0-4d88-aadc-66b7c31adc82-kube-api-access-28gj4\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lfrn\" (UID: \"497a77e0-e6f0-4d88-aadc-66b7c31adc82\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lfrn"
Sep 18 13:51:00.554998 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:00.554979    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/497a77e0-e6f0-4d88-aadc-66b7c31adc82-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lfrn\" (UID: \"497a77e0-e6f0-4d88-aadc-66b7c31adc82\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lfrn"
Sep 18 13:51:00.555043 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:00.554977    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/497a77e0-e6f0-4d88-aadc-66b7c31adc82-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lfrn\" (UID: \"497a77e0-e6f0-4d88-aadc-66b7c31adc82\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lfrn"
Sep 18 13:51:00.563492 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:00.563468    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28gj4\" (UniqueName: \"kubernetes.io/projected/497a77e0-e6f0-4d88-aadc-66b7c31adc82-kube-api-access-28gj4\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lfrn\" (UID: \"497a77e0-e6f0-4d88-aadc-66b7c31adc82\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lfrn"
Sep 18 13:51:00.592355 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:00.592325    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65l5sr"
Sep 18 13:51:00.655355 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:00.655325    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/617f1c94-bd5b-499e-9ba9-d079ab543794-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75mghv\" (UID: \"617f1c94-bd5b-499e-9ba9-d079ab543794\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75mghv"
Sep 18 13:51:00.655497 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:00.655359    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/617f1c94-bd5b-499e-9ba9-d079ab543794-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75mghv\" (UID: \"617f1c94-bd5b-499e-9ba9-d079ab543794\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75mghv"
Sep 18 13:51:00.655497 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:00.655477    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs2fz\" (UniqueName: \"kubernetes.io/projected/617f1c94-bd5b-499e-9ba9-d079ab543794-kube-api-access-zs2fz\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75mghv\" (UID: \"617f1c94-bd5b-499e-9ba9-d079ab543794\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75mghv"
Sep 18 13:51:00.655705 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:00.655688    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/617f1c94-bd5b-499e-9ba9-d079ab543794-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75mghv\" (UID: \"617f1c94-bd5b-499e-9ba9-d079ab543794\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75mghv"
Sep 18 13:51:00.655785 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:00.655767    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/617f1c94-bd5b-499e-9ba9-d079ab543794-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75mghv\" (UID: \"617f1c94-bd5b-499e-9ba9-d079ab543794\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75mghv"
Sep 18 13:51:00.664595 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:00.664566    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs2fz\" (UniqueName: \"kubernetes.io/projected/617f1c94-bd5b-499e-9ba9-d079ab543794-kube-api-access-zs2fz\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75mghv\" (UID: \"617f1c94-bd5b-499e-9ba9-d079ab543794\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75mghv"
Sep 18 13:51:00.693408 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:00.693376    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lfrn"
Sep 18 13:51:00.710281 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:00.710244    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ablnsh"]
Sep 18 13:51:00.712950 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:51:00.712926    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74f6f854_679e_4427_af63_622ffcc4d74b.slice/crio-bc5c203ddecb1cd233882be0e30773277ae17e35d7c26016e97a9986a2b8f381 WatchSource:0}: Error finding container bc5c203ddecb1cd233882be0e30773277ae17e35d7c26016e97a9986a2b8f381: Status 404 returned error can't find the container with id bc5c203ddecb1cd233882be0e30773277ae17e35d7c26016e97a9986a2b8f381
Sep 18 13:51:00.774564 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:00.774526    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75mghv"
Sep 18 13:51:00.871295 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:00.871268    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65l5sr"]
Sep 18 13:51:00.892636 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:51:00.892576    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ed3c114_22de_45a3_abee_7e663d48f87d.slice/crio-3f006c5f98626243371a365f3327059d198814fe10060e0ca9a5e81b27a3cec0 WatchSource:0}: Error finding container 3f006c5f98626243371a365f3327059d198814fe10060e0ca9a5e81b27a3cec0: Status 404 returned error can't find the container with id 3f006c5f98626243371a365f3327059d198814fe10060e0ca9a5e81b27a3cec0
Sep 18 13:51:00.953678 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:00.953650    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65l5sr" event={"ID":"6ed3c114-22de-45a3-abee-7e663d48f87d","Type":"ContainerStarted","Data":"3f006c5f98626243371a365f3327059d198814fe10060e0ca9a5e81b27a3cec0"}
Sep 18 13:51:00.955449 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:00.955427    2577 generic.go:356] "Generic (PLEG): container finished" podID="74f6f854-679e-4427-af63-622ffcc4d74b" containerID="96f24f9d0a5f449c6ac3e47987250c8207ca57fd4b72939b912db0cffd81c193" exitCode=0
Sep 18 13:51:00.955554 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:00.955512    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ablnsh" event={"ID":"74f6f854-679e-4427-af63-622ffcc4d74b","Type":"ContainerDied","Data":"96f24f9d0a5f449c6ac3e47987250c8207ca57fd4b72939b912db0cffd81c193"}
Sep 18 13:51:00.955612 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:00.955562    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ablnsh" event={"ID":"74f6f854-679e-4427-af63-622ffcc4d74b","Type":"ContainerStarted","Data":"bc5c203ddecb1cd233882be0e30773277ae17e35d7c26016e97a9986a2b8f381"}
Sep 18 13:51:00.973962 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:00.973939    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lfrn"]
Sep 18 13:51:01.061776 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:01.061752    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75mghv"]
Sep 18 13:51:01.071559 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:51:01.071532    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod617f1c94_bd5b_499e_9ba9_d079ab543794.slice/crio-8fd2d737e2cfc4687c8da1585b431f56395cca31784e6fa697fca4e9e2c00cff WatchSource:0}: Error finding container 8fd2d737e2cfc4687c8da1585b431f56395cca31784e6fa697fca4e9e2c00cff: Status 404 returned error can't find the container with id 8fd2d737e2cfc4687c8da1585b431f56395cca31784e6fa697fca4e9e2c00cff
Sep 18 13:51:01.962155 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:01.962122    2577 generic.go:356] "Generic (PLEG): container finished" podID="497a77e0-e6f0-4d88-aadc-66b7c31adc82" containerID="552ba6db5cb3f95c02b0d325f2495fb7cc54c0bcc821cf976ec1f6b017efc590" exitCode=0
Sep 18 13:51:01.962625 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:01.962206    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lfrn" event={"ID":"497a77e0-e6f0-4d88-aadc-66b7c31adc82","Type":"ContainerDied","Data":"552ba6db5cb3f95c02b0d325f2495fb7cc54c0bcc821cf976ec1f6b017efc590"}
Sep 18 13:51:01.962625 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:01.962253    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lfrn" event={"ID":"497a77e0-e6f0-4d88-aadc-66b7c31adc82","Type":"ContainerStarted","Data":"2fbe6507a54d8c2e90fa7bc1808ac47be7fee0bfb5b03c197893e37ed2779de5"}
Sep 18 13:51:01.963806 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:01.963782    2577 generic.go:356] "Generic (PLEG): container finished" podID="617f1c94-bd5b-499e-9ba9-d079ab543794" containerID="918199305a20ab73ef151051e3f4e1f87a602efbf9f574d388fceeeea5903280" exitCode=0
Sep 18 13:51:01.963906 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:01.963876    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75mghv" event={"ID":"617f1c94-bd5b-499e-9ba9-d079ab543794","Type":"ContainerDied","Data":"918199305a20ab73ef151051e3f4e1f87a602efbf9f574d388fceeeea5903280"}
Sep 18 13:51:01.963970 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:01.963933    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75mghv" event={"ID":"617f1c94-bd5b-499e-9ba9-d079ab543794","Type":"ContainerStarted","Data":"8fd2d737e2cfc4687c8da1585b431f56395cca31784e6fa697fca4e9e2c00cff"}
Sep 18 13:51:01.965684 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:01.965660    2577 generic.go:356] "Generic (PLEG): container finished" podID="6ed3c114-22de-45a3-abee-7e663d48f87d" containerID="5e6243e33bd6ceaa92092b9a3f6b1a24d3acddfaa06ac1577bd812593f68bce8" exitCode=0
Sep 18 13:51:01.965763 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:01.965686    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65l5sr" event={"ID":"6ed3c114-22de-45a3-abee-7e663d48f87d","Type":"ContainerDied","Data":"5e6243e33bd6ceaa92092b9a3f6b1a24d3acddfaa06ac1577bd812593f68bce8"}
Sep 18 13:51:04.979893 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:04.979863    2577 generic.go:356] "Generic (PLEG): container finished" podID="497a77e0-e6f0-4d88-aadc-66b7c31adc82" containerID="c162af28c033671c4492f4d4098bafbf1d2f4ec8cb8830a51d413850c2387a49" exitCode=0
Sep 18 13:51:04.980229 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:04.979954    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lfrn" event={"ID":"497a77e0-e6f0-4d88-aadc-66b7c31adc82","Type":"ContainerDied","Data":"c162af28c033671c4492f4d4098bafbf1d2f4ec8cb8830a51d413850c2387a49"}
Sep 18 13:51:05.986614 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:05.986578    2577 generic.go:356] "Generic (PLEG): container finished" podID="6ed3c114-22de-45a3-abee-7e663d48f87d" containerID="fd8e9d704580eb247fc594e39b376a7d43d5533738a32eeb8baf9b1574fde27f" exitCode=0
Sep 18 13:51:05.987049 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:05.986659    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65l5sr" event={"ID":"6ed3c114-22de-45a3-abee-7e663d48f87d","Type":"ContainerDied","Data":"fd8e9d704580eb247fc594e39b376a7d43d5533738a32eeb8baf9b1574fde27f"}
Sep 18 13:51:05.989098 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:05.989074    2577 generic.go:356] "Generic (PLEG): container finished" podID="497a77e0-e6f0-4d88-aadc-66b7c31adc82" containerID="6bb7227ca5760648e8e8a7f87cfb07221908cbc7c2b53124c7828e2b47481a02" exitCode=0
Sep 18 13:51:05.989200 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:05.989102    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lfrn" event={"ID":"497a77e0-e6f0-4d88-aadc-66b7c31adc82","Type":"ContainerDied","Data":"6bb7227ca5760648e8e8a7f87cfb07221908cbc7c2b53124c7828e2b47481a02"}
Sep 18 13:51:06.997525 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:06.997487    2577 generic.go:356] "Generic (PLEG): container finished" podID="6ed3c114-22de-45a3-abee-7e663d48f87d" containerID="647ebcbefd1937bfd0892a403dde8e7e6a472df9be40de0a013143e8598e72f1" exitCode=0
Sep 18 13:51:06.997934 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:06.997577    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65l5sr" event={"ID":"6ed3c114-22de-45a3-abee-7e663d48f87d","Type":"ContainerDied","Data":"647ebcbefd1937bfd0892a403dde8e7e6a472df9be40de0a013143e8598e72f1"}
Sep 18 13:51:07.121163 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:07.121142    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lfrn"
Sep 18 13:51:07.211756 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:07.211717    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-28gj4\" (UniqueName: \"kubernetes.io/projected/497a77e0-e6f0-4d88-aadc-66b7c31adc82-kube-api-access-28gj4\") pod \"497a77e0-e6f0-4d88-aadc-66b7c31adc82\" (UID: \"497a77e0-e6f0-4d88-aadc-66b7c31adc82\") "
Sep 18 13:51:07.211756 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:07.211760    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/497a77e0-e6f0-4d88-aadc-66b7c31adc82-bundle\") pod \"497a77e0-e6f0-4d88-aadc-66b7c31adc82\" (UID: \"497a77e0-e6f0-4d88-aadc-66b7c31adc82\") "
Sep 18 13:51:07.211992 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:07.211810    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/497a77e0-e6f0-4d88-aadc-66b7c31adc82-util\") pod \"497a77e0-e6f0-4d88-aadc-66b7c31adc82\" (UID: \"497a77e0-e6f0-4d88-aadc-66b7c31adc82\") "
Sep 18 13:51:07.212377 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:07.212350    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/497a77e0-e6f0-4d88-aadc-66b7c31adc82-bundle" (OuterVolumeSpecName: "bundle") pod "497a77e0-e6f0-4d88-aadc-66b7c31adc82" (UID: "497a77e0-e6f0-4d88-aadc-66b7c31adc82"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:51:07.213913 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:07.213890    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/497a77e0-e6f0-4d88-aadc-66b7c31adc82-kube-api-access-28gj4" (OuterVolumeSpecName: "kube-api-access-28gj4") pod "497a77e0-e6f0-4d88-aadc-66b7c31adc82" (UID: "497a77e0-e6f0-4d88-aadc-66b7c31adc82"). InnerVolumeSpecName "kube-api-access-28gj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:51:07.216628 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:07.216601    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/497a77e0-e6f0-4d88-aadc-66b7c31adc82-util" (OuterVolumeSpecName: "util") pod "497a77e0-e6f0-4d88-aadc-66b7c31adc82" (UID: "497a77e0-e6f0-4d88-aadc-66b7c31adc82"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:51:07.312615 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:07.312523    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28gj4\" (UniqueName: \"kubernetes.io/projected/497a77e0-e6f0-4d88-aadc-66b7c31adc82-kube-api-access-28gj4\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:51:07.312615 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:07.312555    2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/497a77e0-e6f0-4d88-aadc-66b7c31adc82-bundle\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:51:07.312615 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:07.312570    2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/497a77e0-e6f0-4d88-aadc-66b7c31adc82-util\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:51:08.004805 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:08.004773    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lfrn"
Sep 18 13:51:08.005174 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:08.004808    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lfrn" event={"ID":"497a77e0-e6f0-4d88-aadc-66b7c31adc82","Type":"ContainerDied","Data":"2fbe6507a54d8c2e90fa7bc1808ac47be7fee0bfb5b03c197893e37ed2779de5"}
Sep 18 13:51:08.005174 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:08.004868    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2fbe6507a54d8c2e90fa7bc1808ac47be7fee0bfb5b03c197893e37ed2779de5"
Sep 18 13:51:08.129990 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:08.129961    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65l5sr"
Sep 18 13:51:08.220815 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:08.220785    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sfxkl\" (UniqueName: \"kubernetes.io/projected/6ed3c114-22de-45a3-abee-7e663d48f87d-kube-api-access-sfxkl\") pod \"6ed3c114-22de-45a3-abee-7e663d48f87d\" (UID: \"6ed3c114-22de-45a3-abee-7e663d48f87d\") "
Sep 18 13:51:08.220982 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:08.220879    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6ed3c114-22de-45a3-abee-7e663d48f87d-util\") pod \"6ed3c114-22de-45a3-abee-7e663d48f87d\" (UID: \"6ed3c114-22de-45a3-abee-7e663d48f87d\") "
Sep 18 13:51:08.220982 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:08.220945    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6ed3c114-22de-45a3-abee-7e663d48f87d-bundle\") pod \"6ed3c114-22de-45a3-abee-7e663d48f87d\" (UID: \"6ed3c114-22de-45a3-abee-7e663d48f87d\") "
Sep 18 13:51:08.221414 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:08.221385    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ed3c114-22de-45a3-abee-7e663d48f87d-bundle" (OuterVolumeSpecName: "bundle") pod "6ed3c114-22de-45a3-abee-7e663d48f87d" (UID: "6ed3c114-22de-45a3-abee-7e663d48f87d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:51:08.222889 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:08.222867    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ed3c114-22de-45a3-abee-7e663d48f87d-kube-api-access-sfxkl" (OuterVolumeSpecName: "kube-api-access-sfxkl") pod "6ed3c114-22de-45a3-abee-7e663d48f87d" (UID: "6ed3c114-22de-45a3-abee-7e663d48f87d"). InnerVolumeSpecName "kube-api-access-sfxkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:51:08.225666 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:08.225642    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ed3c114-22de-45a3-abee-7e663d48f87d-util" (OuterVolumeSpecName: "util") pod "6ed3c114-22de-45a3-abee-7e663d48f87d" (UID: "6ed3c114-22de-45a3-abee-7e663d48f87d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:51:08.322457 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:08.322354    2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6ed3c114-22de-45a3-abee-7e663d48f87d-util\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:51:08.322457 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:08.322400    2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6ed3c114-22de-45a3-abee-7e663d48f87d-bundle\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:51:08.322457 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:08.322409    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfxkl\" (UniqueName: \"kubernetes.io/projected/6ed3c114-22de-45a3-abee-7e663d48f87d-kube-api-access-sfxkl\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:51:09.012571 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:09.012537    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65l5sr" event={"ID":"6ed3c114-22de-45a3-abee-7e663d48f87d","Type":"ContainerDied","Data":"3f006c5f98626243371a365f3327059d198814fe10060e0ca9a5e81b27a3cec0"}
Sep 18 13:51:09.012961 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:09.012588    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f006c5f98626243371a365f3327059d198814fe10060e0ca9a5e81b27a3cec0"
Sep 18 13:51:09.012961 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:09.012554    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65l5sr"
Sep 18 13:51:19.401587 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:19.401521    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-77b97c7f9b-q6nzl" podUID="8a8f3c05-66db-42a8-85e7-aaf007bb50fe" containerName="console" containerID="cri-o://86a72f358933feffd0385edfd0c5b39b47800cc53eebb1d7e3f108ea2246002c" gracePeriod=15
Sep 18 13:51:19.639292 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:19.639269    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77b97c7f9b-q6nzl"
Sep 18 13:51:19.720840 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:19.720743    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a8f3c05-66db-42a8-85e7-aaf007bb50fe-trusted-ca-bundle\") pod \"8a8f3c05-66db-42a8-85e7-aaf007bb50fe\" (UID: \"8a8f3c05-66db-42a8-85e7-aaf007bb50fe\") "
Sep 18 13:51:19.720840 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:19.720795    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8a8f3c05-66db-42a8-85e7-aaf007bb50fe-console-config\") pod \"8a8f3c05-66db-42a8-85e7-aaf007bb50fe\" (UID: \"8a8f3c05-66db-42a8-85e7-aaf007bb50fe\") "
Sep 18 13:51:19.721035 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:19.720860    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8a8f3c05-66db-42a8-85e7-aaf007bb50fe-oauth-serving-cert\") pod \"8a8f3c05-66db-42a8-85e7-aaf007bb50fe\" (UID: \"8a8f3c05-66db-42a8-85e7-aaf007bb50fe\") "
Sep 18 13:51:19.721035 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:19.720890    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8a8f3c05-66db-42a8-85e7-aaf007bb50fe-console-serving-cert\") pod \"8a8f3c05-66db-42a8-85e7-aaf007bb50fe\" (UID: \"8a8f3c05-66db-42a8-85e7-aaf007bb50fe\") "
Sep 18 13:51:19.721035 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:19.720919    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v299j\" (UniqueName: \"kubernetes.io/projected/8a8f3c05-66db-42a8-85e7-aaf007bb50fe-kube-api-access-v299j\") pod \"8a8f3c05-66db-42a8-85e7-aaf007bb50fe\" (UID: \"8a8f3c05-66db-42a8-85e7-aaf007bb50fe\") "
Sep 18 13:51:19.721035 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:19.720946    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8a8f3c05-66db-42a8-85e7-aaf007bb50fe-service-ca\") pod \"8a8f3c05-66db-42a8-85e7-aaf007bb50fe\" (UID: \"8a8f3c05-66db-42a8-85e7-aaf007bb50fe\") "
Sep 18 13:51:19.721035 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:19.720968    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8a8f3c05-66db-42a8-85e7-aaf007bb50fe-console-oauth-config\") pod \"8a8f3c05-66db-42a8-85e7-aaf007bb50fe\" (UID: \"8a8f3c05-66db-42a8-85e7-aaf007bb50fe\") "
Sep 18 13:51:19.721396 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:19.721233    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a8f3c05-66db-42a8-85e7-aaf007bb50fe-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "8a8f3c05-66db-42a8-85e7-aaf007bb50fe" (UID: "8a8f3c05-66db-42a8-85e7-aaf007bb50fe"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:51:19.721526 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:19.721365    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a8f3c05-66db-42a8-85e7-aaf007bb50fe-console-config" (OuterVolumeSpecName: "console-config") pod "8a8f3c05-66db-42a8-85e7-aaf007bb50fe" (UID: "8a8f3c05-66db-42a8-85e7-aaf007bb50fe"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:51:19.721526 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:19.721368    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a8f3c05-66db-42a8-85e7-aaf007bb50fe-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "8a8f3c05-66db-42a8-85e7-aaf007bb50fe" (UID: "8a8f3c05-66db-42a8-85e7-aaf007bb50fe"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:51:19.721526 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:19.721423    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a8f3c05-66db-42a8-85e7-aaf007bb50fe-service-ca" (OuterVolumeSpecName: "service-ca") pod "8a8f3c05-66db-42a8-85e7-aaf007bb50fe" (UID: "8a8f3c05-66db-42a8-85e7-aaf007bb50fe"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:51:19.723240 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:19.723211    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a8f3c05-66db-42a8-85e7-aaf007bb50fe-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "8a8f3c05-66db-42a8-85e7-aaf007bb50fe" (UID: "8a8f3c05-66db-42a8-85e7-aaf007bb50fe"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:51:19.723338 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:19.723263    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a8f3c05-66db-42a8-85e7-aaf007bb50fe-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "8a8f3c05-66db-42a8-85e7-aaf007bb50fe" (UID: "8a8f3c05-66db-42a8-85e7-aaf007bb50fe"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:51:19.723338 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:19.723277    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a8f3c05-66db-42a8-85e7-aaf007bb50fe-kube-api-access-v299j" (OuterVolumeSpecName: "kube-api-access-v299j") pod "8a8f3c05-66db-42a8-85e7-aaf007bb50fe" (UID: "8a8f3c05-66db-42a8-85e7-aaf007bb50fe"). InnerVolumeSpecName "kube-api-access-v299j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:51:19.821815 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:19.821780    2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8a8f3c05-66db-42a8-85e7-aaf007bb50fe-console-config\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:51:19.821815 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:19.821809    2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8a8f3c05-66db-42a8-85e7-aaf007bb50fe-oauth-serving-cert\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:51:19.821815 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:19.821840    2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8a8f3c05-66db-42a8-85e7-aaf007bb50fe-console-serving-cert\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:51:19.821815 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:19.821850    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v299j\" (UniqueName: \"kubernetes.io/projected/8a8f3c05-66db-42a8-85e7-aaf007bb50fe-kube-api-access-v299j\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:51:19.822121 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:19.821860    2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8a8f3c05-66db-42a8-85e7-aaf007bb50fe-service-ca\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:51:19.822121 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:19.821868    2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8a8f3c05-66db-42a8-85e7-aaf007bb50fe-console-oauth-config\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:51:19.822121 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:19.821876    2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a8f3c05-66db-42a8-85e7-aaf007bb50fe-trusted-ca-bundle\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:51:20.065185 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:20.065094    2577 generic.go:356] "Generic (PLEG): container finished" podID="8a8f3c05-66db-42a8-85e7-aaf007bb50fe" containerID="86a72f358933feffd0385edfd0c5b39b47800cc53eebb1d7e3f108ea2246002c" exitCode=0
Sep 18 13:51:20.065185 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:20.065145    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77b97c7f9b-q6nzl"
Sep 18 13:51:20.065185 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:20.065152    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77b97c7f9b-q6nzl" event={"ID":"8a8f3c05-66db-42a8-85e7-aaf007bb50fe","Type":"ContainerDied","Data":"86a72f358933feffd0385edfd0c5b39b47800cc53eebb1d7e3f108ea2246002c"}
Sep 18 13:51:20.065457 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:20.065193    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77b97c7f9b-q6nzl" event={"ID":"8a8f3c05-66db-42a8-85e7-aaf007bb50fe","Type":"ContainerDied","Data":"9ff454ca39ad1164387e3a4e08862485ecf051fddcbb04e3705a812c09339f14"}
Sep 18 13:51:20.065457 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:20.065212    2577 scope.go:117] "RemoveContainer" containerID="86a72f358933feffd0385edfd0c5b39b47800cc53eebb1d7e3f108ea2246002c"
Sep 18 13:51:20.074707 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:20.074689    2577 scope.go:117] "RemoveContainer" containerID="86a72f358933feffd0385edfd0c5b39b47800cc53eebb1d7e3f108ea2246002c"
Sep 18 13:51:20.074982 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:51:20.074962    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86a72f358933feffd0385edfd0c5b39b47800cc53eebb1d7e3f108ea2246002c\": container with ID starting with 86a72f358933feffd0385edfd0c5b39b47800cc53eebb1d7e3f108ea2246002c not found: ID does not exist" containerID="86a72f358933feffd0385edfd0c5b39b47800cc53eebb1d7e3f108ea2246002c"
Sep 18 13:51:20.075064 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:20.074993    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86a72f358933feffd0385edfd0c5b39b47800cc53eebb1d7e3f108ea2246002c"} err="failed to get container status \"86a72f358933feffd0385edfd0c5b39b47800cc53eebb1d7e3f108ea2246002c\": rpc error: code = NotFound desc = could not find container \"86a72f358933feffd0385edfd0c5b39b47800cc53eebb1d7e3f108ea2246002c\": container with ID starting with 86a72f358933feffd0385edfd0c5b39b47800cc53eebb1d7e3f108ea2246002c not found: ID does not exist"
Sep 18 13:51:20.087559 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:20.087536    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77b97c7f9b-q6nzl"]
Sep 18 13:51:20.092573 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:20.092547    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-77b97c7f9b-q6nzl"]
Sep 18 13:51:21.071426 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:21.071344    2577 generic.go:356] "Generic (PLEG): container finished" podID="74f6f854-679e-4427-af63-622ffcc4d74b" containerID="8f8d3a8ad1de58f4ab1c195021c44666c6f20e62aa7cbbe2398731514ee34d36" exitCode=0
Sep 18 13:51:21.071426 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:21.071413    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ablnsh" event={"ID":"74f6f854-679e-4427-af63-622ffcc4d74b","Type":"ContainerDied","Data":"8f8d3a8ad1de58f4ab1c195021c44666c6f20e62aa7cbbe2398731514ee34d36"}
Sep 18 13:51:21.073405 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:21.073383    2577 generic.go:356] "Generic (PLEG): container finished" podID="617f1c94-bd5b-499e-9ba9-d079ab543794" containerID="85a8b6f8cc24addf7b6a8383d13687077df59c3e6f5f36f6ceece09a521ac79a" exitCode=0
Sep 18 13:51:21.073494 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:21.073464    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75mghv" event={"ID":"617f1c94-bd5b-499e-9ba9-d079ab543794","Type":"ContainerDied","Data":"85a8b6f8cc24addf7b6a8383d13687077df59c3e6f5f36f6ceece09a521ac79a"}
Sep 18 13:51:21.106704 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:21.106683    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a8f3c05-66db-42a8-85e7-aaf007bb50fe" path="/var/lib/kubelet/pods/8a8f3c05-66db-42a8-85e7-aaf007bb50fe/volumes"
Sep 18 13:51:22.081194 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:22.081160    2577 generic.go:356] "Generic (PLEG): container finished" podID="74f6f854-679e-4427-af63-622ffcc4d74b" containerID="557d97cec9c076996f810ae10d6794d430694b365cef116bea5a5d67808e27f5" exitCode=0
Sep 18 13:51:22.081599 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:22.081249    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ablnsh" event={"ID":"74f6f854-679e-4427-af63-622ffcc4d74b","Type":"ContainerDied","Data":"557d97cec9c076996f810ae10d6794d430694b365cef116bea5a5d67808e27f5"}
Sep 18 13:51:22.083332 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:22.083308    2577 generic.go:356] "Generic (PLEG): container finished" podID="617f1c94-bd5b-499e-9ba9-d079ab543794" containerID="e044a9979da333bd9a8dc729a158a9dbb389d3cd8385e525feed389140a5ea3f" exitCode=0
Sep 18 13:51:22.083435 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:22.083387    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75mghv" event={"ID":"617f1c94-bd5b-499e-9ba9-d079ab543794","Type":"ContainerDied","Data":"e044a9979da333bd9a8dc729a158a9dbb389d3cd8385e525feed389140a5ea3f"}
Sep 18 13:51:23.229117 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:23.229041    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75mghv"
Sep 18 13:51:23.238583 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:23.238552    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ablnsh"
Sep 18 13:51:23.350684 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:23.350581    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/74f6f854-679e-4427-af63-622ffcc4d74b-bundle\") pod \"74f6f854-679e-4427-af63-622ffcc4d74b\" (UID: \"74f6f854-679e-4427-af63-622ffcc4d74b\") "
Sep 18 13:51:23.350684 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:23.350649    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/74f6f854-679e-4427-af63-622ffcc4d74b-util\") pod \"74f6f854-679e-4427-af63-622ffcc4d74b\" (UID: \"74f6f854-679e-4427-af63-622ffcc4d74b\") "
Sep 18 13:51:23.350684 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:23.350681    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/617f1c94-bd5b-499e-9ba9-d079ab543794-bundle\") pod \"617f1c94-bd5b-499e-9ba9-d079ab543794\" (UID: \"617f1c94-bd5b-499e-9ba9-d079ab543794\") "
Sep 18 13:51:23.350992 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:23.350720    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/617f1c94-bd5b-499e-9ba9-d079ab543794-util\") pod \"617f1c94-bd5b-499e-9ba9-d079ab543794\" (UID: \"617f1c94-bd5b-499e-9ba9-d079ab543794\") "
Sep 18 13:51:23.350992 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:23.350793    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j6td5\" (UniqueName: \"kubernetes.io/projected/74f6f854-679e-4427-af63-622ffcc4d74b-kube-api-access-j6td5\") pod \"74f6f854-679e-4427-af63-622ffcc4d74b\" (UID: \"74f6f854-679e-4427-af63-622ffcc4d74b\") "
Sep 18 13:51:23.350992 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:23.350845    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zs2fz\" (UniqueName: \"kubernetes.io/projected/617f1c94-bd5b-499e-9ba9-d079ab543794-kube-api-access-zs2fz\") pod \"617f1c94-bd5b-499e-9ba9-d079ab543794\" (UID: \"617f1c94-bd5b-499e-9ba9-d079ab543794\") "
Sep 18 13:51:23.351295 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:23.351262    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/617f1c94-bd5b-499e-9ba9-d079ab543794-bundle" (OuterVolumeSpecName: "bundle") pod "617f1c94-bd5b-499e-9ba9-d079ab543794" (UID: "617f1c94-bd5b-499e-9ba9-d079ab543794"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:51:23.351427 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:23.351403    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74f6f854-679e-4427-af63-622ffcc4d74b-bundle" (OuterVolumeSpecName: "bundle") pod "74f6f854-679e-4427-af63-622ffcc4d74b" (UID: "74f6f854-679e-4427-af63-622ffcc4d74b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:51:23.353581 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:23.353557    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/617f1c94-bd5b-499e-9ba9-d079ab543794-kube-api-access-zs2fz" (OuterVolumeSpecName: "kube-api-access-zs2fz") pod "617f1c94-bd5b-499e-9ba9-d079ab543794" (UID: "617f1c94-bd5b-499e-9ba9-d079ab543794"). InnerVolumeSpecName "kube-api-access-zs2fz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:51:23.353581 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:23.353570    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74f6f854-679e-4427-af63-622ffcc4d74b-kube-api-access-j6td5" (OuterVolumeSpecName: "kube-api-access-j6td5") pod "74f6f854-679e-4427-af63-622ffcc4d74b" (UID: "74f6f854-679e-4427-af63-622ffcc4d74b"). InnerVolumeSpecName "kube-api-access-j6td5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:51:23.355813 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:23.355787    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/617f1c94-bd5b-499e-9ba9-d079ab543794-util" (OuterVolumeSpecName: "util") pod "617f1c94-bd5b-499e-9ba9-d079ab543794" (UID: "617f1c94-bd5b-499e-9ba9-d079ab543794"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:51:23.356348 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:23.356328    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74f6f854-679e-4427-af63-622ffcc4d74b-util" (OuterVolumeSpecName: "util") pod "74f6f854-679e-4427-af63-622ffcc4d74b" (UID: "74f6f854-679e-4427-af63-622ffcc4d74b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:51:23.451555 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:23.451524    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6td5\" (UniqueName: \"kubernetes.io/projected/74f6f854-679e-4427-af63-622ffcc4d74b-kube-api-access-j6td5\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:51:23.451555 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:23.451550    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zs2fz\" (UniqueName: \"kubernetes.io/projected/617f1c94-bd5b-499e-9ba9-d079ab543794-kube-api-access-zs2fz\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:51:23.451555 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:23.451561    2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/74f6f854-679e-4427-af63-622ffcc4d74b-bundle\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:51:23.451754 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:23.451570    2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/74f6f854-679e-4427-af63-622ffcc4d74b-util\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:51:23.451754 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:23.451578    2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/617f1c94-bd5b-499e-9ba9-d079ab543794-bundle\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:51:23.451754 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:23.451586    2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/617f1c94-bd5b-499e-9ba9-d079ab543794-util\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:51:24.096257 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:24.096221    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ablnsh"
Sep 18 13:51:24.096257 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:24.096222    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ablnsh" event={"ID":"74f6f854-679e-4427-af63-622ffcc4d74b","Type":"ContainerDied","Data":"bc5c203ddecb1cd233882be0e30773277ae17e35d7c26016e97a9986a2b8f381"}
Sep 18 13:51:24.096488 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:24.096352    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc5c203ddecb1cd233882be0e30773277ae17e35d7c26016e97a9986a2b8f381"
Sep 18 13:51:24.098248 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:24.098225    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75mghv" event={"ID":"617f1c94-bd5b-499e-9ba9-d079ab543794","Type":"ContainerDied","Data":"8fd2d737e2cfc4687c8da1585b431f56395cca31784e6fa697fca4e9e2c00cff"}
Sep 18 13:51:24.098248 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:24.098246    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75mghv"
Sep 18 13:51:24.098409 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:24.098261    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8fd2d737e2cfc4687c8da1585b431f56395cca31784e6fa697fca4e9e2c00cff"
Sep 18 13:51:27.551575 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:27.551502    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:51:28.360810 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:28.360775    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:51:29.160865 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:29.160810    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:51:32.267729 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:32.267695    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:51:32.363737 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:32.363702    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:51:32.767251 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:32.767220    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:51:32.891103 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:32.891071    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:51:34.761979 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:34.761945    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:51:34.863007 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:34.862974    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:51:36.028519 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:36.028485    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:51:36.751686 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:36.751640    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:51:37.168266 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:37.168233    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:51:38.205143 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:38.205098    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:51:39.783070 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:39.783031    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-2ll27"]
Sep 18 13:51:39.783519 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:39.783227    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:51:39.783519 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:39.783405    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="497a77e0-e6f0-4d88-aadc-66b7c31adc82" containerName="pull"
Sep 18 13:51:39.783519 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:39.783418    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="497a77e0-e6f0-4d88-aadc-66b7c31adc82" containerName="pull"
Sep 18 13:51:39.783519 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:39.783436    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ed3c114-22de-45a3-abee-7e663d48f87d" containerName="util"
Sep 18 13:51:39.783519 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:39.783441    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ed3c114-22de-45a3-abee-7e663d48f87d" containerName="util"
Sep 18 13:51:39.783519 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:39.783449    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ed3c114-22de-45a3-abee-7e663d48f87d" containerName="pull"
Sep 18 13:51:39.783519 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:39.783454    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ed3c114-22de-45a3-abee-7e663d48f87d" containerName="pull"
Sep 18 13:51:39.783519 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:39.783461    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="497a77e0-e6f0-4d88-aadc-66b7c31adc82" containerName="util"
Sep 18 13:51:39.783519 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:39.783466    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="497a77e0-e6f0-4d88-aadc-66b7c31adc82" containerName="util"
Sep 18 13:51:39.783519 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:39.783471    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74f6f854-679e-4427-af63-622ffcc4d74b" containerName="extract"
Sep 18 13:51:39.783519 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:39.783476    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="74f6f854-679e-4427-af63-622ffcc4d74b" containerName="extract"
Sep 18 13:51:39.783519 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:39.783482    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74f6f854-679e-4427-af63-622ffcc4d74b" containerName="pull"
Sep 18 13:51:39.783519 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:39.783486    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="74f6f854-679e-4427-af63-622ffcc4d74b" containerName="pull"
Sep 18 13:51:39.783519 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:39.783495    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ed3c114-22de-45a3-abee-7e663d48f87d" containerName="extract"
Sep 18 13:51:39.783519 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:39.783500    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ed3c114-22de-45a3-abee-7e663d48f87d" containerName="extract"
Sep 18 13:51:39.783519 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:39.783505    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="617f1c94-bd5b-499e-9ba9-d079ab543794" containerName="pull"
Sep 18 13:51:39.783519 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:39.783511    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="617f1c94-bd5b-499e-9ba9-d079ab543794" containerName="pull"
Sep 18 13:51:39.783519 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:39.783517    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="497a77e0-e6f0-4d88-aadc-66b7c31adc82" containerName="extract"
Sep 18 13:51:39.783519 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:39.783522    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="497a77e0-e6f0-4d88-aadc-66b7c31adc82" containerName="extract"
Sep 18 13:51:39.783519 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:39.783529    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a8f3c05-66db-42a8-85e7-aaf007bb50fe" containerName="console"
Sep 18 13:51:39.784241 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:39.783534    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a8f3c05-66db-42a8-85e7-aaf007bb50fe" containerName="console"
Sep 18 13:51:39.784241 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:39.783540    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74f6f854-679e-4427-af63-622ffcc4d74b" containerName="util"
Sep 18 13:51:39.784241 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:39.783545    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="74f6f854-679e-4427-af63-622ffcc4d74b" containerName="util"
Sep 18 13:51:39.784241 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:39.783550    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="617f1c94-bd5b-499e-9ba9-d079ab543794" containerName="util"
Sep 18 13:51:39.784241 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:39.783555    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="617f1c94-bd5b-499e-9ba9-d079ab543794" containerName="util"
Sep 18 13:51:39.784241 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:39.783559    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="617f1c94-bd5b-499e-9ba9-d079ab543794" containerName="extract"
Sep 18 13:51:39.784241 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:39.783564    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="617f1c94-bd5b-499e-9ba9-d079ab543794" containerName="extract"
Sep 18 13:51:39.784241 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:39.783610    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="617f1c94-bd5b-499e-9ba9-d079ab543794" containerName="extract"
Sep 18 13:51:39.784241 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:39.783617    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ed3c114-22de-45a3-abee-7e663d48f87d" containerName="extract"
Sep 18 13:51:39.784241 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:39.783623    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="74f6f854-679e-4427-af63-622ffcc4d74b" containerName="extract"
Sep 18 13:51:39.784241 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:39.783628    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a8f3c05-66db-42a8-85e7-aaf007bb50fe" containerName="console"
Sep 18 13:51:39.784241 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:39.783634    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="497a77e0-e6f0-4d88-aadc-66b7c31adc82" containerName="extract"
Sep 18 13:51:39.787797 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:39.787775    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-2ll27"
Sep 18 13:51:39.789907 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:39.789890    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 18 13:51:39.790133 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:39.790121    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-xhqst\""
Sep 18 13:51:39.790729 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:39.790712    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 18 13:51:39.790802 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:39.790718    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 18 13:51:39.790802 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:39.790746    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 18 13:51:39.795567 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:39.795545    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-2ll27"]
Sep 18 13:51:39.879754 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:39.879720    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pk8w\" (UniqueName: \"kubernetes.io/projected/109ada03-9032-44a0-b216-e6852ce3abb9-kube-api-access-2pk8w\") pod \"kuadrant-console-plugin-fbdd4f445-2ll27\" (UID: \"109ada03-9032-44a0-b216-e6852ce3abb9\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-2ll27"
Sep 18 13:51:39.879938 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:39.879778    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/109ada03-9032-44a0-b216-e6852ce3abb9-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-2ll27\" (UID: \"109ada03-9032-44a0-b216-e6852ce3abb9\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-2ll27"
Sep 18 13:51:39.879938 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:39.879860    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/109ada03-9032-44a0-b216-e6852ce3abb9-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-2ll27\" (UID: \"109ada03-9032-44a0-b216-e6852ce3abb9\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-2ll27"
Sep 18 13:51:39.980443 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:39.980413    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pk8w\" (UniqueName: \"kubernetes.io/projected/109ada03-9032-44a0-b216-e6852ce3abb9-kube-api-access-2pk8w\") pod \"kuadrant-console-plugin-fbdd4f445-2ll27\" (UID: \"109ada03-9032-44a0-b216-e6852ce3abb9\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-2ll27"
Sep 18 13:51:39.980599 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:39.980472    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/109ada03-9032-44a0-b216-e6852ce3abb9-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-2ll27\" (UID: \"109ada03-9032-44a0-b216-e6852ce3abb9\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-2ll27"
Sep 18 13:51:39.980599 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:39.980496    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/109ada03-9032-44a0-b216-e6852ce3abb9-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-2ll27\" (UID: \"109ada03-9032-44a0-b216-e6852ce3abb9\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-2ll27"
Sep 18 13:51:39.981188 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:39.981163    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/109ada03-9032-44a0-b216-e6852ce3abb9-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-2ll27\" (UID: \"109ada03-9032-44a0-b216-e6852ce3abb9\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-2ll27"
Sep 18 13:51:39.982833 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:39.982809    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/109ada03-9032-44a0-b216-e6852ce3abb9-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-2ll27\" (UID: \"109ada03-9032-44a0-b216-e6852ce3abb9\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-2ll27"
Sep 18 13:51:39.985229 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:39.985195    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:51:39.989576 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:39.989556    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pk8w\" (UniqueName: \"kubernetes.io/projected/109ada03-9032-44a0-b216-e6852ce3abb9-kube-api-access-2pk8w\") pod \"kuadrant-console-plugin-fbdd4f445-2ll27\" (UID: \"109ada03-9032-44a0-b216-e6852ce3abb9\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-2ll27"
Sep 18 13:51:40.097695 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:40.097664    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-2ll27"
Sep 18 13:51:40.373488 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:40.373466    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-2ll27"]
Sep 18 13:51:40.375482 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:51:40.375458    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod109ada03_9032_44a0_b216_e6852ce3abb9.slice/crio-20bc0d5aeb69c9d81586494e23e6792f75aa99c19bd13eebee7368bd94bb71f4 WatchSource:0}: Error finding container 20bc0d5aeb69c9d81586494e23e6792f75aa99c19bd13eebee7368bd94bb71f4: Status 404 returned error can't find the container with id 20bc0d5aeb69c9d81586494e23e6792f75aa99c19bd13eebee7368bd94bb71f4
Sep 18 13:51:41.185536 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:41.185504    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-2ll27" event={"ID":"109ada03-9032-44a0-b216-e6852ce3abb9","Type":"ContainerStarted","Data":"20bc0d5aeb69c9d81586494e23e6792f75aa99c19bd13eebee7368bd94bb71f4"}
Sep 18 13:51:42.764695 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:42.764652    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:51:42.823676 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:42.823625    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:51:43.163669 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:43.163636    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:51:43.222926 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:43.222864    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:51:43.585045 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:43.584937    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:51:43.670315 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:43.670277    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:51:45.215398 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:45.215366    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-2ll27" event={"ID":"109ada03-9032-44a0-b216-e6852ce3abb9","Type":"ContainerStarted","Data":"ac3d4f4f862b923dc086237efc2bf2b7ab88fd1efafe409e12aee1a65be47352"}
Sep 18 13:51:45.233244 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:45.233178    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-2ll27" podStartSLOduration=1.569423899 podStartE2EDuration="6.233161765s" totalImagesPullingTime="4.663737866s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:51:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:51:45.231435975 +0000 UTC m=+454.284095301" watchObservedRunningTime="2026-09-18 13:51:45.233161765 +0000 UTC m=+454.285821092"
Sep 18 13:51:49.865613 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:49.865582    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:51:49.964746 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:49.964706    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:51:52.260913 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:52.260878    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:51:52.366113 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:52.366083    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:51:55.740876 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:55.740813    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:51:58.780073 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:58.780033    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:51:58.880379 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:51:58.880334    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:52:03.797692 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:03.797652    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:52:03.895183 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:03.895151    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:52:04.181864 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:04.181830    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:52:04.283476 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:04.283448    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:52:04.619887 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:04.619853    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:52:04.716184 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:04.716151    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:52:11.218389 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:11.218356    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:52:11.317100 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:11.317065    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:52:13.647292 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:13.647256    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:52:13.748410 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:13.748377    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:52:20.215621 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:20.215585    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:52:20.325485 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:20.325456    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:52:20.444793 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:20.444762    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:52:21.214325 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:21.214292    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:52:21.428341 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:21.428305    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:52:21.594628 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:21.594599    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:52:21.695310 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:21.695277    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:52:23.943734 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:23.943690    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:52:24.160536 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:24.160486    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:52:24.294488 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:24.294390    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:52:24.503647 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:24.503607    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 13:52:24.527356 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:24.527324    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 13:52:24.527535 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:24.527428    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"
Sep 18 13:52:24.529634 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:24.529608    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 18 13:52:24.541703 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:24.541674    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 13:52:24.647788 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:24.647751    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2272b7c5-5194-414b-84ea-5380c4a73d2e-config-file\") pod \"limitador-limitador-84994f7ffc-ljk2l\" (UID: \"2272b7c5-5194-414b-84ea-5380c4a73d2e\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"
Sep 18 13:52:24.647788 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:24.647787    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls7dk\" (UniqueName: \"kubernetes.io/projected/2272b7c5-5194-414b-84ea-5380c4a73d2e-kube-api-access-ls7dk\") pod \"limitador-limitador-84994f7ffc-ljk2l\" (UID: \"2272b7c5-5194-414b-84ea-5380c4a73d2e\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"
Sep 18 13:52:24.748722 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:24.748681    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2272b7c5-5194-414b-84ea-5380c4a73d2e-config-file\") pod \"limitador-limitador-84994f7ffc-ljk2l\" (UID: \"2272b7c5-5194-414b-84ea-5380c4a73d2e\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"
Sep 18 13:52:24.748722 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:24.748726    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls7dk\" (UniqueName: \"kubernetes.io/projected/2272b7c5-5194-414b-84ea-5380c4a73d2e-kube-api-access-ls7dk\") pod \"limitador-limitador-84994f7ffc-ljk2l\" (UID: \"2272b7c5-5194-414b-84ea-5380c4a73d2e\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"
Sep 18 13:52:24.749372 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:24.749350    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2272b7c5-5194-414b-84ea-5380c4a73d2e-config-file\") pod \"limitador-limitador-84994f7ffc-ljk2l\" (UID: \"2272b7c5-5194-414b-84ea-5380c4a73d2e\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"
Sep 18 13:52:24.758565 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:24.758542    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls7dk\" (UniqueName: \"kubernetes.io/projected/2272b7c5-5194-414b-84ea-5380c4a73d2e-kube-api-access-ls7dk\") pod \"limitador-limitador-84994f7ffc-ljk2l\" (UID: \"2272b7c5-5194-414b-84ea-5380c4a73d2e\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"
Sep 18 13:52:24.840603 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:24.840560    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"
Sep 18 13:52:25.007594 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:25.007510    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:52:25.107776 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:25.107711    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:52:25.118243 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:52:25.118208    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2272b7c5_5194_414b_84ea_5380c4a73d2e.slice/crio-01b15d481c6c0e95cf1b84fe0b30361d4b78d10e3d1adcd723a56b2b01bd768e WatchSource:0}: Error finding container 01b15d481c6c0e95cf1b84fe0b30361d4b78d10e3d1adcd723a56b2b01bd768e: Status 404 returned error can't find the container with id 01b15d481c6c0e95cf1b84fe0b30361d4b78d10e3d1adcd723a56b2b01bd768e
Sep 18 13:52:25.119227 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:25.119204    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 13:52:25.235371 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:25.235338    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-82642"]
Sep 18 13:52:25.239590 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:25.239568    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-82642"
Sep 18 13:52:25.241747 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:25.241726    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-9j8ss\""
Sep 18 13:52:25.248314 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:25.248292    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-82642"]
Sep 18 13:52:25.353789 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:25.353753    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pmmr\" (UniqueName: \"kubernetes.io/projected/91a1af10-1b4b-4908-be3b-7f4e4fe4bbb7-kube-api-access-9pmmr\") pod \"authorino-59c974bd48-82642\" (UID: \"91a1af10-1b4b-4908-be3b-7f4e4fe4bbb7\") " pod="kuadrant-system/authorino-59c974bd48-82642"
Sep 18 13:52:25.407887 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:25.407854    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-ljk2l" event={"ID":"2272b7c5-5194-414b-84ea-5380c4a73d2e","Type":"ContainerStarted","Data":"01b15d481c6c0e95cf1b84fe0b30361d4b78d10e3d1adcd723a56b2b01bd768e"}
Sep 18 13:52:25.455016 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:25.454979    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pmmr\" (UniqueName: \"kubernetes.io/projected/91a1af10-1b4b-4908-be3b-7f4e4fe4bbb7-kube-api-access-9pmmr\") pod \"authorino-59c974bd48-82642\" (UID: \"91a1af10-1b4b-4908-be3b-7f4e4fe4bbb7\") " pod="kuadrant-system/authorino-59c974bd48-82642"
Sep 18 13:52:25.464275 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:25.464247    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pmmr\" (UniqueName: \"kubernetes.io/projected/91a1af10-1b4b-4908-be3b-7f4e4fe4bbb7-kube-api-access-9pmmr\") pod \"authorino-59c974bd48-82642\" (UID: \"91a1af10-1b4b-4908-be3b-7f4e4fe4bbb7\") " pod="kuadrant-system/authorino-59c974bd48-82642"
Sep 18 13:52:25.549686 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:25.549645    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-82642"
Sep 18 13:52:25.850739 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:25.850704    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-82642"]
Sep 18 13:52:25.853702 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:52:25.853669    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod91a1af10_1b4b_4908_be3b_7f4e4fe4bbb7.slice/crio-3c00ffd99f75e0b046f0e03af5662ba2798f013fa87631aceed86d06fa210b42 WatchSource:0}: Error finding container 3c00ffd99f75e0b046f0e03af5662ba2798f013fa87631aceed86d06fa210b42: Status 404 returned error can't find the container with id 3c00ffd99f75e0b046f0e03af5662ba2798f013fa87631aceed86d06fa210b42
Sep 18 13:52:25.866927 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:25.866900    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:52:26.413847 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:26.413778    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-82642" event={"ID":"91a1af10-1b4b-4908-be3b-7f4e4fe4bbb7","Type":"ContainerStarted","Data":"3c00ffd99f75e0b046f0e03af5662ba2798f013fa87631aceed86d06fa210b42"}
Sep 18 13:52:27.514729 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:27.514678    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:52:29.437439 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:29.437401    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-82642" event={"ID":"91a1af10-1b4b-4908-be3b-7f4e4fe4bbb7","Type":"ContainerStarted","Data":"93550041429a810420db0aeefe37cb075bd1baaf581eb2804856863df23127d5"}
Sep 18 13:52:29.439033 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:29.439010    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-ljk2l" event={"ID":"2272b7c5-5194-414b-84ea-5380c4a73d2e","Type":"ContainerStarted","Data":"103922d92dde4ed9d86c72dc9d3d5a47045bb60e74eac41a33b21338f7443511"}
Sep 18 13:52:29.439155 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:29.439063    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"
Sep 18 13:52:29.456320 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:29.456261    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-82642" podStartSLOduration=1.653726636 podStartE2EDuration="4.456244941s" totalImagesPullingTime="2.802518305s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:52:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:52:29.453565448 +0000 UTC m=+498.506224786" watchObservedRunningTime="2026-09-18 13:52:29.456244941 +0000 UTC m=+498.508904328"
Sep 18 13:52:29.475642 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:29.475593    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-ljk2l" podStartSLOduration=1.943993999 podStartE2EDuration="5.475576188s" totalImagesPullingTime="3.531582189s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:52:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:52:29.474725236 +0000 UTC m=+498.527384563" watchObservedRunningTime="2026-09-18 13:52:29.475576188 +0000 UTC m=+498.528235514"
Sep 18 13:52:40.444930 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:40.444890    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"
Sep 18 13:52:45.191714 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:45.191657    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:52:45.291138 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:45.291095    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:52:45.686185 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:45.686140    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:52:58.263293 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:58.263254    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:52:58.363061 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:58.363027    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:52:58.659062 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:58.659026    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-t8lln"]
Sep 18 13:52:58.662143 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:58.662125    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-t8lln"
Sep 18 13:52:58.664590 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:58.664573    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 18 13:52:58.670324 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:58.670295    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-t8lln"]
Sep 18 13:52:58.706344 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:58.706315    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ba639896-2225-48fe-9ee8-a67782779a19-tls-cert\") pod \"authorino-6d4dfc649f-t8lln\" (UID: \"ba639896-2225-48fe-9ee8-a67782779a19\") " pod="kuadrant-system/authorino-6d4dfc649f-t8lln"
Sep 18 13:52:58.706344 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:58.706346    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mdt4\" (UniqueName: \"kubernetes.io/projected/ba639896-2225-48fe-9ee8-a67782779a19-kube-api-access-5mdt4\") pod \"authorino-6d4dfc649f-t8lln\" (UID: \"ba639896-2225-48fe-9ee8-a67782779a19\") " pod="kuadrant-system/authorino-6d4dfc649f-t8lln"
Sep 18 13:52:58.757003 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:58.756969    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-t8lln"]
Sep 18 13:52:58.757246 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:52:58.757224    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-5mdt4 tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-t8lln" podUID="ba639896-2225-48fe-9ee8-a67782779a19"
Sep 18 13:52:58.807642 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:58.807606    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mdt4\" (UniqueName: \"kubernetes.io/projected/ba639896-2225-48fe-9ee8-a67782779a19-kube-api-access-5mdt4\") pod \"authorino-6d4dfc649f-t8lln\" (UID: \"ba639896-2225-48fe-9ee8-a67782779a19\") " pod="kuadrant-system/authorino-6d4dfc649f-t8lln"
Sep 18 13:52:58.807805 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:58.807686    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ba639896-2225-48fe-9ee8-a67782779a19-tls-cert\") pod \"authorino-6d4dfc649f-t8lln\" (UID: \"ba639896-2225-48fe-9ee8-a67782779a19\") " pod="kuadrant-system/authorino-6d4dfc649f-t8lln"
Sep 18 13:52:58.810080 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:58.810051    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ba639896-2225-48fe-9ee8-a67782779a19-tls-cert\") pod \"authorino-6d4dfc649f-t8lln\" (UID: \"ba639896-2225-48fe-9ee8-a67782779a19\") " pod="kuadrant-system/authorino-6d4dfc649f-t8lln"
Sep 18 13:52:58.816263 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:58.816236    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mdt4\" (UniqueName: \"kubernetes.io/projected/ba639896-2225-48fe-9ee8-a67782779a19-kube-api-access-5mdt4\") pod \"authorino-6d4dfc649f-t8lln\" (UID: \"ba639896-2225-48fe-9ee8-a67782779a19\") " pod="kuadrant-system/authorino-6d4dfc649f-t8lln"
Sep 18 13:52:59.592797 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:59.592764    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-t8lln"
Sep 18 13:52:59.599870 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:59.599846    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-t8lln"
Sep 18 13:52:59.716147 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:59.716112    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5mdt4\" (UniqueName: \"kubernetes.io/projected/ba639896-2225-48fe-9ee8-a67782779a19-kube-api-access-5mdt4\") pod \"ba639896-2225-48fe-9ee8-a67782779a19\" (UID: \"ba639896-2225-48fe-9ee8-a67782779a19\") "
Sep 18 13:52:59.716316 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:59.716163    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ba639896-2225-48fe-9ee8-a67782779a19-tls-cert\") pod \"ba639896-2225-48fe-9ee8-a67782779a19\" (UID: \"ba639896-2225-48fe-9ee8-a67782779a19\") "
Sep 18 13:52:59.718239 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:59.718205    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba639896-2225-48fe-9ee8-a67782779a19-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "ba639896-2225-48fe-9ee8-a67782779a19" (UID: "ba639896-2225-48fe-9ee8-a67782779a19"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:52:59.718368 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:59.718229    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba639896-2225-48fe-9ee8-a67782779a19-kube-api-access-5mdt4" (OuterVolumeSpecName: "kube-api-access-5mdt4") pod "ba639896-2225-48fe-9ee8-a67782779a19" (UID: "ba639896-2225-48fe-9ee8-a67782779a19"). InnerVolumeSpecName "kube-api-access-5mdt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:52:59.817608 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:59.817568    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mdt4\" (UniqueName: \"kubernetes.io/projected/ba639896-2225-48fe-9ee8-a67782779a19-kube-api-access-5mdt4\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:52:59.817608 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:52:59.817602    2577 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ba639896-2225-48fe-9ee8-a67782779a19-tls-cert\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:53:00.597950 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:53:00.597913    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-t8lln"
Sep 18 13:53:00.629777 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:53:00.629749    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-t8lln"]
Sep 18 13:53:00.635430 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:53:00.635403    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-t8lln"]
Sep 18 13:53:00.958069 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:53:00.957989    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-82642"]
Sep 18 13:53:00.958214 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:53:00.958165    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-82642" podUID="91a1af10-1b4b-4908-be3b-7f4e4fe4bbb7" containerName="authorino" containerID="cri-o://93550041429a810420db0aeefe37cb075bd1baaf581eb2804856863df23127d5" gracePeriod=30
Sep 18 13:53:01.108787 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:53:01.108753    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba639896-2225-48fe-9ee8-a67782779a19" path="/var/lib/kubelet/pods/ba639896-2225-48fe-9ee8-a67782779a19/volumes"
Sep 18 13:53:01.211949 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:53:01.211897    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-82642"
Sep 18 13:53:01.328698 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:53:01.328661    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9pmmr\" (UniqueName: \"kubernetes.io/projected/91a1af10-1b4b-4908-be3b-7f4e4fe4bbb7-kube-api-access-9pmmr\") pod \"91a1af10-1b4b-4908-be3b-7f4e4fe4bbb7\" (UID: \"91a1af10-1b4b-4908-be3b-7f4e4fe4bbb7\") "
Sep 18 13:53:01.330928 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:53:01.330900    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91a1af10-1b4b-4908-be3b-7f4e4fe4bbb7-kube-api-access-9pmmr" (OuterVolumeSpecName: "kube-api-access-9pmmr") pod "91a1af10-1b4b-4908-be3b-7f4e4fe4bbb7" (UID: "91a1af10-1b4b-4908-be3b-7f4e4fe4bbb7"). InnerVolumeSpecName "kube-api-access-9pmmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:53:01.429139 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:53:01.429099    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pmmr\" (UniqueName: \"kubernetes.io/projected/91a1af10-1b4b-4908-be3b-7f4e4fe4bbb7-kube-api-access-9pmmr\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:53:01.604333 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:53:01.604302    2577 generic.go:356] "Generic (PLEG): container finished" podID="91a1af10-1b4b-4908-be3b-7f4e4fe4bbb7" containerID="93550041429a810420db0aeefe37cb075bd1baaf581eb2804856863df23127d5" exitCode=0
Sep 18 13:53:01.604847 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:53:01.604353    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-82642"
Sep 18 13:53:01.604847 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:53:01.604379    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-82642" event={"ID":"91a1af10-1b4b-4908-be3b-7f4e4fe4bbb7","Type":"ContainerDied","Data":"93550041429a810420db0aeefe37cb075bd1baaf581eb2804856863df23127d5"}
Sep 18 13:53:01.604847 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:53:01.604410    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-82642" event={"ID":"91a1af10-1b4b-4908-be3b-7f4e4fe4bbb7","Type":"ContainerDied","Data":"3c00ffd99f75e0b046f0e03af5662ba2798f013fa87631aceed86d06fa210b42"}
Sep 18 13:53:01.604847 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:53:01.604426    2577 scope.go:117] "RemoveContainer" containerID="93550041429a810420db0aeefe37cb075bd1baaf581eb2804856863df23127d5"
Sep 18 13:53:01.613239 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:53:01.613222    2577 scope.go:117] "RemoveContainer" containerID="93550041429a810420db0aeefe37cb075bd1baaf581eb2804856863df23127d5"
Sep 18 13:53:01.613533 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:53:01.613512    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93550041429a810420db0aeefe37cb075bd1baaf581eb2804856863df23127d5\": container with ID starting with 93550041429a810420db0aeefe37cb075bd1baaf581eb2804856863df23127d5 not found: ID does not exist" containerID="93550041429a810420db0aeefe37cb075bd1baaf581eb2804856863df23127d5"
Sep 18 13:53:01.613606 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:53:01.613545    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93550041429a810420db0aeefe37cb075bd1baaf581eb2804856863df23127d5"} err="failed to get container status \"93550041429a810420db0aeefe37cb075bd1baaf581eb2804856863df23127d5\": rpc error: code = NotFound desc = could not find container \"93550041429a810420db0aeefe37cb075bd1baaf581eb2804856863df23127d5\": container with ID starting with 93550041429a810420db0aeefe37cb075bd1baaf581eb2804856863df23127d5 not found: ID does not exist"
Sep 18 13:53:01.635043 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:53:01.635009    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-82642"]
Sep 18 13:53:01.638609 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:53:01.638585    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-82642"]
Sep 18 13:53:03.106464 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:53:03.106429    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91a1af10-1b4b-4908-be3b-7f4e4fe4bbb7" path="/var/lib/kubelet/pods/91a1af10-1b4b-4908-be3b-7f4e4fe4bbb7/volumes"
Sep 18 13:53:05.960585 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:53:05.960551    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:53:06.029217 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:53:06.029184    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:53:06.101537 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:53:06.101506    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:53:06.204207 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:53:06.204175    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:53:06.306319 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:53:06.306242    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:53:06.656959 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:53:06.656930    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:53:10.055279 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:53:10.055247    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:53:10.127246 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:53:10.127212    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:53:27.808154 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:53:27.808104    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:53:41.602081 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:53:41.602044    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:53:41.686081 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:53:41.686051    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:53:43.083945 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:53:43.083913    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:53:43.177357 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:53:43.177328    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:53:43.277275 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:53:43.277240    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:53:43.416751 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:53:43.416713    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:53:43.615017 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:53:43.614985    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:53:43.715480 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:53:43.715395    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:53:43.904214 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:53:43.904184    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 13:53:53.866108 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:53:53.866074    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:53:53.971761 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:53:53.971727    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:54:00.732131 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:00.732094    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:54:01.189859 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:01.189807    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 13:54:10.597105 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:10.597069    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:54:11.190070 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:11.190036    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 13:54:13.086537 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:13.086511    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-8tnxl_69d3178a-5769-4b5f-a243-30a1bedc5bd6/console-operator/1.log"
Sep 18 13:54:13.087065 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:13.086520    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-8tnxl_69d3178a-5769-4b5f-a243-30a1bedc5bd6/console-operator/1.log"
Sep 18 13:54:16.356602 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:16.356569    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:54:21.940798 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:21.940761    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:54:23.095879 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:23.095849    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 13:54:26.024657 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:26.024576    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:54:26.996027 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:26.995992    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 13:54:34.504466 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:34.504429    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2gnbv"]
Sep 18 13:54:34.504878 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:34.504724    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91a1af10-1b4b-4908-be3b-7f4e4fe4bbb7" containerName="authorino"
Sep 18 13:54:34.504878 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:34.504735    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="91a1af10-1b4b-4908-be3b-7f4e4fe4bbb7" containerName="authorino"
Sep 18 13:54:34.504878 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:34.504801    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="91a1af10-1b4b-4908-be3b-7f4e4fe4bbb7" containerName="authorino"
Sep 18 13:54:34.507795 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:34.507776    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2gnbv"
Sep 18 13:54:34.510426 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:34.510402    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 18 13:54:34.510613 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:34.510593    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 18 13:54:34.510735 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:34.510652    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-mt56b\""
Sep 18 13:54:34.510735 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:34.510652    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 18 13:54:34.516793 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:34.516765    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2gnbv"]
Sep 18 13:54:34.601403 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:34.601365    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:54:34.674500 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:34.674458    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e5aeebd-6155-49a7-883c-0309a304a033-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2gnbv\" (UID: \"3e5aeebd-6155-49a7-883c-0309a304a033\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2gnbv"
Sep 18 13:54:34.674500 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:34.674498    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlkzw\" (UniqueName: \"kubernetes.io/projected/3e5aeebd-6155-49a7-883c-0309a304a033-kube-api-access-hlkzw\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2gnbv\" (UID: \"3e5aeebd-6155-49a7-883c-0309a304a033\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2gnbv"
Sep 18 13:54:34.674716 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:34.674564    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e5aeebd-6155-49a7-883c-0309a304a033-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2gnbv\" (UID: \"3e5aeebd-6155-49a7-883c-0309a304a033\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2gnbv"
Sep 18 13:54:34.674716 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:34.674592    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e5aeebd-6155-49a7-883c-0309a304a033-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2gnbv\" (UID: \"3e5aeebd-6155-49a7-883c-0309a304a033\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2gnbv"
Sep 18 13:54:34.674716 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:34.674660    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e5aeebd-6155-49a7-883c-0309a304a033-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2gnbv\" (UID: \"3e5aeebd-6155-49a7-883c-0309a304a033\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2gnbv"
Sep 18 13:54:34.674813 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:34.674733    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e5aeebd-6155-49a7-883c-0309a304a033-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2gnbv\" (UID: \"3e5aeebd-6155-49a7-883c-0309a304a033\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2gnbv"
Sep 18 13:54:34.775877 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:34.775753    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e5aeebd-6155-49a7-883c-0309a304a033-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2gnbv\" (UID: \"3e5aeebd-6155-49a7-883c-0309a304a033\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2gnbv"
Sep 18 13:54:34.775877 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:34.775843    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e5aeebd-6155-49a7-883c-0309a304a033-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2gnbv\" (UID: \"3e5aeebd-6155-49a7-883c-0309a304a033\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2gnbv"
Sep 18 13:54:34.775877 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:34.775878    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlkzw\" (UniqueName: \"kubernetes.io/projected/3e5aeebd-6155-49a7-883c-0309a304a033-kube-api-access-hlkzw\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2gnbv\" (UID: \"3e5aeebd-6155-49a7-883c-0309a304a033\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2gnbv"
Sep 18 13:54:34.776133 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:34.775915    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e5aeebd-6155-49a7-883c-0309a304a033-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2gnbv\" (UID: \"3e5aeebd-6155-49a7-883c-0309a304a033\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2gnbv"
Sep 18 13:54:34.776133 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:34.775941    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e5aeebd-6155-49a7-883c-0309a304a033-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2gnbv\" (UID: \"3e5aeebd-6155-49a7-883c-0309a304a033\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2gnbv"
Sep 18 13:54:34.776133 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:34.775964    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e5aeebd-6155-49a7-883c-0309a304a033-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2gnbv\" (UID: \"3e5aeebd-6155-49a7-883c-0309a304a033\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2gnbv"
Sep 18 13:54:34.776352 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:34.776311    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e5aeebd-6155-49a7-883c-0309a304a033-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2gnbv\" (UID: \"3e5aeebd-6155-49a7-883c-0309a304a033\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2gnbv"
Sep 18 13:54:34.776473 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:34.776351    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e5aeebd-6155-49a7-883c-0309a304a033-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2gnbv\" (UID: \"3e5aeebd-6155-49a7-883c-0309a304a033\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2gnbv"
Sep 18 13:54:34.776473 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:34.776382    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e5aeebd-6155-49a7-883c-0309a304a033-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2gnbv\" (UID: \"3e5aeebd-6155-49a7-883c-0309a304a033\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2gnbv"
Sep 18 13:54:34.778060 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:34.778031    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e5aeebd-6155-49a7-883c-0309a304a033-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2gnbv\" (UID: \"3e5aeebd-6155-49a7-883c-0309a304a033\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2gnbv"
Sep 18 13:54:34.778320 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:34.778304    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e5aeebd-6155-49a7-883c-0309a304a033-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2gnbv\" (UID: \"3e5aeebd-6155-49a7-883c-0309a304a033\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2gnbv"
Sep 18 13:54:34.785645 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:34.785621    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlkzw\" (UniqueName: \"kubernetes.io/projected/3e5aeebd-6155-49a7-883c-0309a304a033-kube-api-access-hlkzw\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2gnbv\" (UID: \"3e5aeebd-6155-49a7-883c-0309a304a033\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2gnbv"
Sep 18 13:54:34.818637 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:34.818613    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2gnbv"
Sep 18 13:54:35.092335 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:35.092299    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2gnbv"]
Sep 18 13:54:35.095190 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:54:35.095161    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e5aeebd_6155_49a7_883c_0309a304a033.slice/crio-857e4f083146be78cc1a81826efd1aea1a0957c8f0b3ae2660215a0237cb6317 WatchSource:0}: Error finding container 857e4f083146be78cc1a81826efd1aea1a0957c8f0b3ae2660215a0237cb6317: Status 404 returned error can't find the container with id 857e4f083146be78cc1a81826efd1aea1a0957c8f0b3ae2660215a0237cb6317
Sep 18 13:54:35.096574 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:35.096556    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 13:54:36.070662 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:36.070633    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2gnbv" event={"ID":"3e5aeebd-6155-49a7-883c-0309a304a033","Type":"ContainerStarted","Data":"857e4f083146be78cc1a81826efd1aea1a0957c8f0b3ae2660215a0237cb6317"}
Sep 18 13:54:36.092171 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:36.092138    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 13:54:37.080551 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:37.080506    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2gnbv" event={"ID":"3e5aeebd-6155-49a7-883c-0309a304a033","Type":"ContainerStarted","Data":"eccbb08520e2be92142f277c4d41a647a1ca6987f05494786975c584579661e9"}
Sep 18 13:54:37.107786 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:37.107720    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2gnbv" podStartSLOduration=2.145964269 podStartE2EDuration="3.107700746s" totalImagesPullingTime="961.736477ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:54:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:54:37.105777185 +0000 UTC m=+626.158436513" watchObservedRunningTime="2026-09-18 13:54:37.107700746 +0000 UTC m=+626.160360075"
Sep 18 13:54:39.024314 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:39.024278    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:54:41.794280 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:41.794243    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 13:54:44.819457 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:44.819423    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2gnbv"
Sep 18 13:54:44.819969 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:44.819513    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2gnbv"
Sep 18 13:54:44.832007 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:44.831984    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2gnbv"
Sep 18 13:54:45.130172 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:45.130141    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2gnbv"
Sep 18 13:54:51.315302 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:51.315269    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:54:51.376179 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:51.376145    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:54:53.603252 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:53.603220    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-9vhpj"]
Sep 18 13:54:53.613057 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:53.613026    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9vhpj"
Sep 18 13:54:53.616548 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:53.616523    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 18 13:54:53.618736 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:53.618712    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-9vhpj"]
Sep 18 13:54:53.629646 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:53.629625    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e41aa385-7758-49b1-8d2f-abfa51d38a99-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9vhpj\" (UID: \"e41aa385-7758-49b1-8d2f-abfa51d38a99\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9vhpj"
Sep 18 13:54:53.629755 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:53.629680    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e41aa385-7758-49b1-8d2f-abfa51d38a99-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9vhpj\" (UID: \"e41aa385-7758-49b1-8d2f-abfa51d38a99\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9vhpj"
Sep 18 13:54:53.629755 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:53.629745    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf2kt\" (UniqueName: \"kubernetes.io/projected/e41aa385-7758-49b1-8d2f-abfa51d38a99-kube-api-access-cf2kt\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9vhpj\" (UID: \"e41aa385-7758-49b1-8d2f-abfa51d38a99\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9vhpj"
Sep 18 13:54:53.629873 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:53.629786    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e41aa385-7758-49b1-8d2f-abfa51d38a99-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9vhpj\" (UID: \"e41aa385-7758-49b1-8d2f-abfa51d38a99\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9vhpj"
Sep 18 13:54:53.629873 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:53.629846    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e41aa385-7758-49b1-8d2f-abfa51d38a99-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9vhpj\" (UID: \"e41aa385-7758-49b1-8d2f-abfa51d38a99\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9vhpj"
Sep 18 13:54:53.629947 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:53.629878    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e41aa385-7758-49b1-8d2f-abfa51d38a99-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9vhpj\" (UID: \"e41aa385-7758-49b1-8d2f-abfa51d38a99\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9vhpj"
Sep 18 13:54:53.653679 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:53.653655    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:54:53.730193 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:53.730171    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e41aa385-7758-49b1-8d2f-abfa51d38a99-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9vhpj\" (UID: \"e41aa385-7758-49b1-8d2f-abfa51d38a99\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9vhpj"
Sep 18 13:54:53.730299 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:53.730203    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf2kt\" (UniqueName: \"kubernetes.io/projected/e41aa385-7758-49b1-8d2f-abfa51d38a99-kube-api-access-cf2kt\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9vhpj\" (UID: \"e41aa385-7758-49b1-8d2f-abfa51d38a99\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9vhpj"
Sep 18 13:54:53.730299 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:53.730244    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e41aa385-7758-49b1-8d2f-abfa51d38a99-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9vhpj\" (UID: \"e41aa385-7758-49b1-8d2f-abfa51d38a99\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9vhpj"
Sep 18 13:54:53.730299 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:53.730279    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e41aa385-7758-49b1-8d2f-abfa51d38a99-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9vhpj\" (UID: \"e41aa385-7758-49b1-8d2f-abfa51d38a99\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9vhpj"
Sep 18 13:54:53.730423 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:53.730310    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e41aa385-7758-49b1-8d2f-abfa51d38a99-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9vhpj\" (UID: \"e41aa385-7758-49b1-8d2f-abfa51d38a99\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9vhpj"
Sep 18 13:54:53.730423 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:53.730337    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e41aa385-7758-49b1-8d2f-abfa51d38a99-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9vhpj\" (UID: \"e41aa385-7758-49b1-8d2f-abfa51d38a99\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9vhpj"
Sep 18 13:54:53.730611 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:53.730592    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e41aa385-7758-49b1-8d2f-abfa51d38a99-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9vhpj\" (UID: \"e41aa385-7758-49b1-8d2f-abfa51d38a99\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9vhpj"
Sep 18 13:54:53.730707 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:53.730679    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e41aa385-7758-49b1-8d2f-abfa51d38a99-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9vhpj\" (UID: \"e41aa385-7758-49b1-8d2f-abfa51d38a99\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9vhpj"
Sep 18 13:54:53.730707 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:53.730696    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e41aa385-7758-49b1-8d2f-abfa51d38a99-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9vhpj\" (UID: \"e41aa385-7758-49b1-8d2f-abfa51d38a99\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9vhpj"
Sep 18 13:54:53.732506 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:53.732488    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e41aa385-7758-49b1-8d2f-abfa51d38a99-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9vhpj\" (UID: \"e41aa385-7758-49b1-8d2f-abfa51d38a99\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9vhpj"
Sep 18 13:54:53.732765 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:53.732747    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e41aa385-7758-49b1-8d2f-abfa51d38a99-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9vhpj\" (UID: \"e41aa385-7758-49b1-8d2f-abfa51d38a99\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9vhpj"
Sep 18 13:54:53.737889 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:53.737873    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf2kt\" (UniqueName: \"kubernetes.io/projected/e41aa385-7758-49b1-8d2f-abfa51d38a99-kube-api-access-cf2kt\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9vhpj\" (UID: \"e41aa385-7758-49b1-8d2f-abfa51d38a99\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9vhpj"
Sep 18 13:54:53.924096 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:53.924012    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9vhpj"
Sep 18 13:54:54.196624 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:54.196598    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-9vhpj"]
Sep 18 13:54:54.198566 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:54:54.198540    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode41aa385_7758_49b1_8d2f_abfa51d38a99.slice/crio-72d6c5a3664db5c325f06993bac9f36fb026ef7a00edf43740a993a6f7ec279f WatchSource:0}: Error finding container 72d6c5a3664db5c325f06993bac9f36fb026ef7a00edf43740a993a6f7ec279f: Status 404 returned error can't find the container with id 72d6c5a3664db5c325f06993bac9f36fb026ef7a00edf43740a993a6f7ec279f
Sep 18 13:54:55.168243 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:55.168209    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9vhpj" event={"ID":"e41aa385-7758-49b1-8d2f-abfa51d38a99","Type":"ContainerStarted","Data":"e9b6d9344f873f333be0b6c08132e73399a6e0632635476850f1e9216d430b47"}
Sep 18 13:54:55.168243 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:55.168253    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9vhpj" event={"ID":"e41aa385-7758-49b1-8d2f-abfa51d38a99","Type":"ContainerStarted","Data":"72d6c5a3664db5c325f06993bac9f36fb026ef7a00edf43740a993a6f7ec279f"}
Sep 18 13:54:55.186756 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:55.186702    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9vhpj" podStartSLOduration=2.002629144 podStartE2EDuration="2.186686325s" totalImagesPullingTime="184.057181ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:54:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:54:55.184267228 +0000 UTC m=+644.236926555" watchObservedRunningTime="2026-09-18 13:54:55.186686325 +0000 UTC m=+644.239345679"
Sep 18 13:54:55.896774 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:54:55.896736    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 13:55:03.925001 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:55:03.924961    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9vhpj"
Sep 18 13:55:03.925001 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:55:03.925010    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9vhpj"
Sep 18 13:55:03.937510 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:55:03.937487    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9vhpj"
Sep 18 13:55:04.222577 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:55:04.222503    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9vhpj"
Sep 18 13:55:49.698045 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:55:49.698004    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6cd9cc5c75-j2z8r"]
Sep 18 13:55:49.701297 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:55:49.701274    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6cd9cc5c75-j2z8r"
Sep 18 13:55:49.703485 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:55:49.703465    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-9j8ss\""
Sep 18 13:55:49.704391 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:55:49.704372    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 18 13:55:49.714646 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:55:49.714621    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6cd9cc5c75-j2z8r"]
Sep 18 13:55:49.860332 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:55:49.860298    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4f42e29e-9888-4eab-847f-159c841c49ed-tls-cert\") pod \"authorino-6cd9cc5c75-j2z8r\" (UID: \"4f42e29e-9888-4eab-847f-159c841c49ed\") " pod="kuadrant-system/authorino-6cd9cc5c75-j2z8r"
Sep 18 13:55:49.860487 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:55:49.860370    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg5tz\" (UniqueName: \"kubernetes.io/projected/4f42e29e-9888-4eab-847f-159c841c49ed-kube-api-access-gg5tz\") pod \"authorino-6cd9cc5c75-j2z8r\" (UID: \"4f42e29e-9888-4eab-847f-159c841c49ed\") " pod="kuadrant-system/authorino-6cd9cc5c75-j2z8r"
Sep 18 13:55:49.961672 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:55:49.961594    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg5tz\" (UniqueName: \"kubernetes.io/projected/4f42e29e-9888-4eab-847f-159c841c49ed-kube-api-access-gg5tz\") pod \"authorino-6cd9cc5c75-j2z8r\" (UID: \"4f42e29e-9888-4eab-847f-159c841c49ed\") " pod="kuadrant-system/authorino-6cd9cc5c75-j2z8r"
Sep 18 13:55:49.961672 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:55:49.961644    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4f42e29e-9888-4eab-847f-159c841c49ed-tls-cert\") pod \"authorino-6cd9cc5c75-j2z8r\" (UID: \"4f42e29e-9888-4eab-847f-159c841c49ed\") " pod="kuadrant-system/authorino-6cd9cc5c75-j2z8r"
Sep 18 13:55:49.964060 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:55:49.964031    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4f42e29e-9888-4eab-847f-159c841c49ed-tls-cert\") pod \"authorino-6cd9cc5c75-j2z8r\" (UID: \"4f42e29e-9888-4eab-847f-159c841c49ed\") " pod="kuadrant-system/authorino-6cd9cc5c75-j2z8r"
Sep 18 13:55:49.969866 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:55:49.969818    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg5tz\" (UniqueName: \"kubernetes.io/projected/4f42e29e-9888-4eab-847f-159c841c49ed-kube-api-access-gg5tz\") pod \"authorino-6cd9cc5c75-j2z8r\" (UID: \"4f42e29e-9888-4eab-847f-159c841c49ed\") " pod="kuadrant-system/authorino-6cd9cc5c75-j2z8r"
Sep 18 13:55:50.009875 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:55:50.009850    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6cd9cc5c75-j2z8r"
Sep 18 13:55:50.279884 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:55:50.279721    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6cd9cc5c75-j2z8r"]
Sep 18 13:55:50.282076 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:55:50.282045    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4f42e29e_9888_4eab_847f_159c841c49ed.slice/crio-fd7c455a6ddfe1f2c624bb634b2759c38f25349dc93813ce3b7b33eae86f402b WatchSource:0}: Error finding container fd7c455a6ddfe1f2c624bb634b2759c38f25349dc93813ce3b7b33eae86f402b: Status 404 returned error can't find the container with id fd7c455a6ddfe1f2c624bb634b2759c38f25349dc93813ce3b7b33eae86f402b
Sep 18 13:55:50.441023 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:55:50.440984    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6cd9cc5c75-j2z8r" event={"ID":"4f42e29e-9888-4eab-847f-159c841c49ed","Type":"ContainerStarted","Data":"fd7c455a6ddfe1f2c624bb634b2759c38f25349dc93813ce3b7b33eae86f402b"}
Sep 18 13:55:51.447744 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:55:51.447637    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6cd9cc5c75-j2z8r" event={"ID":"4f42e29e-9888-4eab-847f-159c841c49ed","Type":"ContainerStarted","Data":"1d8cbb77e8b95625903366ddf2d1289dd9c706311400999557cc3275610d571f"}
Sep 18 13:55:51.465528 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:55:51.465469    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6cd9cc5c75-j2z8r" podStartSLOduration=1.617438193 podStartE2EDuration="2.465451833s" totalImagesPullingTime="848.01364ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:55:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:55:51.463690805 +0000 UTC m=+700.516350132" watchObservedRunningTime="2026-09-18 13:55:51.465451833 +0000 UTC m=+700.518111164"
Sep 18 13:55:52.012059 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:55:52.012022    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:55:52.110782 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:55:52.110733    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:55:53.121842 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:55:53.121794    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:55:53.282723 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:55:53.282687    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:55:57.070998 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:55:57.070918    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:55:57.165524 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:55:57.165491    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:10.232203 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:10.232170    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:10.289174 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:10.289139    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:10.355904 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:10.355872    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:10.664541 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:10.664511    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:10.809951 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:10.809914    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:10.899620 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:10.899588    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:11.003349 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:11.003265    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:11.104057 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:11.104026    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:11.165738 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:11.165701    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:11.710329 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:11.710259    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:11.807745 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:11.807716    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:12.140637 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:12.140606    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:12.205683 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:12.205647    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7636-a4b2bb"]
Sep 18 13:56:12.209025 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:12.209007    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7636-a4b2bb"
Sep 18 13:56:12.211426 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:12.211401    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-knszx\""
Sep 18 13:56:12.216688 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:12.216666    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7636-a4b2bb"]
Sep 18 13:56:12.251937 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:12.251902    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g4dp\" (UniqueName: \"kubernetes.io/projected/133619cc-1c31-4586-8b11-2c9ddb988180-kube-api-access-2g4dp\") pod \"test-curl-7636-a4b2bb\" (UID: \"133619cc-1c31-4586-8b11-2c9ddb988180\") " pod="opendatahub/test-curl-7636-a4b2bb"
Sep 18 13:56:12.353235 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:12.353202    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g4dp\" (UniqueName: \"kubernetes.io/projected/133619cc-1c31-4586-8b11-2c9ddb988180-kube-api-access-2g4dp\") pod \"test-curl-7636-a4b2bb\" (UID: \"133619cc-1c31-4586-8b11-2c9ddb988180\") " pod="opendatahub/test-curl-7636-a4b2bb"
Sep 18 13:56:12.362415 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:12.362384    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g4dp\" (UniqueName: \"kubernetes.io/projected/133619cc-1c31-4586-8b11-2c9ddb988180-kube-api-access-2g4dp\") pod \"test-curl-7636-a4b2bb\" (UID: \"133619cc-1c31-4586-8b11-2c9ddb988180\") " pod="opendatahub/test-curl-7636-a4b2bb"
Sep 18 13:56:12.519464 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:12.519361    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7636-a4b2bb"
Sep 18 13:56:12.792164 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:12.792136    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7636-a4b2bb"]
Sep 18 13:56:12.794211 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:56:12.794178    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod133619cc_1c31_4586_8b11_2c9ddb988180.slice/crio-19a2503bdb902828e848e41c84cce2c415429c5897257ffc9a84a7fdbc467619 WatchSource:0}: Error finding container 19a2503bdb902828e848e41c84cce2c415429c5897257ffc9a84a7fdbc467619: Status 404 returned error can't find the container with id 19a2503bdb902828e848e41c84cce2c415429c5897257ffc9a84a7fdbc467619
Sep 18 13:56:13.563110 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:13.563073    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7636-a4b2bb" event={"ID":"133619cc-1c31-4586-8b11-2c9ddb988180","Type":"ContainerStarted","Data":"19a2503bdb902828e848e41c84cce2c415429c5897257ffc9a84a7fdbc467619"}
Sep 18 13:56:13.938506 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:13.938474    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:14.034236 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:14.034195    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:14.184173 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:14.184132    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:14.270735 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:14.270654    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 13:56:15.143932 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:15.143906    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:15.242224 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:15.242130    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:15.342343 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:15.342309    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:15.576642 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:15.576550    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7636-a4b2bb" event={"ID":"133619cc-1c31-4586-8b11-2c9ddb988180","Type":"ContainerStarted","Data":"0136804dfb455c8e25840c6009bf123086525fc829f2cb6835d66239fd7a7a96"}
Sep 18 13:56:15.594950 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:15.594894    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7636-a4b2bb" podStartSLOduration=1.415164268 podStartE2EDuration="3.594878987s" totalImagesPullingTime="2.179714719s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:56:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:56:15.593964498 +0000 UTC m=+724.646623825" watchObservedRunningTime="2026-09-18 13:56:15.594878987 +0000 UTC m=+724.647538329"
Sep 18 13:56:15.733595 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:15.733565    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:15.823287 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:15.823253    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:16.208585 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:16.208551    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7636-a4b2bb"]
Sep 18 13:56:16.221388 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:16.221362    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7636-a4b2bb"]
Sep 18 13:56:16.222322 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:16.222299    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:16.329495 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:16.329393    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:16.334427 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:16.334399    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7636-50821c"]
Sep 18 13:56:16.334733 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:16.334721    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="133619cc-1c31-4586-8b11-2c9ddb988180" containerName="test-curl-7636-a4b2bb"
Sep 18 13:56:16.334781 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:16.334735    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="133619cc-1c31-4586-8b11-2c9ddb988180" containerName="test-curl-7636-a4b2bb"
Sep 18 13:56:16.334817 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:16.334803    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="133619cc-1c31-4586-8b11-2c9ddb988180" containerName="test-curl-7636-a4b2bb"
Sep 18 13:56:16.337922 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:16.337906    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7636-50821c"
Sep 18 13:56:16.344079 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:16.344056    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7636-50821c"]
Sep 18 13:56:16.382211 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:16.382175    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w6ww\" (UniqueName: \"kubernetes.io/projected/d13cb3f8-13f6-4976-8bce-25b64f9ce2c1-kube-api-access-6w6ww\") pod \"test-curl-7636-50821c\" (UID: \"d13cb3f8-13f6-4976-8bce-25b64f9ce2c1\") " pod="opendatahub/test-curl-7636-50821c"
Sep 18 13:56:16.483194 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:16.483152    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w6ww\" (UniqueName: \"kubernetes.io/projected/d13cb3f8-13f6-4976-8bce-25b64f9ce2c1-kube-api-access-6w6ww\") pod \"test-curl-7636-50821c\" (UID: \"d13cb3f8-13f6-4976-8bce-25b64f9ce2c1\") " pod="opendatahub/test-curl-7636-50821c"
Sep 18 13:56:16.491746 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:16.491722    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w6ww\" (UniqueName: \"kubernetes.io/projected/d13cb3f8-13f6-4976-8bce-25b64f9ce2c1-kube-api-access-6w6ww\") pod \"test-curl-7636-50821c\" (UID: \"d13cb3f8-13f6-4976-8bce-25b64f9ce2c1\") " pod="opendatahub/test-curl-7636-50821c"
Sep 18 13:56:16.581740 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:56:16.581661    2577 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7636-a4b2bb" container="test-curl-7636-a4b2bb"
Sep 18 13:56:16.648470 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:16.648432    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7636-50821c"
Sep 18 13:56:16.699863 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:16.699812    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:16.798066 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:16.798027    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:16.924171 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:16.924134    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:16.941163 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:16.941133    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7636-50821c"]
Sep 18 13:56:16.943322 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:56:16.943296    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd13cb3f8_13f6_4976_8bce_25b64f9ce2c1.slice/crio-5ab412cff8c88045d217ca318709bec7aba44ead24145f8b0b77d607fdf2a3e5 WatchSource:0}: Error finding container 5ab412cff8c88045d217ca318709bec7aba44ead24145f8b0b77d607fdf2a3e5: Status 404 returned error can't find the container with id 5ab412cff8c88045d217ca318709bec7aba44ead24145f8b0b77d607fdf2a3e5
Sep 18 13:56:17.331172 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:17.331084    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:17.588416 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:17.588380    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7636-50821c" event={"ID":"d13cb3f8-13f6-4976-8bce-25b64f9ce2c1","Type":"ContainerStarted","Data":"86ab2aaa68962b839a833da6a3a9e55b9586fda34f6437052546ac451387be5c"}
Sep 18 13:56:17.588416 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:17.588421    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7636-50821c" event={"ID":"d13cb3f8-13f6-4976-8bce-25b64f9ce2c1","Type":"ContainerStarted","Data":"5ab412cff8c88045d217ca318709bec7aba44ead24145f8b0b77d607fdf2a3e5"}
Sep 18 13:56:17.588667 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:17.588587    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7636-a4b2bb" podUID="133619cc-1c31-4586-8b11-2c9ddb988180" containerName="test-curl-7636-a4b2bb" containerID="cri-o://0136804dfb455c8e25840c6009bf123086525fc829f2cb6835d66239fd7a7a96" gracePeriod=2
Sep 18 13:56:17.606749 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:17.606701    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7636-50821c" podStartSLOduration=1.606683432 podStartE2EDuration="1.606683432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:56:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:56:17.605619036 +0000 UTC m=+726.658278354" watchObservedRunningTime="2026-09-18 13:56:17.606683432 +0000 UTC m=+726.659342753"
Sep 18 13:56:18.131935 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:18.131902    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:18.190023 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:18.189991    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7636-50821c"]
Sep 18 13:56:18.194188 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:18.194163    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7636-50821c"]
Sep 18 13:56:18.593394 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:56:18.593365    2577 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7636-50821c" container="test-curl-7636-50821c"
Sep 18 13:56:18.678418 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:18.678385    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:18.949598 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:18.949519    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c"]
Sep 18 13:56:18.949886 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:18.949872    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d13cb3f8-13f6-4976-8bce-25b64f9ce2c1" containerName="test-curl-7636-50821c"
Sep 18 13:56:18.949942 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:18.949888    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d13cb3f8-13f6-4976-8bce-25b64f9ce2c1" containerName="test-curl-7636-50821c"
Sep 18 13:56:18.949975 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:18.949953    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d13cb3f8-13f6-4976-8bce-25b64f9ce2c1" containerName="test-curl-7636-50821c"
Sep 18 13:56:18.952974 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:18.952957    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c"
Sep 18 13:56:18.955857 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:18.955832    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 18 13:56:18.959297 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:18.959279    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-70f06e-openshift-default-dockercfg-cslrm\""
Sep 18 13:56:18.974421 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:18.974395    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c"]
Sep 18 13:56:18.989847 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:18.989796    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:19.004790 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:19.004769    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62972a5e-7296-4486-928b-29f00380ef52-istio-envoy\") pod \"e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c\" (UID: \"62972a5e-7296-4486-928b-29f00380ef52\") " pod="openshift-ingress/e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c"
Sep 18 13:56:19.004894 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:19.004810    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62972a5e-7296-4486-928b-29f00380ef52-workload-certs\") pod \"e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c\" (UID: \"62972a5e-7296-4486-928b-29f00380ef52\") " pod="openshift-ingress/e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c"
Sep 18 13:56:19.004894 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:19.004867    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62972a5e-7296-4486-928b-29f00380ef52-istio-data\") pod \"e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c\" (UID: \"62972a5e-7296-4486-928b-29f00380ef52\") " pod="openshift-ingress/e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c"
Sep 18 13:56:19.004967 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:19.004903    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62972a5e-7296-4486-928b-29f00380ef52-istio-token\") pod \"e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c\" (UID: \"62972a5e-7296-4486-928b-29f00380ef52\") " pod="openshift-ingress/e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c"
Sep 18 13:56:19.004967 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:19.004919    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvgpt\" (UniqueName: \"kubernetes.io/projected/62972a5e-7296-4486-928b-29f00380ef52-kube-api-access-jvgpt\") pod \"e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c\" (UID: \"62972a5e-7296-4486-928b-29f00380ef52\") " pod="openshift-ingress/e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c"
Sep 18 13:56:19.004967 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:19.004937    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62972a5e-7296-4486-928b-29f00380ef52-istiod-ca-cert\") pod \"e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c\" (UID: \"62972a5e-7296-4486-928b-29f00380ef52\") " pod="openshift-ingress/e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c"
Sep 18 13:56:19.004967 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:19.004955    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62972a5e-7296-4486-928b-29f00380ef52-workload-socket\") pod \"e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c\" (UID: \"62972a5e-7296-4486-928b-29f00380ef52\") " pod="openshift-ingress/e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c"
Sep 18 13:56:19.005111 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:19.005032    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62972a5e-7296-4486-928b-29f00380ef52-credential-socket\") pod \"e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c\" (UID: \"62972a5e-7296-4486-928b-29f00380ef52\") " pod="openshift-ingress/e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c"
Sep 18 13:56:19.005111 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:19.005056    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62972a5e-7296-4486-928b-29f00380ef52-istio-podinfo\") pod \"e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c\" (UID: \"62972a5e-7296-4486-928b-29f00380ef52\") " pod="openshift-ingress/e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c"
Sep 18 13:56:19.105816 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:19.105776    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62972a5e-7296-4486-928b-29f00380ef52-credential-socket\") pod \"e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c\" (UID: \"62972a5e-7296-4486-928b-29f00380ef52\") " pod="openshift-ingress/e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c"
Sep 18 13:56:19.106036 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:19.105810    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62972a5e-7296-4486-928b-29f00380ef52-istio-podinfo\") pod \"e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c\" (UID: \"62972a5e-7296-4486-928b-29f00380ef52\") " pod="openshift-ingress/e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c"
Sep 18 13:56:19.106036 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:19.105892    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62972a5e-7296-4486-928b-29f00380ef52-istio-envoy\") pod \"e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c\" (UID: \"62972a5e-7296-4486-928b-29f00380ef52\") " pod="openshift-ingress/e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c"
Sep 18 13:56:19.106036 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:19.105931    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62972a5e-7296-4486-928b-29f00380ef52-workload-certs\") pod \"e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c\" (UID: \"62972a5e-7296-4486-928b-29f00380ef52\") " pod="openshift-ingress/e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c"
Sep 18 13:56:19.106036 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:19.105961    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62972a5e-7296-4486-928b-29f00380ef52-istio-data\") pod \"e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c\" (UID: \"62972a5e-7296-4486-928b-29f00380ef52\") " pod="openshift-ingress/e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c"
Sep 18 13:56:19.106036 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:19.106008    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62972a5e-7296-4486-928b-29f00380ef52-istio-token\") pod \"e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c\" (UID: \"62972a5e-7296-4486-928b-29f00380ef52\") " pod="openshift-ingress/e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c"
Sep 18 13:56:19.106036 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:19.106031    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvgpt\" (UniqueName: \"kubernetes.io/projected/62972a5e-7296-4486-928b-29f00380ef52-kube-api-access-jvgpt\") pod \"e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c\" (UID: \"62972a5e-7296-4486-928b-29f00380ef52\") " pod="openshift-ingress/e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c"
Sep 18 13:56:19.106372 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:19.106199    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62972a5e-7296-4486-928b-29f00380ef52-credential-socket\") pod \"e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c\" (UID: \"62972a5e-7296-4486-928b-29f00380ef52\") " pod="openshift-ingress/e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c"
Sep 18 13:56:19.106372 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:19.106254    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62972a5e-7296-4486-928b-29f00380ef52-istiod-ca-cert\") pod \"e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c\" (UID: \"62972a5e-7296-4486-928b-29f00380ef52\") " pod="openshift-ingress/e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c"
Sep 18 13:56:19.106372 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:19.106287    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62972a5e-7296-4486-928b-29f00380ef52-workload-socket\") pod \"e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c\" (UID: \"62972a5e-7296-4486-928b-29f00380ef52\") " pod="openshift-ingress/e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c"
Sep 18 13:56:19.106540 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:19.106396    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62972a5e-7296-4486-928b-29f00380ef52-istio-data\") pod \"e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c\" (UID: \"62972a5e-7296-4486-928b-29f00380ef52\") " pod="openshift-ingress/e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c"
Sep 18 13:56:19.107044 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:19.107017    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62972a5e-7296-4486-928b-29f00380ef52-istiod-ca-cert\") pod \"e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c\" (UID: \"62972a5e-7296-4486-928b-29f00380ef52\") " pod="openshift-ingress/e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c"
Sep 18 13:56:19.107303 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:19.107282    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62972a5e-7296-4486-928b-29f00380ef52-workload-certs\") pod \"e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c\" (UID: \"62972a5e-7296-4486-928b-29f00380ef52\") " pod="openshift-ingress/e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c"
Sep 18 13:56:19.107343 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:19.107285    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62972a5e-7296-4486-928b-29f00380ef52-workload-socket\") pod \"e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c\" (UID: \"62972a5e-7296-4486-928b-29f00380ef52\") " pod="openshift-ingress/e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c"
Sep 18 13:56:19.108381 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:19.108360    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62972a5e-7296-4486-928b-29f00380ef52-istio-envoy\") pod \"e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c\" (UID: \"62972a5e-7296-4486-928b-29f00380ef52\") " pod="openshift-ingress/e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c"
Sep 18 13:56:19.108711 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:19.108693    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62972a5e-7296-4486-928b-29f00380ef52-istio-podinfo\") pod \"e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c\" (UID: \"62972a5e-7296-4486-928b-29f00380ef52\") " pod="openshift-ingress/e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c"
Sep 18 13:56:19.114596 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:19.114573    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62972a5e-7296-4486-928b-29f00380ef52-istio-token\") pod \"e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c\" (UID: \"62972a5e-7296-4486-928b-29f00380ef52\") " pod="openshift-ingress/e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c"
Sep 18 13:56:19.114703 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:19.114639    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvgpt\" (UniqueName: \"kubernetes.io/projected/62972a5e-7296-4486-928b-29f00380ef52-kube-api-access-jvgpt\") pod \"e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c\" (UID: \"62972a5e-7296-4486-928b-29f00380ef52\") " pod="openshift-ingress/e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c"
Sep 18 13:56:19.262688 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:19.262607    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c"
Sep 18 13:56:19.555635 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:19.555430    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c"]
Sep 18 13:56:19.558174 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:56:19.558149    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62972a5e_7296_4486_928b_29f00380ef52.slice/crio-3c15f8303ed621b3eeafaf03ab3057990525c78cee3bc3b548437d04444cac20 WatchSource:0}: Error finding container 3c15f8303ed621b3eeafaf03ab3057990525c78cee3bc3b548437d04444cac20: Status 404 returned error can't find the container with id 3c15f8303ed621b3eeafaf03ab3057990525c78cee3bc3b548437d04444cac20
Sep 18 13:56:19.600362 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:19.600338    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c" event={"ID":"62972a5e-7296-4486-928b-29f00380ef52","Type":"ContainerStarted","Data":"3c15f8303ed621b3eeafaf03ab3057990525c78cee3bc3b548437d04444cac20"}
Sep 18 13:56:19.600683 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:19.600392    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7636-50821c" podUID="d13cb3f8-13f6-4976-8bce-25b64f9ce2c1" containerName="test-curl-7636-50821c" containerID="cri-o://86ab2aaa68962b839a833da6a3a9e55b9586fda34f6437052546ac451387be5c" gracePeriod=2
Sep 18 13:56:19.745053 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:19.745028    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7636-a4b2bb"
Sep 18 13:56:19.812462 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:19.812378    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2g4dp\" (UniqueName: \"kubernetes.io/projected/133619cc-1c31-4586-8b11-2c9ddb988180-kube-api-access-2g4dp\") pod \"133619cc-1c31-4586-8b11-2c9ddb988180\" (UID: \"133619cc-1c31-4586-8b11-2c9ddb988180\") "
Sep 18 13:56:19.814493 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:19.814470    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/133619cc-1c31-4586-8b11-2c9ddb988180-kube-api-access-2g4dp" (OuterVolumeSpecName: "kube-api-access-2g4dp") pod "133619cc-1c31-4586-8b11-2c9ddb988180" (UID: "133619cc-1c31-4586-8b11-2c9ddb988180"). InnerVolumeSpecName "kube-api-access-2g4dp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:56:19.845011 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:19.844979    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:19.913670 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:19.913637    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2g4dp\" (UniqueName: \"kubernetes.io/projected/133619cc-1c31-4586-8b11-2c9ddb988180-kube-api-access-2g4dp\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:56:19.952634 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:19.952604    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:20.046270 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:20.046233    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:20.168007 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:20.167952    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:20.486856 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:20.486756    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:20.608498 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:20.608460    2577 generic.go:356] "Generic (PLEG): container finished" podID="133619cc-1c31-4586-8b11-2c9ddb988180" containerID="0136804dfb455c8e25840c6009bf123086525fc829f2cb6835d66239fd7a7a96" exitCode=137
Sep 18 13:56:20.609000 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:20.608525    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7636-a4b2bb"
Sep 18 13:56:20.609000 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:20.608557    2577 scope.go:117] "RemoveContainer" containerID="0136804dfb455c8e25840c6009bf123086525fc829f2cb6835d66239fd7a7a96"
Sep 18 13:56:20.616854 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:20.616809    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:20.619618 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:20.619584    2577 scope.go:117] "RemoveContainer" containerID="0136804dfb455c8e25840c6009bf123086525fc829f2cb6835d66239fd7a7a96"
Sep 18 13:56:20.619979 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:56:20.619931    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0136804dfb455c8e25840c6009bf123086525fc829f2cb6835d66239fd7a7a96\": container with ID starting with 0136804dfb455c8e25840c6009bf123086525fc829f2cb6835d66239fd7a7a96 not found: ID does not exist" containerID="0136804dfb455c8e25840c6009bf123086525fc829f2cb6835d66239fd7a7a96"
Sep 18 13:56:20.620087 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:20.619968    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0136804dfb455c8e25840c6009bf123086525fc829f2cb6835d66239fd7a7a96"} err="failed to get container status \"0136804dfb455c8e25840c6009bf123086525fc829f2cb6835d66239fd7a7a96\": rpc error: code = NotFound desc = could not find container \"0136804dfb455c8e25840c6009bf123086525fc829f2cb6835d66239fd7a7a96\": container with ID starting with 0136804dfb455c8e25840c6009bf123086525fc829f2cb6835d66239fd7a7a96 not found: ID does not exist"
Sep 18 13:56:20.623766 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:56:20.623740    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7636-a4b2bb\" is forbidden: User \"system:node:ip-10-0-141-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-222.ec2.internal' and this object" podUID="133619cc-1c31-4586-8b11-2c9ddb988180" pod="opendatahub/test-curl-7636-a4b2bb"
Sep 18 13:56:20.800984 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:20.800913    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:20.891944 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:20.891910    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:21.109274 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:21.109242    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="133619cc-1c31-4586-8b11-2c9ddb988180" path="/var/lib/kubelet/pods/133619cc-1c31-4586-8b11-2c9ddb988180/volumes"
Sep 18 13:56:21.146279 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:21.146237    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:21.843437 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:21.843403    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:22.289298 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:22.289249    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 13:56:22.289418 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:22.289356    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 13:56:22.289418 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:22.289398    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 13:56:22.369775 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:22.369747    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7636-50821c"
Sep 18 13:56:22.435431 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:22.435400    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6w6ww\" (UniqueName: \"kubernetes.io/projected/d13cb3f8-13f6-4976-8bce-25b64f9ce2c1-kube-api-access-6w6ww\") pod \"d13cb3f8-13f6-4976-8bce-25b64f9ce2c1\" (UID: \"d13cb3f8-13f6-4976-8bce-25b64f9ce2c1\") "
Sep 18 13:56:22.437515 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:22.437478    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d13cb3f8-13f6-4976-8bce-25b64f9ce2c1-kube-api-access-6w6ww" (OuterVolumeSpecName: "kube-api-access-6w6ww") pod "d13cb3f8-13f6-4976-8bce-25b64f9ce2c1" (UID: "d13cb3f8-13f6-4976-8bce-25b64f9ce2c1"). InnerVolumeSpecName "kube-api-access-6w6ww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:56:22.536318 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:22.536232    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6w6ww\" (UniqueName: \"kubernetes.io/projected/d13cb3f8-13f6-4976-8bce-25b64f9ce2c1-kube-api-access-6w6ww\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:56:22.623300 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:22.623264    2577 generic.go:356] "Generic (PLEG): container finished" podID="d13cb3f8-13f6-4976-8bce-25b64f9ce2c1" containerID="86ab2aaa68962b839a833da6a3a9e55b9586fda34f6437052546ac451387be5c" exitCode=137
Sep 18 13:56:22.623495 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:22.623316    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7636-50821c"
Sep 18 13:56:22.623495 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:22.623385    2577 scope.go:117] "RemoveContainer" containerID="86ab2aaa68962b839a833da6a3a9e55b9586fda34f6437052546ac451387be5c"
Sep 18 13:56:22.625855 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:22.625810    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c" event={"ID":"62972a5e-7296-4486-928b-29f00380ef52","Type":"ContainerStarted","Data":"8e684556d7d790083aca3b8cbc784dda56d26cdd45d309249f95b4d9b7fa8ba3"}
Sep 18 13:56:22.634583 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:22.634563    2577 scope.go:117] "RemoveContainer" containerID="86ab2aaa68962b839a833da6a3a9e55b9586fda34f6437052546ac451387be5c"
Sep 18 13:56:22.634911 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:56:22.634881    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86ab2aaa68962b839a833da6a3a9e55b9586fda34f6437052546ac451387be5c\": container with ID starting with 86ab2aaa68962b839a833da6a3a9e55b9586fda34f6437052546ac451387be5c not found: ID does not exist" containerID="86ab2aaa68962b839a833da6a3a9e55b9586fda34f6437052546ac451387be5c"
Sep 18 13:56:22.634911 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:22.634921    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86ab2aaa68962b839a833da6a3a9e55b9586fda34f6437052546ac451387be5c"} err="failed to get container status \"86ab2aaa68962b839a833da6a3a9e55b9586fda34f6437052546ac451387be5c\": rpc error: code = NotFound desc = could not find container \"86ab2aaa68962b839a833da6a3a9e55b9586fda34f6437052546ac451387be5c\": container with ID starting with 86ab2aaa68962b839a833da6a3a9e55b9586fda34f6437052546ac451387be5c not found: ID does not exist"
Sep 18 13:56:22.654510 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:56:22.654470    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7636-50821c\" is forbidden: User \"system:node:ip-10-0-141-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-222.ec2.internal' and this object" podUID="d13cb3f8-13f6-4976-8bce-25b64f9ce2c1" pod="opendatahub/test-curl-7636-50821c"
Sep 18 13:56:22.655178 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:22.655139    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c" podStartSLOduration=1.926676694 podStartE2EDuration="4.655121444s" totalImagesPullingTime="2.72844475s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:56:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:56:22.652803684 +0000 UTC m=+731.705463023" watchObservedRunningTime="2026-09-18 13:56:22.655121444 +0000 UTC m=+731.707780770"
Sep 18 13:56:22.772263 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:22.772227    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:22.832251 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:22.832172    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:22.998182 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:22.998141    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:23.105788 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:56:23.105741    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7636-50821c\" is forbidden: User \"system:node:ip-10-0-141-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-222.ec2.internal' and this object" podUID="d13cb3f8-13f6-4976-8bce-25b64f9ce2c1" pod="opendatahub/test-curl-7636-50821c"
Sep 18 13:56:23.108013 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:23.107986    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d13cb3f8-13f6-4976-8bce-25b64f9ce2c1" path="/var/lib/kubelet/pods/d13cb3f8-13f6-4976-8bce-25b64f9ce2c1/volumes"
Sep 18 13:56:23.265143 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:23.265105    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c"
Sep 18 13:56:23.270172 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:23.270142    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c"
Sep 18 13:56:23.365199 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:23.365122    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:23.466067 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:23.466034    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:23.550676 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:23.550633    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:23.575905 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:23.575873    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ed4c53-59f4997598-7jfng"]
Sep 18 13:56:23.580611 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:23.580592    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ed4c53-59f4997598-7jfng"
Sep 18 13:56:23.583329 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:23.583307    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-ed4c53-dockercfg-xr44d\""
Sep 18 13:56:23.583431 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:23.583307    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-ed4c53\""
Sep 18 13:56:23.607294 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:23.607263    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ed4c53-59f4997598-7jfng"]
Sep 18 13:56:23.634315 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:23.634239    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c"
Sep 18 13:56:23.635378 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:23.635357    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c"
Sep 18 13:56:23.646141 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:23.646102    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8psgl\" (UniqueName: \"kubernetes.io/projected/9d4b52c6-fb44-431d-adff-446bbd7860ea-kube-api-access-8psgl\") pod \"payload-pre-processing-e2e-worker-w0-ed4c53-59f4997598-7jfng\" (UID: \"9d4b52c6-fb44-431d-adff-446bbd7860ea\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ed4c53-59f4997598-7jfng"
Sep 18 13:56:23.646307 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:23.646220    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d4b52c6-fb44-431d-adff-446bbd7860ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-ed4c53-59f4997598-7jfng\" (UID: \"9d4b52c6-fb44-431d-adff-446bbd7860ea\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ed4c53-59f4997598-7jfng"
Sep 18 13:56:23.747283 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:23.747248    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d4b52c6-fb44-431d-adff-446bbd7860ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-ed4c53-59f4997598-7jfng\" (UID: \"9d4b52c6-fb44-431d-adff-446bbd7860ea\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ed4c53-59f4997598-7jfng"
Sep 18 13:56:23.747487 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:23.747351    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8psgl\" (UniqueName: \"kubernetes.io/projected/9d4b52c6-fb44-431d-adff-446bbd7860ea-kube-api-access-8psgl\") pod \"payload-pre-processing-e2e-worker-w0-ed4c53-59f4997598-7jfng\" (UID: \"9d4b52c6-fb44-431d-adff-446bbd7860ea\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ed4c53-59f4997598-7jfng"
Sep 18 13:56:23.748221 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:23.748200    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d4b52c6-fb44-431d-adff-446bbd7860ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-ed4c53-59f4997598-7jfng\" (UID: \"9d4b52c6-fb44-431d-adff-446bbd7860ea\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ed4c53-59f4997598-7jfng"
Sep 18 13:56:23.766308 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:23.766276    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8psgl\" (UniqueName: \"kubernetes.io/projected/9d4b52c6-fb44-431d-adff-446bbd7860ea-kube-api-access-8psgl\") pod \"payload-pre-processing-e2e-worker-w0-ed4c53-59f4997598-7jfng\" (UID: \"9d4b52c6-fb44-431d-adff-446bbd7860ea\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ed4c53-59f4997598-7jfng"
Sep 18 13:56:23.890251 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:23.890162    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ed4c53-59f4997598-7jfng"
Sep 18 13:56:24.167609 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:24.167582    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ed4c53-59f4997598-7jfng"]
Sep 18 13:56:24.169934 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:56:24.169904    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d4b52c6_fb44_431d_adff_446bbd7860ea.slice/crio-f9e62d780f6bffc4b1f6a4702952f0f648242a06c25a99c1dcca4dcb7d9c608d WatchSource:0}: Error finding container f9e62d780f6bffc4b1f6a4702952f0f648242a06c25a99c1dcca4dcb7d9c608d: Status 404 returned error can't find the container with id f9e62d780f6bffc4b1f6a4702952f0f648242a06c25a99c1dcca4dcb7d9c608d
Sep 18 13:56:24.190746 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:24.190692    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:24.642338 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:24.642296    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ed4c53-59f4997598-7jfng" event={"ID":"9d4b52c6-fb44-431d-adff-446bbd7860ea","Type":"ContainerStarted","Data":"f9e62d780f6bffc4b1f6a4702952f0f648242a06c25a99c1dcca4dcb7d9c608d"}
Sep 18 13:56:24.774537 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:24.774496    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:24.885352 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:24.885325    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:25.238082 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:25.238045    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:25.649818 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:25.649782    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ed4c53-59f4997598-7jfng" event={"ID":"9d4b52c6-fb44-431d-adff-446bbd7860ea","Type":"ContainerStarted","Data":"0f9eec2be2f35acc1aa783d33c02ce1706763410d406cf88a87b7d32d54c4ffe"}
Sep 18 13:56:25.650081 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:25.650063    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ed4c53-59f4997598-7jfng"
Sep 18 13:56:25.665877 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:25.665810    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ed4c53-59f4997598-7jfng" podStartSLOduration=1.530798498 podStartE2EDuration="2.665796035s" totalImagesPullingTime="1.134997537s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:56:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:56:25.66495537 +0000 UTC m=+734.717614697" watchObservedRunningTime="2026-09-18 13:56:25.665796035 +0000 UTC m=+734.718455363"
Sep 18 13:56:26.143513 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:26.143477    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:27.111014 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:27.110969    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:27.203092 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:27.203061    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:27.471867 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:27.471756    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:27.596986 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:27.596949    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:27.694000 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:27.693963    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:27.700002 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:27.699978    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-70f06e-8d6b74974-qshdd"]
Sep 18 13:56:27.703604 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:27.703587    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-70f06e-8d6b74974-qshdd"
Sep 18 13:56:27.706022 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:27.705997    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-70f06e-dockercfg-np484\""
Sep 18 13:56:27.706111 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:27.706010    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-70f06e\""
Sep 18 13:56:27.720857 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:27.720812    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-70f06e-8d6b74974-qshdd"]
Sep 18 13:56:27.743712 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:27.743642    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-70f06e-869f668c77-z9mx8"]
Sep 18 13:56:27.747004 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:27.746986    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-70f06e-869f668c77-z9mx8"
Sep 18 13:56:27.754702 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:27.754674    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-70f06e-869f668c77-z9mx8"]
Sep 18 13:56:27.779661 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:27.779630    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc9zn\" (UniqueName: \"kubernetes.io/projected/1730d85c-df98-4db2-99f9-951845ebbc56-kube-api-access-lc9zn\") pod \"payload-pre-processing-e2e-autores-70f06e-8d6b74974-qshdd\" (UID: \"1730d85c-df98-4db2-99f9-951845ebbc56\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-70f06e-8d6b74974-qshdd"
Sep 18 13:56:27.779798 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:27.779667    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7jg7\" (UniqueName: \"kubernetes.io/projected/b07a50e5-19e8-4821-bfa1-225d7160c9cc-kube-api-access-d7jg7\") pod \"payload-processing-e2e-autores-70f06e-869f668c77-z9mx8\" (UID: \"b07a50e5-19e8-4821-bfa1-225d7160c9cc\") " pod="openshift-ingress/payload-processing-e2e-autores-70f06e-869f668c77-z9mx8"
Sep 18 13:56:27.779798 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:27.779706    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1730d85c-df98-4db2-99f9-951845ebbc56-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-70f06e-8d6b74974-qshdd\" (UID: \"1730d85c-df98-4db2-99f9-951845ebbc56\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-70f06e-8d6b74974-qshdd"
Sep 18 13:56:27.779798 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:27.779772    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b07a50e5-19e8-4821-bfa1-225d7160c9cc-plugins-config-volume\") pod \"payload-processing-e2e-autores-70f06e-869f668c77-z9mx8\" (UID: \"b07a50e5-19e8-4821-bfa1-225d7160c9cc\") " pod="openshift-ingress/payload-processing-e2e-autores-70f06e-869f668c77-z9mx8"
Sep 18 13:56:27.880614 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:27.880578    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc9zn\" (UniqueName: \"kubernetes.io/projected/1730d85c-df98-4db2-99f9-951845ebbc56-kube-api-access-lc9zn\") pod \"payload-pre-processing-e2e-autores-70f06e-8d6b74974-qshdd\" (UID: \"1730d85c-df98-4db2-99f9-951845ebbc56\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-70f06e-8d6b74974-qshdd"
Sep 18 13:56:27.880614 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:27.880616    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7jg7\" (UniqueName: \"kubernetes.io/projected/b07a50e5-19e8-4821-bfa1-225d7160c9cc-kube-api-access-d7jg7\") pod \"payload-processing-e2e-autores-70f06e-869f668c77-z9mx8\" (UID: \"b07a50e5-19e8-4821-bfa1-225d7160c9cc\") " pod="openshift-ingress/payload-processing-e2e-autores-70f06e-869f668c77-z9mx8"
Sep 18 13:56:27.880896 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:27.880655    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1730d85c-df98-4db2-99f9-951845ebbc56-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-70f06e-8d6b74974-qshdd\" (UID: \"1730d85c-df98-4db2-99f9-951845ebbc56\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-70f06e-8d6b74974-qshdd"
Sep 18 13:56:27.880896 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:27.880687    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b07a50e5-19e8-4821-bfa1-225d7160c9cc-plugins-config-volume\") pod \"payload-processing-e2e-autores-70f06e-869f668c77-z9mx8\" (UID: \"b07a50e5-19e8-4821-bfa1-225d7160c9cc\") " pod="openshift-ingress/payload-processing-e2e-autores-70f06e-869f668c77-z9mx8"
Sep 18 13:56:27.881475 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:27.881453    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b07a50e5-19e8-4821-bfa1-225d7160c9cc-plugins-config-volume\") pod \"payload-processing-e2e-autores-70f06e-869f668c77-z9mx8\" (UID: \"b07a50e5-19e8-4821-bfa1-225d7160c9cc\") " pod="openshift-ingress/payload-processing-e2e-autores-70f06e-869f668c77-z9mx8"
Sep 18 13:56:27.881516 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:27.881457    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1730d85c-df98-4db2-99f9-951845ebbc56-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-70f06e-8d6b74974-qshdd\" (UID: \"1730d85c-df98-4db2-99f9-951845ebbc56\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-70f06e-8d6b74974-qshdd"
Sep 18 13:56:27.890025 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:27.889997    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7jg7\" (UniqueName: \"kubernetes.io/projected/b07a50e5-19e8-4821-bfa1-225d7160c9cc-kube-api-access-d7jg7\") pod \"payload-processing-e2e-autores-70f06e-869f668c77-z9mx8\" (UID: \"b07a50e5-19e8-4821-bfa1-225d7160c9cc\") " pod="openshift-ingress/payload-processing-e2e-autores-70f06e-869f668c77-z9mx8"
Sep 18 13:56:27.890306 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:27.890287    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc9zn\" (UniqueName: \"kubernetes.io/projected/1730d85c-df98-4db2-99f9-951845ebbc56-kube-api-access-lc9zn\") pod \"payload-pre-processing-e2e-autores-70f06e-8d6b74974-qshdd\" (UID: \"1730d85c-df98-4db2-99f9-951845ebbc56\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-70f06e-8d6b74974-qshdd"
Sep 18 13:56:28.013779 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:28.013693    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-70f06e-8d6b74974-qshdd"
Sep 18 13:56:28.057797 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:28.057753    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-70f06e-869f668c77-z9mx8"
Sep 18 13:56:28.147964 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:28.147936    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:28.286030 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:28.285950    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-70f06e-8d6b74974-qshdd"]
Sep 18 13:56:28.288733 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:56:28.288701    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1730d85c_df98_4db2_99f9_951845ebbc56.slice/crio-c8b92f91e7ead6ad824860f6e5b4e4d042c0473bd973be66969bf50af3477b8e WatchSource:0}: Error finding container c8b92f91e7ead6ad824860f6e5b4e4d042c0473bd973be66969bf50af3477b8e: Status 404 returned error can't find the container with id c8b92f91e7ead6ad824860f6e5b4e4d042c0473bd973be66969bf50af3477b8e
Sep 18 13:56:28.350076 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:28.350047    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-70f06e-869f668c77-z9mx8"]
Sep 18 13:56:28.350852 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:56:28.350816    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb07a50e5_19e8_4821_bfa1_225d7160c9cc.slice/crio-91c4754da326ade58a6b5eec7f042d17fff8a7a1a3b0d11cccc86c4ef5851cc8 WatchSource:0}: Error finding container 91c4754da326ade58a6b5eec7f042d17fff8a7a1a3b0d11cccc86c4ef5851cc8: Status 404 returned error can't find the container with id 91c4754da326ade58a6b5eec7f042d17fff8a7a1a3b0d11cccc86c4ef5851cc8
Sep 18 13:56:28.625359 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:28.625320    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:28.666898 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:28.666858    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-70f06e-869f668c77-z9mx8" event={"ID":"b07a50e5-19e8-4821-bfa1-225d7160c9cc","Type":"ContainerStarted","Data":"ba0b33db6de1b2ff9c74adb32575fe8bd0b574ae67d9b5510b06c63cdb53ca0a"}
Sep 18 13:56:28.667089 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:28.666929    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-70f06e-869f668c77-z9mx8" event={"ID":"b07a50e5-19e8-4821-bfa1-225d7160c9cc","Type":"ContainerStarted","Data":"91c4754da326ade58a6b5eec7f042d17fff8a7a1a3b0d11cccc86c4ef5851cc8"}
Sep 18 13:56:28.667089 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:28.666960    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-70f06e-869f668c77-z9mx8"
Sep 18 13:56:28.668559 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:28.668535    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-70f06e-8d6b74974-qshdd" event={"ID":"1730d85c-df98-4db2-99f9-951845ebbc56","Type":"ContainerStarted","Data":"fd708f8494d708935f9b1b11b1350404f357b567aaf9d4e37063d887aec0907a"}
Sep 18 13:56:28.668670 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:28.668574    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-70f06e-8d6b74974-qshdd" event={"ID":"1730d85c-df98-4db2-99f9-951845ebbc56","Type":"ContainerStarted","Data":"c8b92f91e7ead6ad824860f6e5b4e4d042c0473bd973be66969bf50af3477b8e"}
Sep 18 13:56:28.668670 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:28.668622    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-70f06e-8d6b74974-qshdd"
Sep 18 13:56:28.685551 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:28.685487    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-70f06e-869f668c77-z9mx8" podStartSLOduration=1.6854485179999998 podStartE2EDuration="1.685448518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:56:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:56:28.683529542 +0000 UTC m=+737.736188870" watchObservedRunningTime="2026-09-18 13:56:28.685448518 +0000 UTC m=+737.738107846"
Sep 18 13:56:28.704277 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:28.704225    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-70f06e-8d6b74974-qshdd" podStartSLOduration=1.7042078059999999 podStartE2EDuration="1.704207806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:56:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:56:28.703089794 +0000 UTC m=+737.755749120" watchObservedRunningTime="2026-09-18 13:56:28.704207806 +0000 UTC m=+737.756867144"
Sep 18 13:56:28.708679 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:28.708650    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:28.902070 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:28.901997    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e095dac8-ff577f99-lggh8"]
Sep 18 13:56:28.905556 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:28.905535    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e095dac8-ff577f99-lggh8"
Sep 18 13:56:28.908202 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:28.907634    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-e095dac8-serving-cert\""
Sep 18 13:56:28.908202 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:28.907724    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 13:56:28.908202 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:28.907737    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 18 13:56:28.908202 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:28.908022    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vx9xd\""
Sep 18 13:56:28.908202 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:28.908053    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 18 13:56:28.917794 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:28.917766    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e095dac8-ff577f99-lggh8"]
Sep 18 13:56:28.938936 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:28.938901    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:28.991682 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:28.991646    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjjcv\" (UniqueName: \"kubernetes.io/projected/37dbe38b-965f-4e46-ba31-b40183261aa6-kube-api-access-jjjcv\") pod \"maas-api-e2e-ait-e095dac8-ff577f99-lggh8\" (UID: \"37dbe38b-965f-4e46-ba31-b40183261aa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e095dac8-ff577f99-lggh8"
Sep 18 13:56:28.991906 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:28.991699    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37dbe38b-965f-4e46-ba31-b40183261aa6-maas-api-tls\") pod \"maas-api-e2e-ait-e095dac8-ff577f99-lggh8\" (UID: \"37dbe38b-965f-4e46-ba31-b40183261aa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e095dac8-ff577f99-lggh8"
Sep 18 13:56:28.991906 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:28.991760    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37dbe38b-965f-4e46-ba31-b40183261aa6-openshift-service-ca\") pod \"maas-api-e2e-ait-e095dac8-ff577f99-lggh8\" (UID: \"37dbe38b-965f-4e46-ba31-b40183261aa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e095dac8-ff577f99-lggh8"
Sep 18 13:56:28.991906 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:28.991851    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37dbe38b-965f-4e46-ba31-b40183261aa6-metrics-certs\") pod \"maas-api-e2e-ait-e095dac8-ff577f99-lggh8\" (UID: \"37dbe38b-965f-4e46-ba31-b40183261aa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e095dac8-ff577f99-lggh8"
Sep 18 13:56:29.000672 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:29.000643    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:29.086966 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:29.086927    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:29.093048 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:29.093016    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37dbe38b-965f-4e46-ba31-b40183261aa6-metrics-certs\") pod \"maas-api-e2e-ait-e095dac8-ff577f99-lggh8\" (UID: \"37dbe38b-965f-4e46-ba31-b40183261aa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e095dac8-ff577f99-lggh8"
Sep 18 13:56:29.093203 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:29.093078    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjjcv\" (UniqueName: \"kubernetes.io/projected/37dbe38b-965f-4e46-ba31-b40183261aa6-kube-api-access-jjjcv\") pod \"maas-api-e2e-ait-e095dac8-ff577f99-lggh8\" (UID: \"37dbe38b-965f-4e46-ba31-b40183261aa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e095dac8-ff577f99-lggh8"
Sep 18 13:56:29.093203 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:29.093133    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37dbe38b-965f-4e46-ba31-b40183261aa6-maas-api-tls\") pod \"maas-api-e2e-ait-e095dac8-ff577f99-lggh8\" (UID: \"37dbe38b-965f-4e46-ba31-b40183261aa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e095dac8-ff577f99-lggh8"
Sep 18 13:56:29.093203 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:29.093193    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37dbe38b-965f-4e46-ba31-b40183261aa6-openshift-service-ca\") pod \"maas-api-e2e-ait-e095dac8-ff577f99-lggh8\" (UID: \"37dbe38b-965f-4e46-ba31-b40183261aa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e095dac8-ff577f99-lggh8"
Sep 18 13:56:29.094047 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:29.094025    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37dbe38b-965f-4e46-ba31-b40183261aa6-openshift-service-ca\") pod \"maas-api-e2e-ait-e095dac8-ff577f99-lggh8\" (UID: \"37dbe38b-965f-4e46-ba31-b40183261aa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e095dac8-ff577f99-lggh8"
Sep 18 13:56:29.096041 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:29.096015    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37dbe38b-965f-4e46-ba31-b40183261aa6-metrics-certs\") pod \"maas-api-e2e-ait-e095dac8-ff577f99-lggh8\" (UID: \"37dbe38b-965f-4e46-ba31-b40183261aa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e095dac8-ff577f99-lggh8"
Sep 18 13:56:29.096165 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:29.096127    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37dbe38b-965f-4e46-ba31-b40183261aa6-maas-api-tls\") pod \"maas-api-e2e-ait-e095dac8-ff577f99-lggh8\" (UID: \"37dbe38b-965f-4e46-ba31-b40183261aa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e095dac8-ff577f99-lggh8"
Sep 18 13:56:29.102995 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:29.102971    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjjcv\" (UniqueName: \"kubernetes.io/projected/37dbe38b-965f-4e46-ba31-b40183261aa6-kube-api-access-jjjcv\") pod \"maas-api-e2e-ait-e095dac8-ff577f99-lggh8\" (UID: \"37dbe38b-965f-4e46-ba31-b40183261aa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e095dac8-ff577f99-lggh8"
Sep 18 13:56:29.197308 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:29.197231    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:29.220410 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:29.220377    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e095dac8-ff577f99-lggh8"
Sep 18 13:56:29.310097 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:29.310062    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:29.410567 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:29.410534    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:29.498015 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:29.497984    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e095dac8-ff577f99-lggh8"]
Sep 18 13:56:29.499162 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:56:29.499135    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37dbe38b_965f_4e46_ba31_b40183261aa6.slice/crio-80265ca9b2ae8832d4917c6ffb02014a8279d4955d885c917f58634397230d96 WatchSource:0}: Error finding container 80265ca9b2ae8832d4917c6ffb02014a8279d4955d885c917f58634397230d96: Status 404 returned error can't find the container with id 80265ca9b2ae8832d4917c6ffb02014a8279d4955d885c917f58634397230d96
Sep 18 13:56:29.613381 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:29.613347    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:29.675921 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:29.675887    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e095dac8-ff577f99-lggh8" event={"ID":"37dbe38b-965f-4e46-ba31-b40183261aa6","Type":"ContainerStarted","Data":"80265ca9b2ae8832d4917c6ffb02014a8279d4955d885c917f58634397230d96"}
Sep 18 13:56:29.681930 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:29.681904    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:30.140472 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:30.140409    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:30.834617 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:30.834580    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:31.634339 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:31.634305    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:31.833939 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:31.833898    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:31.984846 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:31.984782    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:32.078759 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:32.078715    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:32.176387 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:32.176301    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:32.288210 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:32.288154    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:32.695284 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:32.695247    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e095dac8-ff577f99-lggh8" event={"ID":"37dbe38b-965f-4e46-ba31-b40183261aa6","Type":"ContainerStarted","Data":"92e0b8f993102211c720d11d88c451c181db31f69a83d339ddbea48da9963c48"}
Sep 18 13:56:32.695466 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:32.695402    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e095dac8-ff577f99-lggh8"
Sep 18 13:56:32.735212 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:32.735156    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e095dac8-ff577f99-lggh8" podStartSLOduration=2.392737276 podStartE2EDuration="4.735138243s" totalImagesPullingTime="2.342400967s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:56:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:56:32.734591914 +0000 UTC m=+741.787251242" watchObservedRunningTime="2026-09-18 13:56:32.735138243 +0000 UTC m=+741.787797569"
Sep 18 13:56:32.751812 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:32.751765    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:33.455613 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:33.455574    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:34.249618 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:34.249586    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:34.351651 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:34.351621    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:34.724585 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:34.724552    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:34.822707 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:34.822666    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:34.842895 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:34.842865    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b63f26-65bd777cc4-z8cfp"]
Sep 18 13:56:34.846391 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:34.846373    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b63f26-65bd777cc4-z8cfp"
Sep 18 13:56:34.848398 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:34.848375    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-b63f26\""
Sep 18 13:56:34.855615 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:34.855595    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b63f26-65bd777cc4-z8cfp"]
Sep 18 13:56:34.923023 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:34.922983    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:34.949079 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:34.949040    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d080cb6-7e9f-4bf3-82ca-acb9b838f943-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b63f26-65bd777cc4-z8cfp\" (UID: \"2d080cb6-7e9f-4bf3-82ca-acb9b838f943\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b63f26-65bd777cc4-z8cfp"
Sep 18 13:56:34.949246 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:34.949097    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgkj5\" (UniqueName: \"kubernetes.io/projected/2d080cb6-7e9f-4bf3-82ca-acb9b838f943-kube-api-access-tgkj5\") pod \"payload-pre-processing-e2e-inf-b-b63f26-65bd777cc4-z8cfp\" (UID: \"2d080cb6-7e9f-4bf3-82ca-acb9b838f943\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b63f26-65bd777cc4-z8cfp"
Sep 18 13:56:34.996119 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:34.996043    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b63f26-fd596bb8f-fhz8b"]
Sep 18 13:56:34.999418 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:34.999402    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b63f26-fd596bb8f-fhz8b"
Sep 18 13:56:35.001459 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:35.001440    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-b63f26-dockercfg-zmnfm\""
Sep 18 13:56:35.008769 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:35.008559    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b63f26-fd596bb8f-fhz8b"]
Sep 18 13:56:35.050171 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:35.050138    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d080cb6-7e9f-4bf3-82ca-acb9b838f943-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b63f26-65bd777cc4-z8cfp\" (UID: \"2d080cb6-7e9f-4bf3-82ca-acb9b838f943\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b63f26-65bd777cc4-z8cfp"
Sep 18 13:56:35.050377 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:35.050183    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgkj5\" (UniqueName: \"kubernetes.io/projected/2d080cb6-7e9f-4bf3-82ca-acb9b838f943-kube-api-access-tgkj5\") pod \"payload-pre-processing-e2e-inf-b-b63f26-65bd777cc4-z8cfp\" (UID: \"2d080cb6-7e9f-4bf3-82ca-acb9b838f943\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b63f26-65bd777cc4-z8cfp"
Sep 18 13:56:35.050377 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:35.050243    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e06b525-41a0-43ac-a7c9-337a24138e89-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b63f26-fd596bb8f-fhz8b\" (UID: \"7e06b525-41a0-43ac-a7c9-337a24138e89\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b63f26-fd596bb8f-fhz8b"
Sep 18 13:56:35.050377 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:35.050274    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kph7s\" (UniqueName: \"kubernetes.io/projected/7e06b525-41a0-43ac-a7c9-337a24138e89-kube-api-access-kph7s\") pod \"payload-processing-e2e-inf-b-b63f26-fd596bb8f-fhz8b\" (UID: \"7e06b525-41a0-43ac-a7c9-337a24138e89\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b63f26-fd596bb8f-fhz8b"
Sep 18 13:56:35.050881 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:35.050863    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d080cb6-7e9f-4bf3-82ca-acb9b838f943-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b63f26-65bd777cc4-z8cfp\" (UID: \"2d080cb6-7e9f-4bf3-82ca-acb9b838f943\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b63f26-65bd777cc4-z8cfp"
Sep 18 13:56:35.058097 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:35.058069    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgkj5\" (UniqueName: \"kubernetes.io/projected/2d080cb6-7e9f-4bf3-82ca-acb9b838f943-kube-api-access-tgkj5\") pod \"payload-pre-processing-e2e-inf-b-b63f26-65bd777cc4-z8cfp\" (UID: \"2d080cb6-7e9f-4bf3-82ca-acb9b838f943\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b63f26-65bd777cc4-z8cfp"
Sep 18 13:56:35.151652 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:35.151611    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e06b525-41a0-43ac-a7c9-337a24138e89-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b63f26-fd596bb8f-fhz8b\" (UID: \"7e06b525-41a0-43ac-a7c9-337a24138e89\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b63f26-fd596bb8f-fhz8b"
Sep 18 13:56:35.151652 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:35.151652    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kph7s\" (UniqueName: \"kubernetes.io/projected/7e06b525-41a0-43ac-a7c9-337a24138e89-kube-api-access-kph7s\") pod \"payload-processing-e2e-inf-b-b63f26-fd596bb8f-fhz8b\" (UID: \"7e06b525-41a0-43ac-a7c9-337a24138e89\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b63f26-fd596bb8f-fhz8b"
Sep 18 13:56:35.152372 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:35.152350    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e06b525-41a0-43ac-a7c9-337a24138e89-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b63f26-fd596bb8f-fhz8b\" (UID: \"7e06b525-41a0-43ac-a7c9-337a24138e89\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b63f26-fd596bb8f-fhz8b"
Sep 18 13:56:35.153506 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:35.153481    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:35.158078 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:35.158056    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b63f26-65bd777cc4-z8cfp"
Sep 18 13:56:35.159247 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:35.159228    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kph7s\" (UniqueName: \"kubernetes.io/projected/7e06b525-41a0-43ac-a7c9-337a24138e89-kube-api-access-kph7s\") pod \"payload-processing-e2e-inf-b-b63f26-fd596bb8f-fhz8b\" (UID: \"7e06b525-41a0-43ac-a7c9-337a24138e89\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b63f26-fd596bb8f-fhz8b"
Sep 18 13:56:35.313361 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:35.313259    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b63f26-fd596bb8f-fhz8b"
Sep 18 13:56:35.438880 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:35.438853    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b63f26-65bd777cc4-z8cfp"]
Sep 18 13:56:35.440981 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:56:35.440951    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d080cb6_7e9f_4bf3_82ca_acb9b838f943.slice/crio-7455839ade90a8942e1307be7d3de7a76cc57741ceb01d22efece6589d4c0a8f WatchSource:0}: Error finding container 7455839ade90a8942e1307be7d3de7a76cc57741ceb01d22efece6589d4c0a8f: Status 404 returned error can't find the container with id 7455839ade90a8942e1307be7d3de7a76cc57741ceb01d22efece6589d4c0a8f
Sep 18 13:56:35.507438 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:35.507405    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 13:56:35.591679 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:35.591645    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b63f26-fd596bb8f-fhz8b"]
Sep 18 13:56:35.594175 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:56:35.594143    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e06b525_41a0_43ac_a7c9_337a24138e89.slice/crio-c12a3c8a702cd3f9b2a6f52ad4662de19200dee0fb8b0ed734f15970354e56cc WatchSource:0}: Error finding container c12a3c8a702cd3f9b2a6f52ad4662de19200dee0fb8b0ed734f15970354e56cc: Status 404 returned error can't find the container with id c12a3c8a702cd3f9b2a6f52ad4662de19200dee0fb8b0ed734f15970354e56cc
Sep 18 13:56:35.715247 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:35.715205    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b63f26-65bd777cc4-z8cfp" event={"ID":"2d080cb6-7e9f-4bf3-82ca-acb9b838f943","Type":"ContainerStarted","Data":"3ceff005192bd6a3cb9065c9edf2d55ec3b0263ed95ff68e85bdacedfc330941"}
Sep 18 13:56:35.715426 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:35.715285    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b63f26-65bd777cc4-z8cfp"
Sep 18 13:56:35.715426 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:35.715311    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b63f26-65bd777cc4-z8cfp" event={"ID":"2d080cb6-7e9f-4bf3-82ca-acb9b838f943","Type":"ContainerStarted","Data":"7455839ade90a8942e1307be7d3de7a76cc57741ceb01d22efece6589d4c0a8f"}
Sep 18 13:56:35.717135 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:35.717104    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b63f26-fd596bb8f-fhz8b" event={"ID":"7e06b525-41a0-43ac-a7c9-337a24138e89","Type":"ContainerStarted","Data":"0438d633f236bb85b26ab317c38ef6cf26fda60a0591dfe39f80e4500b2e26c5"}
Sep 18 13:56:35.717283 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:35.717157    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b63f26-fd596bb8f-fhz8b" event={"ID":"7e06b525-41a0-43ac-a7c9-337a24138e89","Type":"ContainerStarted","Data":"c12a3c8a702cd3f9b2a6f52ad4662de19200dee0fb8b0ed734f15970354e56cc"}
Sep 18 13:56:35.717283 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:35.717200    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-b63f26-fd596bb8f-fhz8b"
Sep 18 13:56:35.732918 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:35.732858    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b63f26-65bd777cc4-z8cfp" podStartSLOduration=1.732839775 podStartE2EDuration="1.732839775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:56:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:56:35.731386304 +0000 UTC m=+744.784045631" watchObservedRunningTime="2026-09-18 13:56:35.732839775 +0000 UTC m=+744.785499146"
Sep 18 13:56:35.751542 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:35.751481    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-b63f26-fd596bb8f-fhz8b" podStartSLOduration=1.751458977 podStartE2EDuration="1.751458977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:56:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:56:35.750201263 +0000 UTC m=+744.802860591" watchObservedRunningTime="2026-09-18 13:56:35.751458977 +0000 UTC m=+744.804118309"
Sep 18 13:56:36.019539 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:36.019450    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:36.657654 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:36.657619    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ed4c53-59f4997598-7jfng"
Sep 18 13:56:36.924048 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:36.923951    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:37.785265 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:37.785223    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:38.648693 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:38.648657    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:38.707177 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:38.707148    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e095dac8-ff577f99-lggh8"
Sep 18 13:56:39.644621 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:39.644579    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:39.678990 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:39.678955    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-70f06e-869f668c77-z9mx8"
Sep 18 13:56:39.680016 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:39.679989    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-70f06e-8d6b74974-qshdd"
Sep 18 13:56:39.921735 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:39.921654    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:40.327799 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:40.327718    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:40.428766 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:40.428737    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:40.754513 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:40.754474    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:40.856521 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:40.856487    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:40.953200 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:40.953167    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:41.442575 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:41.442548    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:42.450571 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:42.450539    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:43.256728 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:43.256622    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:44.146200 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:44.146158    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:44.442193 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:44.442105    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:44.931451 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:44.931416    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:45.630372 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:45.630340    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:46.267579 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:46.267487    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:46.726061 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:46.726029    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-b63f26-fd596bb8f-fhz8b"
Sep 18 13:56:46.726445 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:46.726232    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b63f26-65bd777cc4-z8cfp"
Sep 18 13:56:46.755542 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:46.755502    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:46.792076 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:46.792045    2577 generic.go:356] "Generic (PLEG): container finished" podID="62972a5e-7296-4486-928b-29f00380ef52" containerID="8e684556d7d790083aca3b8cbc784dda56d26cdd45d309249f95b4d9b7fa8ba3" exitCode=0
Sep 18 13:56:46.792231 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:46.792116    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c" event={"ID":"62972a5e-7296-4486-928b-29f00380ef52","Type":"ContainerDied","Data":"8e684556d7d790083aca3b8cbc784dda56d26cdd45d309249f95b4d9b7fa8ba3"}
Sep 18 13:56:46.792526 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:46.792511    2577 scope.go:117] "RemoveContainer" containerID="8e684556d7d790083aca3b8cbc784dda56d26cdd45d309249f95b4d9b7fa8ba3"
Sep 18 13:56:46.794713 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:46.794675    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 13:56:46.794876 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:46.794762    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 13:56:46.794876 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:46.794833    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 13:56:47.063688 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:47.063599    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:47.804424 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:47.804372    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c" event={"ID":"62972a5e-7296-4486-928b-29f00380ef52","Type":"ContainerStarted","Data":"c9a4a4db39fc559cc5b310136d64745f968bc98adf9bc459681072fd610774a9"}
Sep 18 13:56:47.979413 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:47.979371    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:48.263458 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:48.263419    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c"
Sep 18 13:56:48.263665 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:48.263618    2577 patch_prober.go:28] interesting pod/e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Sep 18 13:56:48.263747 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:48.263675    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c" podUID="62972a5e-7296-4486-928b-29f00380ef52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Sep 18 13:56:48.721960 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:48.721925    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:49.004178 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:49.004077    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:49.264012 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:49.263919    2577 patch_prober.go:28] interesting pod/e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Sep 18 13:56:49.264012 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:49.263982    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c" podUID="62972a5e-7296-4486-928b-29f00380ef52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Sep 18 13:56:49.510222 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:49.510190    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:50.263316 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:50.263270    2577 patch_prober.go:28] interesting pod/e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Sep 18 13:56:50.263792 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:50.263357    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c" podUID="62972a5e-7296-4486-928b-29f00380ef52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Sep 18 13:56:50.470022 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:50.469985    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:51.263908 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:51.263871    2577 patch_prober.go:28] interesting pod/e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Sep 18 13:56:51.264305 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:51.263941    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c" podUID="62972a5e-7296-4486-928b-29f00380ef52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Sep 18 13:56:51.395968 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:51.395906    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:52.153774 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:52.153734    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:52.263728 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:52.263691    2577 patch_prober.go:28] interesting pod/e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Sep 18 13:56:52.263947 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:52.263756    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c" podUID="62972a5e-7296-4486-928b-29f00380ef52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Sep 18 13:56:53.039088 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:53.039058    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:53.267342 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:53.267312    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c"
Sep 18 13:56:53.267810 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:53.267710    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c"
Sep 18 13:56:53.268394 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:53.268377    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c"
Sep 18 13:56:54.198633 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:54.198591    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e095dac8-ff577f99-lggh8"]
Sep 18 13:56:54.198898 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:54.198870    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e095dac8-ff577f99-lggh8" podUID="37dbe38b-965f-4e46-ba31-b40183261aa6" containerName="maas-api" containerID="cri-o://92e0b8f993102211c720d11d88c451c181db31f69a83d339ddbea48da9963c48" gracePeriod=30
Sep 18 13:56:54.225606 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:54.225577    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:54.301154 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:54.301124    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:54.368632 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:54.368597    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:54.436405 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:54.436378    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:54.440290 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:54.440266    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e095dac8-ff577f99-lggh8"
Sep 18 13:56:54.539519 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:54.539421    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jjjcv\" (UniqueName: \"kubernetes.io/projected/37dbe38b-965f-4e46-ba31-b40183261aa6-kube-api-access-jjjcv\") pod \"37dbe38b-965f-4e46-ba31-b40183261aa6\" (UID: \"37dbe38b-965f-4e46-ba31-b40183261aa6\") "
Sep 18 13:56:54.539700 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:54.539550    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37dbe38b-965f-4e46-ba31-b40183261aa6-openshift-service-ca\") pod \"37dbe38b-965f-4e46-ba31-b40183261aa6\" (UID: \"37dbe38b-965f-4e46-ba31-b40183261aa6\") "
Sep 18 13:56:54.539700 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:54.539609    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37dbe38b-965f-4e46-ba31-b40183261aa6-metrics-certs\") pod \"37dbe38b-965f-4e46-ba31-b40183261aa6\" (UID: \"37dbe38b-965f-4e46-ba31-b40183261aa6\") "
Sep 18 13:56:54.539700 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:54.539648    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37dbe38b-965f-4e46-ba31-b40183261aa6-maas-api-tls\") pod \"37dbe38b-965f-4e46-ba31-b40183261aa6\" (UID: \"37dbe38b-965f-4e46-ba31-b40183261aa6\") "
Sep 18 13:56:54.540313 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:54.540282    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37dbe38b-965f-4e46-ba31-b40183261aa6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "37dbe38b-965f-4e46-ba31-b40183261aa6" (UID: "37dbe38b-965f-4e46-ba31-b40183261aa6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:56:54.542353 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:54.542326    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37dbe38b-965f-4e46-ba31-b40183261aa6-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "37dbe38b-965f-4e46-ba31-b40183261aa6" (UID: "37dbe38b-965f-4e46-ba31-b40183261aa6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:56:54.544430 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:54.544384    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37dbe38b-965f-4e46-ba31-b40183261aa6-kube-api-access-jjjcv" (OuterVolumeSpecName: "kube-api-access-jjjcv") pod "37dbe38b-965f-4e46-ba31-b40183261aa6" (UID: "37dbe38b-965f-4e46-ba31-b40183261aa6"). InnerVolumeSpecName "kube-api-access-jjjcv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:56:54.544723 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:54.544693    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37dbe38b-965f-4e46-ba31-b40183261aa6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "37dbe38b-965f-4e46-ba31-b40183261aa6" (UID: "37dbe38b-965f-4e46-ba31-b40183261aa6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:56:54.589277 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:54.589245    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:54.641204 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:54.641170    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37dbe38b-965f-4e46-ba31-b40183261aa6-openshift-service-ca\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:56:54.641204 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:54.641201    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37dbe38b-965f-4e46-ba31-b40183261aa6-metrics-certs\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:56:54.641204 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:54.641211    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37dbe38b-965f-4e46-ba31-b40183261aa6-maas-api-tls\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:56:54.641421 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:54.641219    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjjcv\" (UniqueName: \"kubernetes.io/projected/37dbe38b-965f-4e46-ba31-b40183261aa6-kube-api-access-jjjcv\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:56:54.843473 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:54.843438    2577 generic.go:356] "Generic (PLEG): container finished" podID="37dbe38b-965f-4e46-ba31-b40183261aa6" containerID="92e0b8f993102211c720d11d88c451c181db31f69a83d339ddbea48da9963c48" exitCode=0
Sep 18 13:56:54.843654 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:54.843488    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e095dac8-ff577f99-lggh8"
Sep 18 13:56:54.843654 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:54.843488    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e095dac8-ff577f99-lggh8" event={"ID":"37dbe38b-965f-4e46-ba31-b40183261aa6","Type":"ContainerDied","Data":"92e0b8f993102211c720d11d88c451c181db31f69a83d339ddbea48da9963c48"}
Sep 18 13:56:54.843654 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:54.843586    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e095dac8-ff577f99-lggh8" event={"ID":"37dbe38b-965f-4e46-ba31-b40183261aa6","Type":"ContainerDied","Data":"80265ca9b2ae8832d4917c6ffb02014a8279d4955d885c917f58634397230d96"}
Sep 18 13:56:54.843654 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:54.843614    2577 scope.go:117] "RemoveContainer" containerID="92e0b8f993102211c720d11d88c451c181db31f69a83d339ddbea48da9963c48"
Sep 18 13:56:54.853885 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:54.853736    2577 scope.go:117] "RemoveContainer" containerID="92e0b8f993102211c720d11d88c451c181db31f69a83d339ddbea48da9963c48"
Sep 18 13:56:54.854131 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:56:54.854087    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92e0b8f993102211c720d11d88c451c181db31f69a83d339ddbea48da9963c48\": container with ID starting with 92e0b8f993102211c720d11d88c451c181db31f69a83d339ddbea48da9963c48 not found: ID does not exist" containerID="92e0b8f993102211c720d11d88c451c181db31f69a83d339ddbea48da9963c48"
Sep 18 13:56:54.854244 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:54.854142    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92e0b8f993102211c720d11d88c451c181db31f69a83d339ddbea48da9963c48"} err="failed to get container status \"92e0b8f993102211c720d11d88c451c181db31f69a83d339ddbea48da9963c48\": rpc error: code = NotFound desc = could not find container \"92e0b8f993102211c720d11d88c451c181db31f69a83d339ddbea48da9963c48\": container with ID starting with 92e0b8f993102211c720d11d88c451c181db31f69a83d339ddbea48da9963c48 not found: ID does not exist"
Sep 18 13:56:54.868496 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:54.868464    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:54.870532 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:54.870512    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e095dac8-ff577f99-lggh8"]
Sep 18 13:56:54.872684 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:54.872666    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e095dac8-ff577f99-lggh8"]
Sep 18 13:56:55.057279 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:55.057247    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:55.107979 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:55.107902    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37dbe38b-965f-4e46-ba31-b40183261aa6" path="/var/lib/kubelet/pods/37dbe38b-965f-4e46-ba31-b40183261aa6/volumes"
Sep 18 13:56:55.157996 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:55.157969    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:55.270340 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:55.270301    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:55.547771 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:55.547691    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv"]
Sep 18 13:56:55.548145 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:55.548070    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37dbe38b-965f-4e46-ba31-b40183261aa6" containerName="maas-api"
Sep 18 13:56:55.548145 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:55.548082    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="37dbe38b-965f-4e46-ba31-b40183261aa6" containerName="maas-api"
Sep 18 13:56:55.548145 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:55.548145    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="37dbe38b-965f-4e46-ba31-b40183261aa6" containerName="maas-api"
Sep 18 13:56:55.552740 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:55.552721    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv"
Sep 18 13:56:55.554989 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:55.554970    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-3675964e-gw-openshift-default-dockercfg-hwsbd\""
Sep 18 13:56:55.563975 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:55.563952    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv"]
Sep 18 13:56:55.579616 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:55.579593    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:55.650962 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:55.650926    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/776c8aaa-e523-4ddb-a4f3-f786607ba536-istio-data\") pod \"e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv\" (UID: \"776c8aaa-e523-4ddb-a4f3-f786607ba536\") " pod="openshift-ingress/e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv"
Sep 18 13:56:55.650962 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:55.650962    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/776c8aaa-e523-4ddb-a4f3-f786607ba536-istiod-ca-cert\") pod \"e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv\" (UID: \"776c8aaa-e523-4ddb-a4f3-f786607ba536\") " pod="openshift-ingress/e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv"
Sep 18 13:56:55.651205 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:55.650982    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/776c8aaa-e523-4ddb-a4f3-f786607ba536-istio-envoy\") pod \"e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv\" (UID: \"776c8aaa-e523-4ddb-a4f3-f786607ba536\") " pod="openshift-ingress/e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv"
Sep 18 13:56:55.651205 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:55.651057    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrslc\" (UniqueName: \"kubernetes.io/projected/776c8aaa-e523-4ddb-a4f3-f786607ba536-kube-api-access-wrslc\") pod \"e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv\" (UID: \"776c8aaa-e523-4ddb-a4f3-f786607ba536\") " pod="openshift-ingress/e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv"
Sep 18 13:56:55.651205 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:55.651163    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/776c8aaa-e523-4ddb-a4f3-f786607ba536-istio-podinfo\") pod \"e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv\" (UID: \"776c8aaa-e523-4ddb-a4f3-f786607ba536\") " pod="openshift-ingress/e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv"
Sep 18 13:56:55.651205 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:55.651199    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/776c8aaa-e523-4ddb-a4f3-f786607ba536-istio-token\") pod \"e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv\" (UID: \"776c8aaa-e523-4ddb-a4f3-f786607ba536\") " pod="openshift-ingress/e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv"
Sep 18 13:56:55.651351 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:55.651244    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/776c8aaa-e523-4ddb-a4f3-f786607ba536-workload-socket\") pod \"e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv\" (UID: \"776c8aaa-e523-4ddb-a4f3-f786607ba536\") " pod="openshift-ingress/e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv"
Sep 18 13:56:55.651351 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:55.651330    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/776c8aaa-e523-4ddb-a4f3-f786607ba536-credential-socket\") pod \"e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv\" (UID: \"776c8aaa-e523-4ddb-a4f3-f786607ba536\") " pod="openshift-ingress/e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv"
Sep 18 13:56:55.651416 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:55.651369    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/776c8aaa-e523-4ddb-a4f3-f786607ba536-workload-certs\") pod \"e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv\" (UID: \"776c8aaa-e523-4ddb-a4f3-f786607ba536\") " pod="openshift-ingress/e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv"
Sep 18 13:56:55.752665 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:55.752627    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrslc\" (UniqueName: \"kubernetes.io/projected/776c8aaa-e523-4ddb-a4f3-f786607ba536-kube-api-access-wrslc\") pod \"e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv\" (UID: \"776c8aaa-e523-4ddb-a4f3-f786607ba536\") " pod="openshift-ingress/e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv"
Sep 18 13:56:55.752665 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:55.752670    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/776c8aaa-e523-4ddb-a4f3-f786607ba536-istio-podinfo\") pod \"e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv\" (UID: \"776c8aaa-e523-4ddb-a4f3-f786607ba536\") " pod="openshift-ingress/e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv"
Sep 18 13:56:55.752945 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:55.752688    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/776c8aaa-e523-4ddb-a4f3-f786607ba536-istio-token\") pod \"e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv\" (UID: \"776c8aaa-e523-4ddb-a4f3-f786607ba536\") " pod="openshift-ingress/e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv"
Sep 18 13:56:55.752945 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:55.752724    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/776c8aaa-e523-4ddb-a4f3-f786607ba536-workload-socket\") pod \"e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv\" (UID: \"776c8aaa-e523-4ddb-a4f3-f786607ba536\") " pod="openshift-ingress/e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv"
Sep 18 13:56:55.752945 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:55.752770    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/776c8aaa-e523-4ddb-a4f3-f786607ba536-credential-socket\") pod \"e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv\" (UID: \"776c8aaa-e523-4ddb-a4f3-f786607ba536\") " pod="openshift-ingress/e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv"
Sep 18 13:56:55.752945 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:55.752800    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/776c8aaa-e523-4ddb-a4f3-f786607ba536-workload-certs\") pod \"e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv\" (UID: \"776c8aaa-e523-4ddb-a4f3-f786607ba536\") " pod="openshift-ingress/e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv"
Sep 18 13:56:55.752945 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:55.752865    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/776c8aaa-e523-4ddb-a4f3-f786607ba536-istio-data\") pod \"e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv\" (UID: \"776c8aaa-e523-4ddb-a4f3-f786607ba536\") " pod="openshift-ingress/e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv"
Sep 18 13:56:55.752945 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:55.752891    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/776c8aaa-e523-4ddb-a4f3-f786607ba536-istiod-ca-cert\") pod \"e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv\" (UID: \"776c8aaa-e523-4ddb-a4f3-f786607ba536\") " pod="openshift-ingress/e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv"
Sep 18 13:56:55.752945 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:55.752918    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/776c8aaa-e523-4ddb-a4f3-f786607ba536-istio-envoy\") pod \"e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv\" (UID: \"776c8aaa-e523-4ddb-a4f3-f786607ba536\") " pod="openshift-ingress/e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv"
Sep 18 13:56:55.753298 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:55.753271    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/776c8aaa-e523-4ddb-a4f3-f786607ba536-workload-certs\") pod \"e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv\" (UID: \"776c8aaa-e523-4ddb-a4f3-f786607ba536\") " pod="openshift-ingress/e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv"
Sep 18 13:56:55.753355 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:55.753328    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/776c8aaa-e523-4ddb-a4f3-f786607ba536-istio-data\") pod \"e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv\" (UID: \"776c8aaa-e523-4ddb-a4f3-f786607ba536\") " pod="openshift-ingress/e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv"
Sep 18 13:56:55.753414 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:55.753369    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/776c8aaa-e523-4ddb-a4f3-f786607ba536-credential-socket\") pod \"e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv\" (UID: \"776c8aaa-e523-4ddb-a4f3-f786607ba536\") " pod="openshift-ingress/e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv"
Sep 18 13:56:55.753515 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:55.753494    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/776c8aaa-e523-4ddb-a4f3-f786607ba536-workload-socket\") pod \"e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv\" (UID: \"776c8aaa-e523-4ddb-a4f3-f786607ba536\") " pod="openshift-ingress/e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv"
Sep 18 13:56:55.753756 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:55.753736    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/776c8aaa-e523-4ddb-a4f3-f786607ba536-istiod-ca-cert\") pod \"e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv\" (UID: \"776c8aaa-e523-4ddb-a4f3-f786607ba536\") " pod="openshift-ingress/e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv"
Sep 18 13:56:55.755235 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:55.755215    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/776c8aaa-e523-4ddb-a4f3-f786607ba536-istio-envoy\") pod \"e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv\" (UID: \"776c8aaa-e523-4ddb-a4f3-f786607ba536\") " pod="openshift-ingress/e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv"
Sep 18 13:56:55.755495 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:55.755478    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/776c8aaa-e523-4ddb-a4f3-f786607ba536-istio-podinfo\") pod \"e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv\" (UID: \"776c8aaa-e523-4ddb-a4f3-f786607ba536\") " pod="openshift-ingress/e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv"
Sep 18 13:56:55.761345 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:55.761317    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/776c8aaa-e523-4ddb-a4f3-f786607ba536-istio-token\") pod \"e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv\" (UID: \"776c8aaa-e523-4ddb-a4f3-f786607ba536\") " pod="openshift-ingress/e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv"
Sep 18 13:56:55.762251 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:55.762229    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrslc\" (UniqueName: \"kubernetes.io/projected/776c8aaa-e523-4ddb-a4f3-f786607ba536-kube-api-access-wrslc\") pod \"e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv\" (UID: \"776c8aaa-e523-4ddb-a4f3-f786607ba536\") " pod="openshift-ingress/e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv"
Sep 18 13:56:55.813641 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:55.813562    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:55.864523 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:55.864494    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv"
Sep 18 13:56:56.019283 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:56.019248    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:56.147761 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:56.147726    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv"]
Sep 18 13:56:56.149478 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:56:56.149445    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod776c8aaa_e523_4ddb_a4f3_f786607ba536.slice/crio-5124dc7a2b7e25fc00f77fc2ac14e29a20054aee41a541d5922538e1ead393af WatchSource:0}: Error finding container 5124dc7a2b7e25fc00f77fc2ac14e29a20054aee41a541d5922538e1ead393af: Status 404 returned error can't find the container with id 5124dc7a2b7e25fc00f77fc2ac14e29a20054aee41a541d5922538e1ead393af
Sep 18 13:56:56.151762 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:56.151729    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 13:56:56.151876 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:56.151809    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 13:56:56.151876 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:56.151869    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 13:56:56.691181 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:56.691132    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:56.857488 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:56.857453    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv" event={"ID":"776c8aaa-e523-4ddb-a4f3-f786607ba536","Type":"ContainerStarted","Data":"3f9806c6b77b6883e34650a03413943cb3456a6932e1369c35dd4a6d4664d589"}
Sep 18 13:56:56.857651 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:56.857532    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv" event={"ID":"776c8aaa-e523-4ddb-a4f3-f786607ba536","Type":"ContainerStarted","Data":"5124dc7a2b7e25fc00f77fc2ac14e29a20054aee41a541d5922538e1ead393af"}
Sep 18 13:56:56.865044 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:56.865023    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv"
Sep 18 13:56:56.881532 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:56.881349    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv" podStartSLOduration=1.881328696 podStartE2EDuration="1.881328696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:56:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:56:56.881082111 +0000 UTC m=+765.933741438" watchObservedRunningTime="2026-09-18 13:56:56.881328696 +0000 UTC m=+765.933988024"
Sep 18 13:56:57.749483 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:57.749442    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:57.870468 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:57.870437    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv"
Sep 18 13:56:58.388868 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:58.388814    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:58.490946 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:58.490908    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:58.590462 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:58.590429    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:58.869456 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:58.869427    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv"
Sep 18 13:56:58.870545 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:58.870524    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv"
Sep 18 13:56:59.066300 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:59.066270    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:59.264118 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:59.264027    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:59.385481 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:59.385436    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:56:59.611606 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:56:59.611572    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:57:00.499648 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:00.499603    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:57:00.601363 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:00.601326    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 13:57:00.834326 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:00.834213    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:57:01.148247 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:01.148216    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3675964e-6755cc86f6-bnppb"]
Sep 18 13:57:01.151951 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:01.151926    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3675964e-6755cc86f6-bnppb"
Sep 18 13:57:01.154711 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:01.154625    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-3675964e-serving-cert\""
Sep 18 13:57:01.155184 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:01.155157    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 18 13:57:01.155508 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:01.155479    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vx9xd\""
Sep 18 13:57:01.155743 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:01.155725    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 13:57:01.156246 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:01.155732    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 18 13:57:01.165972 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:01.165772    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3675964e-6755cc86f6-bnppb"]
Sep 18 13:57:01.186494 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:01.186462    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:57:01.250445 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:01.250407    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:57:01.301737 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:01.301695    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51e2a7c9-fc1b-417d-b598-866bdcc4e0f6-maas-api-tls\") pod \"maas-api-e2e-migrate-3675964e-6755cc86f6-bnppb\" (UID: \"51e2a7c9-fc1b-417d-b598-866bdcc4e0f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3675964e-6755cc86f6-bnppb"
Sep 18 13:57:01.301979 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:01.301762    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51e2a7c9-fc1b-417d-b598-866bdcc4e0f6-openshift-service-ca\") pod \"maas-api-e2e-migrate-3675964e-6755cc86f6-bnppb\" (UID: \"51e2a7c9-fc1b-417d-b598-866bdcc4e0f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3675964e-6755cc86f6-bnppb"
Sep 18 13:57:01.301979 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:01.301876    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p67s4\" (UniqueName: \"kubernetes.io/projected/51e2a7c9-fc1b-417d-b598-866bdcc4e0f6-kube-api-access-p67s4\") pod \"maas-api-e2e-migrate-3675964e-6755cc86f6-bnppb\" (UID: \"51e2a7c9-fc1b-417d-b598-866bdcc4e0f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3675964e-6755cc86f6-bnppb"
Sep 18 13:57:01.301979 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:01.301959    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51e2a7c9-fc1b-417d-b598-866bdcc4e0f6-metrics-certs\") pod \"maas-api-e2e-migrate-3675964e-6755cc86f6-bnppb\" (UID: \"51e2a7c9-fc1b-417d-b598-866bdcc4e0f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3675964e-6755cc86f6-bnppb"
Sep 18 13:57:01.402977 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:01.402874    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51e2a7c9-fc1b-417d-b598-866bdcc4e0f6-maas-api-tls\") pod \"maas-api-e2e-migrate-3675964e-6755cc86f6-bnppb\" (UID: \"51e2a7c9-fc1b-417d-b598-866bdcc4e0f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3675964e-6755cc86f6-bnppb"
Sep 18 13:57:01.402977 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:01.402952    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51e2a7c9-fc1b-417d-b598-866bdcc4e0f6-openshift-service-ca\") pod \"maas-api-e2e-migrate-3675964e-6755cc86f6-bnppb\" (UID: \"51e2a7c9-fc1b-417d-b598-866bdcc4e0f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3675964e-6755cc86f6-bnppb"
Sep 18 13:57:01.403215 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:01.402991    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p67s4\" (UniqueName: \"kubernetes.io/projected/51e2a7c9-fc1b-417d-b598-866bdcc4e0f6-kube-api-access-p67s4\") pod \"maas-api-e2e-migrate-3675964e-6755cc86f6-bnppb\" (UID: \"51e2a7c9-fc1b-417d-b598-866bdcc4e0f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3675964e-6755cc86f6-bnppb"
Sep 18 13:57:01.403215 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:01.403051    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51e2a7c9-fc1b-417d-b598-866bdcc4e0f6-metrics-certs\") pod \"maas-api-e2e-migrate-3675964e-6755cc86f6-bnppb\" (UID: \"51e2a7c9-fc1b-417d-b598-866bdcc4e0f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3675964e-6755cc86f6-bnppb"
Sep 18 13:57:01.403215 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:57:01.403054    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-3675964e-serving-cert: secret "maas-api-e2e-migrate-3675964e-serving-cert" not found
Sep 18 13:57:01.403215 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:57:01.403191    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51e2a7c9-fc1b-417d-b598-866bdcc4e0f6-maas-api-tls podName:51e2a7c9-fc1b-417d-b598-866bdcc4e0f6 nodeName:}" failed. No retries permitted until 2026-09-18 13:57:01.903165597 +0000 UTC m=+770.955824922 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/51e2a7c9-fc1b-417d-b598-866bdcc4e0f6-maas-api-tls") pod "maas-api-e2e-migrate-3675964e-6755cc86f6-bnppb" (UID: "51e2a7c9-fc1b-417d-b598-866bdcc4e0f6") : secret "maas-api-e2e-migrate-3675964e-serving-cert" not found
Sep 18 13:57:01.403621 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:01.403579    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51e2a7c9-fc1b-417d-b598-866bdcc4e0f6-openshift-service-ca\") pod \"maas-api-e2e-migrate-3675964e-6755cc86f6-bnppb\" (UID: \"51e2a7c9-fc1b-417d-b598-866bdcc4e0f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3675964e-6755cc86f6-bnppb"
Sep 18 13:57:01.405698 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:01.405676    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51e2a7c9-fc1b-417d-b598-866bdcc4e0f6-metrics-certs\") pod \"maas-api-e2e-migrate-3675964e-6755cc86f6-bnppb\" (UID: \"51e2a7c9-fc1b-417d-b598-866bdcc4e0f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3675964e-6755cc86f6-bnppb"
Sep 18 13:57:01.411746 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:01.411728    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p67s4\" (UniqueName: \"kubernetes.io/projected/51e2a7c9-fc1b-417d-b598-866bdcc4e0f6-kube-api-access-p67s4\") pod \"maas-api-e2e-migrate-3675964e-6755cc86f6-bnppb\" (UID: \"51e2a7c9-fc1b-417d-b598-866bdcc4e0f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3675964e-6755cc86f6-bnppb"
Sep 18 13:57:01.539283 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:01.539246    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:57:01.768944 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:01.768815    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:57:01.867681 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:01.867644    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:57:01.907639 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:01.907608    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51e2a7c9-fc1b-417d-b598-866bdcc4e0f6-maas-api-tls\") pod \"maas-api-e2e-migrate-3675964e-6755cc86f6-bnppb\" (UID: \"51e2a7c9-fc1b-417d-b598-866bdcc4e0f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3675964e-6755cc86f6-bnppb"
Sep 18 13:57:01.910172 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:01.910149    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51e2a7c9-fc1b-417d-b598-866bdcc4e0f6-maas-api-tls\") pod \"maas-api-e2e-migrate-3675964e-6755cc86f6-bnppb\" (UID: \"51e2a7c9-fc1b-417d-b598-866bdcc4e0f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3675964e-6755cc86f6-bnppb"
Sep 18 13:57:02.071560 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:02.071465    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3675964e-6755cc86f6-bnppb"
Sep 18 13:57:02.157877 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:02.157845    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:57:02.347334 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:02.347306    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3675964e-6755cc86f6-bnppb"]
Sep 18 13:57:02.349321 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:57:02.349295    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51e2a7c9_fc1b_417d_b598_866bdcc4e0f6.slice/crio-f540703088198e7963423252109388684983b9eff3f874a67d399c7832a689e6 WatchSource:0}: Error finding container f540703088198e7963423252109388684983b9eff3f874a67d399c7832a689e6: Status 404 returned error can't find the container with id f540703088198e7963423252109388684983b9eff3f874a67d399c7832a689e6
Sep 18 13:57:02.892785 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:02.892747    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3675964e-6755cc86f6-bnppb" event={"ID":"51e2a7c9-fc1b-417d-b598-866bdcc4e0f6","Type":"ContainerStarted","Data":"aa2b25ca51bf928ad6338a750207e608a3c8c0c6a896566b5aaebef0f81172bb"}
Sep 18 13:57:02.893179 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:02.892797    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3675964e-6755cc86f6-bnppb" event={"ID":"51e2a7c9-fc1b-417d-b598-866bdcc4e0f6","Type":"ContainerStarted","Data":"f540703088198e7963423252109388684983b9eff3f874a67d399c7832a689e6"}
Sep 18 13:57:02.893179 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:02.892875    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3675964e-6755cc86f6-bnppb"
Sep 18 13:57:02.911334 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:02.911285    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3675964e-6755cc86f6-bnppb" podStartSLOduration=1.54619216 podStartE2EDuration="1.911265697s" totalImagesPullingTime="365.073537ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:57:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:57:02.910558182 +0000 UTC m=+771.963217510" watchObservedRunningTime="2026-09-18 13:57:02.911265697 +0000 UTC m=+771.963925036"
Sep 18 13:57:02.973655 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:02.973572    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:57:03.629790 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:03.629735    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:57:04.375236 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:04.375204    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:57:05.861165 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:05.861126    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:57:05.932344 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:05.932312    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:57:06.066690 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:06.066648    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:57:06.206095 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:06.206007    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:57:06.304195 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:06.304163    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:57:08.905801 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:08.905773    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3675964e-6755cc86f6-bnppb"
Sep 18 13:57:12.489436 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:12.489396    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:57:12.590390 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:12.590334    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:57:17.495601 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:17.495553    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 13:57:18.082997 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:18.082952    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-377ca2/simulator-w4-377ca2-kserve-797f87bbfb-x986s"]
Sep 18 13:57:18.091615 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:18.091570    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-377ca2/simulator-w4-377ca2-kserve-797f87bbfb-x986s"
Sep 18 13:57:18.097474 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:18.097436    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-377ca2/simulator-w4-377ca2-kserve-797f87bbfb-x986s"]
Sep 18 13:57:18.098089 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:18.098063    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-377ca2\"/\"openshift-service-ca.crt\""
Sep 18 13:57:18.098605 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:18.098584    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-377ca2\"/\"kube-root-ca.crt\""
Sep 18 13:57:18.098748 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:18.098724    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-377ca2\"/\"default-dockercfg-zzxlh\""
Sep 18 13:57:18.098981 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:18.098959    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-377ca2\"/\"simulator-w4-377ca2-kserve-self-signed-certs\""
Sep 18 13:57:18.148787 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:18.148747    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3a3b935-cb2b-4a63-ba38-2906228f737a-home\") pod \"simulator-w4-377ca2-kserve-797f87bbfb-x986s\" (UID: \"e3a3b935-cb2b-4a63-ba38-2906228f737a\") " pod="e2e-models-e2e-worker-w4-377ca2/simulator-w4-377ca2-kserve-797f87bbfb-x986s"
Sep 18 13:57:18.149017 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:18.148818    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3a3b935-cb2b-4a63-ba38-2906228f737a-dshm\") pod \"simulator-w4-377ca2-kserve-797f87bbfb-x986s\" (UID: \"e3a3b935-cb2b-4a63-ba38-2906228f737a\") " pod="e2e-models-e2e-worker-w4-377ca2/simulator-w4-377ca2-kserve-797f87bbfb-x986s"
Sep 18 13:57:18.149017 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:18.148877    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jgfc\" (UniqueName: \"kubernetes.io/projected/e3a3b935-cb2b-4a63-ba38-2906228f737a-kube-api-access-5jgfc\") pod \"simulator-w4-377ca2-kserve-797f87bbfb-x986s\" (UID: \"e3a3b935-cb2b-4a63-ba38-2906228f737a\") " pod="e2e-models-e2e-worker-w4-377ca2/simulator-w4-377ca2-kserve-797f87bbfb-x986s"
Sep 18 13:57:18.149017 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:18.148932    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3a3b935-cb2b-4a63-ba38-2906228f737a-model-cache\") pod \"simulator-w4-377ca2-kserve-797f87bbfb-x986s\" (UID: \"e3a3b935-cb2b-4a63-ba38-2906228f737a\") " pod="e2e-models-e2e-worker-w4-377ca2/simulator-w4-377ca2-kserve-797f87bbfb-x986s"
Sep 18 13:57:18.149017 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:18.148955    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3a3b935-cb2b-4a63-ba38-2906228f737a-tmp-dir\") pod \"simulator-w4-377ca2-kserve-797f87bbfb-x986s\" (UID: \"e3a3b935-cb2b-4a63-ba38-2906228f737a\") " pod="e2e-models-e2e-worker-w4-377ca2/simulator-w4-377ca2-kserve-797f87bbfb-x986s"
Sep 18 13:57:18.149017 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:18.148978    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3a3b935-cb2b-4a63-ba38-2906228f737a-tls-certs\") pod \"simulator-w4-377ca2-kserve-797f87bbfb-x986s\" (UID: \"e3a3b935-cb2b-4a63-ba38-2906228f737a\") " pod="e2e-models-e2e-worker-w4-377ca2/simulator-w4-377ca2-kserve-797f87bbfb-x986s"
Sep 18 13:57:18.250155 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:18.250118    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3a3b935-cb2b-4a63-ba38-2906228f737a-model-cache\") pod \"simulator-w4-377ca2-kserve-797f87bbfb-x986s\" (UID: \"e3a3b935-cb2b-4a63-ba38-2906228f737a\") " pod="e2e-models-e2e-worker-w4-377ca2/simulator-w4-377ca2-kserve-797f87bbfb-x986s"
Sep 18 13:57:18.250155 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:18.250154    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3a3b935-cb2b-4a63-ba38-2906228f737a-tmp-dir\") pod \"simulator-w4-377ca2-kserve-797f87bbfb-x986s\" (UID: \"e3a3b935-cb2b-4a63-ba38-2906228f737a\") " pod="e2e-models-e2e-worker-w4-377ca2/simulator-w4-377ca2-kserve-797f87bbfb-x986s"
Sep 18 13:57:18.250436 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:18.250179    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3a3b935-cb2b-4a63-ba38-2906228f737a-tls-certs\") pod \"simulator-w4-377ca2-kserve-797f87bbfb-x986s\" (UID: \"e3a3b935-cb2b-4a63-ba38-2906228f737a\") " pod="e2e-models-e2e-worker-w4-377ca2/simulator-w4-377ca2-kserve-797f87bbfb-x986s"
Sep 18 13:57:18.250436 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:18.250219    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3a3b935-cb2b-4a63-ba38-2906228f737a-home\") pod \"simulator-w4-377ca2-kserve-797f87bbfb-x986s\" (UID: \"e3a3b935-cb2b-4a63-ba38-2906228f737a\") " pod="e2e-models-e2e-worker-w4-377ca2/simulator-w4-377ca2-kserve-797f87bbfb-x986s"
Sep 18 13:57:18.250436 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:18.250269    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3a3b935-cb2b-4a63-ba38-2906228f737a-dshm\") pod \"simulator-w4-377ca2-kserve-797f87bbfb-x986s\" (UID: \"e3a3b935-cb2b-4a63-ba38-2906228f737a\") " pod="e2e-models-e2e-worker-w4-377ca2/simulator-w4-377ca2-kserve-797f87bbfb-x986s"
Sep 18 13:57:18.250436 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:18.250388    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jgfc\" (UniqueName: \"kubernetes.io/projected/e3a3b935-cb2b-4a63-ba38-2906228f737a-kube-api-access-5jgfc\") pod \"simulator-w4-377ca2-kserve-797f87bbfb-x986s\" (UID: \"e3a3b935-cb2b-4a63-ba38-2906228f737a\") " pod="e2e-models-e2e-worker-w4-377ca2/simulator-w4-377ca2-kserve-797f87bbfb-x986s"
Sep 18 13:57:18.250646 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:18.250600    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3a3b935-cb2b-4a63-ba38-2906228f737a-model-cache\") pod \"simulator-w4-377ca2-kserve-797f87bbfb-x986s\" (UID: \"e3a3b935-cb2b-4a63-ba38-2906228f737a\") " pod="e2e-models-e2e-worker-w4-377ca2/simulator-w4-377ca2-kserve-797f87bbfb-x986s"
Sep 18 13:57:18.250646 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:18.250604    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3a3b935-cb2b-4a63-ba38-2906228f737a-tmp-dir\") pod \"simulator-w4-377ca2-kserve-797f87bbfb-x986s\" (UID: \"e3a3b935-cb2b-4a63-ba38-2906228f737a\") " pod="e2e-models-e2e-worker-w4-377ca2/simulator-w4-377ca2-kserve-797f87bbfb-x986s"
Sep 18 13:57:18.250729 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:18.250671    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3a3b935-cb2b-4a63-ba38-2906228f737a-home\") pod \"simulator-w4-377ca2-kserve-797f87bbfb-x986s\" (UID: \"e3a3b935-cb2b-4a63-ba38-2906228f737a\") " pod="e2e-models-e2e-worker-w4-377ca2/simulator-w4-377ca2-kserve-797f87bbfb-x986s"
Sep 18 13:57:18.252589 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:18.252567    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3a3b935-cb2b-4a63-ba38-2906228f737a-dshm\") pod \"simulator-w4-377ca2-kserve-797f87bbfb-x986s\" (UID: \"e3a3b935-cb2b-4a63-ba38-2906228f737a\") " pod="e2e-models-e2e-worker-w4-377ca2/simulator-w4-377ca2-kserve-797f87bbfb-x986s"
Sep 18 13:57:18.252885 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:18.252865    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3a3b935-cb2b-4a63-ba38-2906228f737a-tls-certs\") pod \"simulator-w4-377ca2-kserve-797f87bbfb-x986s\" (UID: \"e3a3b935-cb2b-4a63-ba38-2906228f737a\") " pod="e2e-models-e2e-worker-w4-377ca2/simulator-w4-377ca2-kserve-797f87bbfb-x986s"
Sep 18 13:57:18.261849 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:18.261813    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jgfc\" (UniqueName: \"kubernetes.io/projected/e3a3b935-cb2b-4a63-ba38-2906228f737a-kube-api-access-5jgfc\") pod \"simulator-w4-377ca2-kserve-797f87bbfb-x986s\" (UID: \"e3a3b935-cb2b-4a63-ba38-2906228f737a\") " pod="e2e-models-e2e-worker-w4-377ca2/simulator-w4-377ca2-kserve-797f87bbfb-x986s"
Sep 18 13:57:18.391360 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:18.391310    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:57:18.410967 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:18.410931    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-377ca2/simulator-w4-377ca2-kserve-797f87bbfb-x986s"
Sep 18 13:57:18.686641 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:18.686537    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-377ca2/simulator-w4-377ca2-kserve-797f87bbfb-x986s"]
Sep 18 13:57:18.702008 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:57:18.693359    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode3a3b935_cb2b_4a63_ba38_2906228f737a.slice/crio-624d0185f943de016d52278dbebd66ba174e2135f352c17f8a33814b7fe80f09 WatchSource:0}: Error finding container 624d0185f943de016d52278dbebd66ba174e2135f352c17f8a33814b7fe80f09: Status 404 returned error can't find the container with id 624d0185f943de016d52278dbebd66ba174e2135f352c17f8a33814b7fe80f09
Sep 18 13:57:18.986601 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:18.986564    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-377ca2/simulator-w4-377ca2-kserve-797f87bbfb-x986s" event={"ID":"e3a3b935-cb2b-4a63-ba38-2906228f737a","Type":"ContainerStarted","Data":"4e3096a5929537d65614b7dbfb62bf40f0c6305620cbba1e4dedcaa41d6ae321"}
Sep 18 13:57:18.986772 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:18.986615    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-377ca2/simulator-w4-377ca2-kserve-797f87bbfb-x986s" event={"ID":"e3a3b935-cb2b-4a63-ba38-2906228f737a","Type":"ContainerStarted","Data":"624d0185f943de016d52278dbebd66ba174e2135f352c17f8a33814b7fe80f09"}
Sep 18 13:57:19.005185 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:19.005121    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-377ca2/simulator-w4-377ca2-kserve-797f87bbfb-x986s" podStartSLOduration=1.005097425 podStartE2EDuration="1.005097425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:57:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:57:19.002469068 +0000 UTC m=+788.055128395" watchObservedRunningTime="2026-09-18 13:57:19.005097425 +0000 UTC m=+788.057756766"
Sep 18 13:57:23.014509 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:23.014472    2577 generic.go:356] "Generic (PLEG): container finished" podID="776c8aaa-e523-4ddb-a4f3-f786607ba536" containerID="3f9806c6b77b6883e34650a03413943cb3456a6932e1369c35dd4a6d4664d589" exitCode=0
Sep 18 13:57:23.014971 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:23.014540    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv" event={"ID":"776c8aaa-e523-4ddb-a4f3-f786607ba536","Type":"ContainerDied","Data":"3f9806c6b77b6883e34650a03413943cb3456a6932e1369c35dd4a6d4664d589"}
Sep 18 13:57:23.014971 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:23.014923    2577 scope.go:117] "RemoveContainer" containerID="3f9806c6b77b6883e34650a03413943cb3456a6932e1369c35dd4a6d4664d589"
Sep 18 13:57:23.016029 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:23.016001    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 13:57:23.016101 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:23.016072    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 13:57:23.016138 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:23.016103    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 13:57:23.319540 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:23.319439    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:57:24.024597 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:24.024554    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv" event={"ID":"776c8aaa-e523-4ddb-a4f3-f786607ba536","Type":"ContainerStarted","Data":"3ad3b0ffd6a1f016f4e8ae1102f3d3d63a573a26f085b80592c57913d28320c0"}
Sep 18 13:57:24.061287 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:24.061246    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:57:24.857174 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:24.857134    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:57:24.865694 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:24.865669    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv"
Sep 18 13:57:24.866594 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:24.866571    2577 patch_prober.go:28] interesting pod/e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Sep 18 13:57:24.866660 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:24.866620    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv" podUID="776c8aaa-e523-4ddb-a4f3-f786607ba536" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Sep 18 13:57:25.763562 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:25.763527    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:57:25.865424 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:25.865387    2577 patch_prober.go:28] interesting pod/e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Sep 18 13:57:25.865610 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:25.865461    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv" podUID="776c8aaa-e523-4ddb-a4f3-f786607ba536" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Sep 18 13:57:26.442185 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:26.442151    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:57:26.865398 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:26.865359    2577 patch_prober.go:28] interesting pod/e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Sep 18 13:57:26.865794 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:26.865414    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv" podUID="776c8aaa-e523-4ddb-a4f3-f786607ba536" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Sep 18 13:57:27.348310 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:27.348275    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:57:27.865643 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:27.865601    2577 patch_prober.go:28] interesting pod/e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Sep 18 13:57:27.866164 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:27.865667    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv" podUID="776c8aaa-e523-4ddb-a4f3-f786607ba536" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Sep 18 13:57:28.144400 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:28.144311    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:57:28.412075 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:28.411982    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-377ca2/simulator-w4-377ca2-kserve-797f87bbfb-x986s"
Sep 18 13:57:28.412075 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:28.412030    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-377ca2/simulator-w4-377ca2-kserve-797f87bbfb-x986s"
Sep 18 13:57:28.424665 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:28.424638    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-377ca2/simulator-w4-377ca2-kserve-797f87bbfb-x986s"
Sep 18 13:57:28.807398 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:28.807310    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:57:28.865757 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:28.865724    2577 patch_prober.go:28] interesting pod/e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Sep 18 13:57:28.866165 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:28.865782    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv" podUID="776c8aaa-e523-4ddb-a4f3-f786607ba536" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Sep 18 13:57:28.869901 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:28.869880    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv"
Sep 18 13:57:29.063517 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:29.063425    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-377ca2/simulator-w4-377ca2-kserve-797f87bbfb-x986s"
Sep 18 13:57:29.504064 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:29.504033    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3675964e-6755cc86f6-bnppb"]
Sep 18 13:57:29.504261 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:29.504239    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3675964e-6755cc86f6-bnppb" podUID="51e2a7c9-fc1b-417d-b598-866bdcc4e0f6" containerName="maas-api" containerID="cri-o://aa2b25ca51bf928ad6338a750207e608a3c8c0c6a896566b5aaebef0f81172bb" gracePeriod=30
Sep 18 13:57:29.513180 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:29.513148    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:57:29.612206 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:29.612167    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:57:29.713561 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:29.713529    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:57:29.758443 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:29.758384    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3675964e-6755cc86f6-bnppb"
Sep 18 13:57:29.786207 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:29.786172    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:57:29.859125 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:29.859084    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51e2a7c9-fc1b-417d-b598-866bdcc4e0f6-metrics-certs\") pod \"51e2a7c9-fc1b-417d-b598-866bdcc4e0f6\" (UID: \"51e2a7c9-fc1b-417d-b598-866bdcc4e0f6\") "
Sep 18 13:57:29.859329 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:29.859182    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51e2a7c9-fc1b-417d-b598-866bdcc4e0f6-maas-api-tls\") pod \"51e2a7c9-fc1b-417d-b598-866bdcc4e0f6\" (UID: \"51e2a7c9-fc1b-417d-b598-866bdcc4e0f6\") "
Sep 18 13:57:29.859329 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:29.859221    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51e2a7c9-fc1b-417d-b598-866bdcc4e0f6-openshift-service-ca\") pod \"51e2a7c9-fc1b-417d-b598-866bdcc4e0f6\" (UID: \"51e2a7c9-fc1b-417d-b598-866bdcc4e0f6\") "
Sep 18 13:57:29.859329 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:29.859284    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p67s4\" (UniqueName: \"kubernetes.io/projected/51e2a7c9-fc1b-417d-b598-866bdcc4e0f6-kube-api-access-p67s4\") pod \"51e2a7c9-fc1b-417d-b598-866bdcc4e0f6\" (UID: \"51e2a7c9-fc1b-417d-b598-866bdcc4e0f6\") "
Sep 18 13:57:29.860419 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:29.860390    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51e2a7c9-fc1b-417d-b598-866bdcc4e0f6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "51e2a7c9-fc1b-417d-b598-866bdcc4e0f6" (UID: "51e2a7c9-fc1b-417d-b598-866bdcc4e0f6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:57:29.864729 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:29.864697    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51e2a7c9-fc1b-417d-b598-866bdcc4e0f6-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "51e2a7c9-fc1b-417d-b598-866bdcc4e0f6" (UID: "51e2a7c9-fc1b-417d-b598-866bdcc4e0f6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:57:29.865070 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:29.865024    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51e2a7c9-fc1b-417d-b598-866bdcc4e0f6-kube-api-access-p67s4" (OuterVolumeSpecName: "kube-api-access-p67s4") pod "51e2a7c9-fc1b-417d-b598-866bdcc4e0f6" (UID: "51e2a7c9-fc1b-417d-b598-866bdcc4e0f6"). InnerVolumeSpecName "kube-api-access-p67s4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:57:29.865234 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:29.865161    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51e2a7c9-fc1b-417d-b598-866bdcc4e0f6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "51e2a7c9-fc1b-417d-b598-866bdcc4e0f6" (UID: "51e2a7c9-fc1b-417d-b598-866bdcc4e0f6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:57:29.887474 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:29.887440    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv"]
Sep 18 13:57:29.888038 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:29.887776    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv" podUID="776c8aaa-e523-4ddb-a4f3-f786607ba536" containerName="istio-proxy" containerID="cri-o://3ad3b0ffd6a1f016f4e8ae1102f3d3d63a573a26f085b80592c57913d28320c0" gracePeriod=30
Sep 18 13:57:29.898191 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:29.898159    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:57:29.960263 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:29.960227    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51e2a7c9-fc1b-417d-b598-866bdcc4e0f6-openshift-service-ca\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:57:29.960263 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:29.960254    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p67s4\" (UniqueName: \"kubernetes.io/projected/51e2a7c9-fc1b-417d-b598-866bdcc4e0f6-kube-api-access-p67s4\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:57:29.960263 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:29.960268    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51e2a7c9-fc1b-417d-b598-866bdcc4e0f6-metrics-certs\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:57:29.960263 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:29.960277    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51e2a7c9-fc1b-417d-b598-866bdcc4e0f6-maas-api-tls\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:57:29.963983 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:29.963950    2577 patch_prober.go:28] interesting pod/e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 18 13:57:29.964134 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:29.964023    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv" podUID="776c8aaa-e523-4ddb-a4f3-f786607ba536" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 13:57:30.059306 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:30.059215    2577 generic.go:356] "Generic (PLEG): container finished" podID="51e2a7c9-fc1b-417d-b598-866bdcc4e0f6" containerID="aa2b25ca51bf928ad6338a750207e608a3c8c0c6a896566b5aaebef0f81172bb" exitCode=0
Sep 18 13:57:30.059306 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:30.059266    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3675964e-6755cc86f6-bnppb"
Sep 18 13:57:30.059503 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:30.059300    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3675964e-6755cc86f6-bnppb" event={"ID":"51e2a7c9-fc1b-417d-b598-866bdcc4e0f6","Type":"ContainerDied","Data":"aa2b25ca51bf928ad6338a750207e608a3c8c0c6a896566b5aaebef0f81172bb"}
Sep 18 13:57:30.059503 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:30.059357    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3675964e-6755cc86f6-bnppb" event={"ID":"51e2a7c9-fc1b-417d-b598-866bdcc4e0f6","Type":"ContainerDied","Data":"f540703088198e7963423252109388684983b9eff3f874a67d399c7832a689e6"}
Sep 18 13:57:30.059503 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:30.059385    2577 scope.go:117] "RemoveContainer" containerID="aa2b25ca51bf928ad6338a750207e608a3c8c0c6a896566b5aaebef0f81172bb"
Sep 18 13:57:30.069434 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:30.069417    2577 scope.go:117] "RemoveContainer" containerID="aa2b25ca51bf928ad6338a750207e608a3c8c0c6a896566b5aaebef0f81172bb"
Sep 18 13:57:30.069701 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:57:30.069682    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa2b25ca51bf928ad6338a750207e608a3c8c0c6a896566b5aaebef0f81172bb\": container with ID starting with aa2b25ca51bf928ad6338a750207e608a3c8c0c6a896566b5aaebef0f81172bb not found: ID does not exist" containerID="aa2b25ca51bf928ad6338a750207e608a3c8c0c6a896566b5aaebef0f81172bb"
Sep 18 13:57:30.069781 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:30.069707    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa2b25ca51bf928ad6338a750207e608a3c8c0c6a896566b5aaebef0f81172bb"} err="failed to get container status \"aa2b25ca51bf928ad6338a750207e608a3c8c0c6a896566b5aaebef0f81172bb\": rpc error: code = NotFound desc = could not find container \"aa2b25ca51bf928ad6338a750207e608a3c8c0c6a896566b5aaebef0f81172bb\": container with ID starting with aa2b25ca51bf928ad6338a750207e608a3c8c0c6a896566b5aaebef0f81172bb not found: ID does not exist"
Sep 18 13:57:30.083684 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:30.083652    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3675964e-6755cc86f6-bnppb"]
Sep 18 13:57:30.087936 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:30.087914    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3675964e-6755cc86f6-bnppb"]
Sep 18 13:57:30.376493 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:30.376458    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:57:30.657994 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:30.657898    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:57:30.757702 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:30.757670    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:57:30.863266 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:30.863234    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:57:31.091782 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:31.091753    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 13:57:31.107342 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:31.107311    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51e2a7c9-fc1b-417d-b598-866bdcc4e0f6" path="/var/lib/kubelet/pods/51e2a7c9-fc1b-417d-b598-866bdcc4e0f6/volumes"
Sep 18 13:57:31.539097 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:31.539018    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:57:32.432479 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:32.432448    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:57:33.105989 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:33.105946    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:57:33.194060 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:33.194030    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:57:34.045691 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:34.045647    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:57:34.833206 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:34.833133    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:57:34.898414 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:34.898381    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:57:34.999002 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:34.998972    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:57:35.092208 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:35.092176    2577 generic.go:356] "Generic (PLEG): container finished" podID="776c8aaa-e523-4ddb-a4f3-f786607ba536" containerID="3ad3b0ffd6a1f016f4e8ae1102f3d3d63a573a26f085b80592c57913d28320c0" exitCode=0
Sep 18 13:57:35.092578 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:35.092244    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv" event={"ID":"776c8aaa-e523-4ddb-a4f3-f786607ba536","Type":"ContainerDied","Data":"3ad3b0ffd6a1f016f4e8ae1102f3d3d63a573a26f085b80592c57913d28320c0"}
Sep 18 13:57:35.092578 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:35.092338    2577 scope.go:117] "RemoveContainer" containerID="3f9806c6b77b6883e34650a03413943cb3456a6932e1369c35dd4a6d4664d589"
Sep 18 13:57:35.139245 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:35.139223    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv"
Sep 18 13:57:35.210111 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:35.210081    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/776c8aaa-e523-4ddb-a4f3-f786607ba536-workload-socket\") pod \"776c8aaa-e523-4ddb-a4f3-f786607ba536\" (UID: \"776c8aaa-e523-4ddb-a4f3-f786607ba536\") "
Sep 18 13:57:35.210282 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:35.210133    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wrslc\" (UniqueName: \"kubernetes.io/projected/776c8aaa-e523-4ddb-a4f3-f786607ba536-kube-api-access-wrslc\") pod \"776c8aaa-e523-4ddb-a4f3-f786607ba536\" (UID: \"776c8aaa-e523-4ddb-a4f3-f786607ba536\") "
Sep 18 13:57:35.210282 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:35.210151    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/776c8aaa-e523-4ddb-a4f3-f786607ba536-istio-podinfo\") pod \"776c8aaa-e523-4ddb-a4f3-f786607ba536\" (UID: \"776c8aaa-e523-4ddb-a4f3-f786607ba536\") "
Sep 18 13:57:35.210282 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:35.210169    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/776c8aaa-e523-4ddb-a4f3-f786607ba536-istio-data\") pod \"776c8aaa-e523-4ddb-a4f3-f786607ba536\" (UID: \"776c8aaa-e523-4ddb-a4f3-f786607ba536\") "
Sep 18 13:57:35.210445 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:35.210292    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/776c8aaa-e523-4ddb-a4f3-f786607ba536-istiod-ca-cert\") pod \"776c8aaa-e523-4ddb-a4f3-f786607ba536\" (UID: \"776c8aaa-e523-4ddb-a4f3-f786607ba536\") "
Sep 18 13:57:35.210445 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:35.210353    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/776c8aaa-e523-4ddb-a4f3-f786607ba536-credential-socket\") pod \"776c8aaa-e523-4ddb-a4f3-f786607ba536\" (UID: \"776c8aaa-e523-4ddb-a4f3-f786607ba536\") "
Sep 18 13:57:35.210445 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:35.210403    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/776c8aaa-e523-4ddb-a4f3-f786607ba536-istio-token\") pod \"776c8aaa-e523-4ddb-a4f3-f786607ba536\" (UID: \"776c8aaa-e523-4ddb-a4f3-f786607ba536\") "
Sep 18 13:57:35.210445 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:35.210413    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/776c8aaa-e523-4ddb-a4f3-f786607ba536-istio-data" (OuterVolumeSpecName: "istio-data") pod "776c8aaa-e523-4ddb-a4f3-f786607ba536" (UID: "776c8aaa-e523-4ddb-a4f3-f786607ba536"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:57:35.210445 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:35.210426    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/776c8aaa-e523-4ddb-a4f3-f786607ba536-workload-certs\") pod \"776c8aaa-e523-4ddb-a4f3-f786607ba536\" (UID: \"776c8aaa-e523-4ddb-a4f3-f786607ba536\") "
Sep 18 13:57:35.210445 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:35.210431    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/776c8aaa-e523-4ddb-a4f3-f786607ba536-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "776c8aaa-e523-4ddb-a4f3-f786607ba536" (UID: "776c8aaa-e523-4ddb-a4f3-f786607ba536"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:57:35.210780 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:35.210463    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/776c8aaa-e523-4ddb-a4f3-f786607ba536-istio-envoy\") pod \"776c8aaa-e523-4ddb-a4f3-f786607ba536\" (UID: \"776c8aaa-e523-4ddb-a4f3-f786607ba536\") "
Sep 18 13:57:35.210780 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:35.210657    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/776c8aaa-e523-4ddb-a4f3-f786607ba536-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "776c8aaa-e523-4ddb-a4f3-f786607ba536" (UID: "776c8aaa-e523-4ddb-a4f3-f786607ba536"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:57:35.210780 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:35.210725    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/776c8aaa-e523-4ddb-a4f3-f786607ba536-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "776c8aaa-e523-4ddb-a4f3-f786607ba536" (UID: "776c8aaa-e523-4ddb-a4f3-f786607ba536"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:57:35.210780 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:35.210771    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/776c8aaa-e523-4ddb-a4f3-f786607ba536-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "776c8aaa-e523-4ddb-a4f3-f786607ba536" (UID: "776c8aaa-e523-4ddb-a4f3-f786607ba536"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:57:35.210973 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:35.210805    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/776c8aaa-e523-4ddb-a4f3-f786607ba536-workload-socket\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:57:35.210973 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:35.210851    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/776c8aaa-e523-4ddb-a4f3-f786607ba536-istio-data\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:57:35.210973 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:35.210866    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/776c8aaa-e523-4ddb-a4f3-f786607ba536-credential-socket\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:57:35.210973 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:35.210879    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/776c8aaa-e523-4ddb-a4f3-f786607ba536-workload-certs\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:57:35.212487 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:35.212464    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/776c8aaa-e523-4ddb-a4f3-f786607ba536-istio-token" (OuterVolumeSpecName: "istio-token") pod "776c8aaa-e523-4ddb-a4f3-f786607ba536" (UID: "776c8aaa-e523-4ddb-a4f3-f786607ba536"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:57:35.212586 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:35.212508    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/776c8aaa-e523-4ddb-a4f3-f786607ba536-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "776c8aaa-e523-4ddb-a4f3-f786607ba536" (UID: "776c8aaa-e523-4ddb-a4f3-f786607ba536"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:57:35.212897 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:35.212881    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/776c8aaa-e523-4ddb-a4f3-f786607ba536-kube-api-access-wrslc" (OuterVolumeSpecName: "kube-api-access-wrslc") pod "776c8aaa-e523-4ddb-a4f3-f786607ba536" (UID: "776c8aaa-e523-4ddb-a4f3-f786607ba536"). InnerVolumeSpecName "kube-api-access-wrslc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:57:35.212978 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:35.212896    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/776c8aaa-e523-4ddb-a4f3-f786607ba536-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "776c8aaa-e523-4ddb-a4f3-f786607ba536" (UID: "776c8aaa-e523-4ddb-a4f3-f786607ba536"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 13:57:35.311873 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:35.311808    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrslc\" (UniqueName: \"kubernetes.io/projected/776c8aaa-e523-4ddb-a4f3-f786607ba536-kube-api-access-wrslc\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:57:35.311873 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:35.311870    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/776c8aaa-e523-4ddb-a4f3-f786607ba536-istio-podinfo\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:57:35.311873 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:35.311880    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/776c8aaa-e523-4ddb-a4f3-f786607ba536-istiod-ca-cert\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:57:35.312100 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:35.311889    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/776c8aaa-e523-4ddb-a4f3-f786607ba536-istio-token\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:57:35.312100 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:35.311898    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/776c8aaa-e523-4ddb-a4f3-f786607ba536-istio-envoy\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:57:35.421727 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:35.421618    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:57:35.514611 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:35.514581    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:57:35.534345 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:35.534305    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-feff2644-84b6cf597c-5rw8v"]
Sep 18 13:57:35.534672 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:35.534660    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51e2a7c9-fc1b-417d-b598-866bdcc4e0f6" containerName="maas-api"
Sep 18 13:57:35.534719 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:35.534674    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="51e2a7c9-fc1b-417d-b598-866bdcc4e0f6" containerName="maas-api"
Sep 18 13:57:35.534719 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:35.534695    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="776c8aaa-e523-4ddb-a4f3-f786607ba536" containerName="istio-proxy"
Sep 18 13:57:35.534719 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:35.534701    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="776c8aaa-e523-4ddb-a4f3-f786607ba536" containerName="istio-proxy"
Sep 18 13:57:35.534719 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:35.534709    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="776c8aaa-e523-4ddb-a4f3-f786607ba536" containerName="istio-proxy"
Sep 18 13:57:35.534719 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:35.534714    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="776c8aaa-e523-4ddb-a4f3-f786607ba536" containerName="istio-proxy"
Sep 18 13:57:35.534912 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:35.534782    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="776c8aaa-e523-4ddb-a4f3-f786607ba536" containerName="istio-proxy"
Sep 18 13:57:35.534912 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:35.534793    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="51e2a7c9-fc1b-417d-b598-866bdcc4e0f6" containerName="maas-api"
Sep 18 13:57:35.538177 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:35.538158    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-feff2644-84b6cf597c-5rw8v"
Sep 18 13:57:35.541048 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:35.541025    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-feff2644\""
Sep 18 13:57:35.553228 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:35.553205    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-feff2644-84b6cf597c-5rw8v"]
Sep 18 13:57:35.614006 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:35.613974    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gr2p\" (UniqueName: \"kubernetes.io/projected/8906d1c0-8d9d-47c6-bdeb-d53aca74499a-kube-api-access-2gr2p\") pod \"payload-processing-e2e-ait-feff2644-84b6cf597c-5rw8v\" (UID: \"8906d1c0-8d9d-47c6-bdeb-d53aca74499a\") " pod="openshift-ingress/payload-processing-e2e-ait-feff2644-84b6cf597c-5rw8v"
Sep 18 13:57:35.614175 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:35.614013    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8906d1c0-8d9d-47c6-bdeb-d53aca74499a-plugins-config-volume\") pod \"payload-processing-e2e-ait-feff2644-84b6cf597c-5rw8v\" (UID: \"8906d1c0-8d9d-47c6-bdeb-d53aca74499a\") " pod="openshift-ingress/payload-processing-e2e-ait-feff2644-84b6cf597c-5rw8v"
Sep 18 13:57:35.714683 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:35.714582    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gr2p\" (UniqueName: \"kubernetes.io/projected/8906d1c0-8d9d-47c6-bdeb-d53aca74499a-kube-api-access-2gr2p\") pod \"payload-processing-e2e-ait-feff2644-84b6cf597c-5rw8v\" (UID: \"8906d1c0-8d9d-47c6-bdeb-d53aca74499a\") " pod="openshift-ingress/payload-processing-e2e-ait-feff2644-84b6cf597c-5rw8v"
Sep 18 13:57:35.714683 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:35.714638    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8906d1c0-8d9d-47c6-bdeb-d53aca74499a-plugins-config-volume\") pod \"payload-processing-e2e-ait-feff2644-84b6cf597c-5rw8v\" (UID: \"8906d1c0-8d9d-47c6-bdeb-d53aca74499a\") " pod="openshift-ingress/payload-processing-e2e-ait-feff2644-84b6cf597c-5rw8v"
Sep 18 13:57:35.715350 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:35.715331    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8906d1c0-8d9d-47c6-bdeb-d53aca74499a-plugins-config-volume\") pod \"payload-processing-e2e-ait-feff2644-84b6cf597c-5rw8v\" (UID: \"8906d1c0-8d9d-47c6-bdeb-d53aca74499a\") " pod="openshift-ingress/payload-processing-e2e-ait-feff2644-84b6cf597c-5rw8v"
Sep 18 13:57:35.722575 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:35.722553    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gr2p\" (UniqueName: \"kubernetes.io/projected/8906d1c0-8d9d-47c6-bdeb-d53aca74499a-kube-api-access-2gr2p\") pod \"payload-processing-e2e-ait-feff2644-84b6cf597c-5rw8v\" (UID: \"8906d1c0-8d9d-47c6-bdeb-d53aca74499a\") " pod="openshift-ingress/payload-processing-e2e-ait-feff2644-84b6cf597c-5rw8v"
Sep 18 13:57:35.836375 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:35.836316    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:57:35.849244 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:35.849210    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-feff2644-84b6cf597c-5rw8v"
Sep 18 13:57:36.099217 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:36.099180    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv" event={"ID":"776c8aaa-e523-4ddb-a4f3-f786607ba536","Type":"ContainerDied","Data":"5124dc7a2b7e25fc00f77fc2ac14e29a20054aee41a541d5922538e1ead393af"}
Sep 18 13:57:36.099634 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:36.099234    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv"
Sep 18 13:57:36.099634 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:36.099249    2577 scope.go:117] "RemoveContainer" containerID="3ad3b0ffd6a1f016f4e8ae1102f3d3d63a573a26f085b80592c57913d28320c0"
Sep 18 13:57:36.136523 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:36.136490    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv"]
Sep 18 13:57:36.143664 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:36.143633    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-feff2644-84b6cf597c-5rw8v"]
Sep 18 13:57:36.145485 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:57:36.145458    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8906d1c0_8d9d_47c6_bdeb_d53aca74499a.slice/crio-f0259cdebf310b1e17e7be15b0e7662b13fbf40ef4ab5fbbb21c9f287a0cb0ee WatchSource:0}: Error finding container f0259cdebf310b1e17e7be15b0e7662b13fbf40ef4ab5fbbb21c9f287a0cb0ee: Status 404 returned error can't find the container with id f0259cdebf310b1e17e7be15b0e7662b13fbf40ef4ab5fbbb21c9f287a0cb0ee
Sep 18 13:57:36.146106 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:36.146088    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-3675964e-gw-openshift-default-77f6cdf7d8-2tptv"]
Sep 18 13:57:36.738685 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:36.738650    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:57:37.115564 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:37.115531    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="776c8aaa-e523-4ddb-a4f3-f786607ba536" path="/var/lib/kubelet/pods/776c8aaa-e523-4ddb-a4f3-f786607ba536/volumes"
Sep 18 13:57:37.116029 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:37.115804    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-feff2644-84b6cf597c-5rw8v"
Sep 18 13:57:37.116029 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:37.115870    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-feff2644-84b6cf597c-5rw8v" event={"ID":"8906d1c0-8d9d-47c6-bdeb-d53aca74499a","Type":"ContainerStarted","Data":"91268599afb7e50dfbbad2e1d63eb73bd47ce527a3fd4dca76bc04b03280af67"}
Sep 18 13:57:37.116029 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:37.115905    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-feff2644-84b6cf597c-5rw8v" event={"ID":"8906d1c0-8d9d-47c6-bdeb-d53aca74499a","Type":"ContainerStarted","Data":"f0259cdebf310b1e17e7be15b0e7662b13fbf40ef4ab5fbbb21c9f287a0cb0ee"}
Sep 18 13:57:37.134464 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:37.134409    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-feff2644-84b6cf597c-5rw8v" podStartSLOduration=2.134393463 podStartE2EDuration="2.134393463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:57:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:57:37.13142407 +0000 UTC m=+806.184083398" watchObservedRunningTime="2026-09-18 13:57:37.134393463 +0000 UTC m=+806.187052789"
Sep 18 13:57:37.414876 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:37.414765    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:57:38.145336 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:38.145285    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:57:38.400529 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:38.400435    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-ed4c53/simulator-w0-ed4c53-kserve-989f79bd5-z4lwd"]
Sep 18 13:57:38.400894 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:38.400880    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="776c8aaa-e523-4ddb-a4f3-f786607ba536" containerName="istio-proxy"
Sep 18 13:57:38.404148 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:38.404132    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ed4c53/simulator-w0-ed4c53-kserve-989f79bd5-z4lwd"
Sep 18 13:57:38.406619 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:38.406586    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ed4c53\"/\"openshift-service-ca.crt\""
Sep 18 13:57:38.406619 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:38.406601    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ed4c53\"/\"kube-root-ca.crt\""
Sep 18 13:57:38.407345 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:38.407323    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ed4c53\"/\"default-dockercfg-ghc9c\""
Sep 18 13:57:38.407462 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:38.407386    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ed4c53\"/\"simulator-w0-ed4c53-kserve-self-signed-certs\""
Sep 18 13:57:38.413235 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:38.413212    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ed4c53/simulator-w0-ed4c53-kserve-989f79bd5-z4lwd"]
Sep 18 13:57:38.540587 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:38.540551    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e-home\") pod \"simulator-w0-ed4c53-kserve-989f79bd5-z4lwd\" (UID: \"3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e\") " pod="e2e-models-e2e-worker-w0-ed4c53/simulator-w0-ed4c53-kserve-989f79bd5-z4lwd"
Sep 18 13:57:38.540587 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:38.540588    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e-model-cache\") pod \"simulator-w0-ed4c53-kserve-989f79bd5-z4lwd\" (UID: \"3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e\") " pod="e2e-models-e2e-worker-w0-ed4c53/simulator-w0-ed4c53-kserve-989f79bd5-z4lwd"
Sep 18 13:57:38.540837 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:38.540608    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e-dshm\") pod \"simulator-w0-ed4c53-kserve-989f79bd5-z4lwd\" (UID: \"3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e\") " pod="e2e-models-e2e-worker-w0-ed4c53/simulator-w0-ed4c53-kserve-989f79bd5-z4lwd"
Sep 18 13:57:38.540837 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:38.540624    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e-tls-certs\") pod \"simulator-w0-ed4c53-kserve-989f79bd5-z4lwd\" (UID: \"3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e\") " pod="e2e-models-e2e-worker-w0-ed4c53/simulator-w0-ed4c53-kserve-989f79bd5-z4lwd"
Sep 18 13:57:38.540837 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:38.540731    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e-tmp-dir\") pod \"simulator-w0-ed4c53-kserve-989f79bd5-z4lwd\" (UID: \"3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e\") " pod="e2e-models-e2e-worker-w0-ed4c53/simulator-w0-ed4c53-kserve-989f79bd5-z4lwd"
Sep 18 13:57:38.540837 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:38.540786    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xh26\" (UniqueName: \"kubernetes.io/projected/3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e-kube-api-access-4xh26\") pod \"simulator-w0-ed4c53-kserve-989f79bd5-z4lwd\" (UID: \"3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e\") " pod="e2e-models-e2e-worker-w0-ed4c53/simulator-w0-ed4c53-kserve-989f79bd5-z4lwd"
Sep 18 13:57:38.641546 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:38.641495    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e-home\") pod \"simulator-w0-ed4c53-kserve-989f79bd5-z4lwd\" (UID: \"3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e\") " pod="e2e-models-e2e-worker-w0-ed4c53/simulator-w0-ed4c53-kserve-989f79bd5-z4lwd"
Sep 18 13:57:38.641546 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:38.641543    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e-model-cache\") pod \"simulator-w0-ed4c53-kserve-989f79bd5-z4lwd\" (UID: \"3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e\") " pod="e2e-models-e2e-worker-w0-ed4c53/simulator-w0-ed4c53-kserve-989f79bd5-z4lwd"
Sep 18 13:57:38.641798 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:38.641567    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e-dshm\") pod \"simulator-w0-ed4c53-kserve-989f79bd5-z4lwd\" (UID: \"3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e\") " pod="e2e-models-e2e-worker-w0-ed4c53/simulator-w0-ed4c53-kserve-989f79bd5-z4lwd"
Sep 18 13:57:38.641798 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:38.641595    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e-tls-certs\") pod \"simulator-w0-ed4c53-kserve-989f79bd5-z4lwd\" (UID: \"3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e\") " pod="e2e-models-e2e-worker-w0-ed4c53/simulator-w0-ed4c53-kserve-989f79bd5-z4lwd"
Sep 18 13:57:38.641798 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:38.641627    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e-tmp-dir\") pod \"simulator-w0-ed4c53-kserve-989f79bd5-z4lwd\" (UID: \"3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e\") " pod="e2e-models-e2e-worker-w0-ed4c53/simulator-w0-ed4c53-kserve-989f79bd5-z4lwd"
Sep 18 13:57:38.641798 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:38.641663    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xh26\" (UniqueName: \"kubernetes.io/projected/3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e-kube-api-access-4xh26\") pod \"simulator-w0-ed4c53-kserve-989f79bd5-z4lwd\" (UID: \"3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e\") " pod="e2e-models-e2e-worker-w0-ed4c53/simulator-w0-ed4c53-kserve-989f79bd5-z4lwd"
Sep 18 13:57:38.642036 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:38.641989    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e-home\") pod \"simulator-w0-ed4c53-kserve-989f79bd5-z4lwd\" (UID: \"3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e\") " pod="e2e-models-e2e-worker-w0-ed4c53/simulator-w0-ed4c53-kserve-989f79bd5-z4lwd"
Sep 18 13:57:38.642036 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:38.642006    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e-model-cache\") pod \"simulator-w0-ed4c53-kserve-989f79bd5-z4lwd\" (UID: \"3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e\") " pod="e2e-models-e2e-worker-w0-ed4c53/simulator-w0-ed4c53-kserve-989f79bd5-z4lwd"
Sep 18 13:57:38.642146 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:38.642104    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e-tmp-dir\") pod \"simulator-w0-ed4c53-kserve-989f79bd5-z4lwd\" (UID: \"3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e\") " pod="e2e-models-e2e-worker-w0-ed4c53/simulator-w0-ed4c53-kserve-989f79bd5-z4lwd"
Sep 18 13:57:38.643975 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:38.643949    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e-dshm\") pod \"simulator-w0-ed4c53-kserve-989f79bd5-z4lwd\" (UID: \"3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e\") " pod="e2e-models-e2e-worker-w0-ed4c53/simulator-w0-ed4c53-kserve-989f79bd5-z4lwd"
Sep 18 13:57:38.644190 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:38.644167    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e-tls-certs\") pod \"simulator-w0-ed4c53-kserve-989f79bd5-z4lwd\" (UID: \"3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e\") " pod="e2e-models-e2e-worker-w0-ed4c53/simulator-w0-ed4c53-kserve-989f79bd5-z4lwd"
Sep 18 13:57:38.650453 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:38.650434    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xh26\" (UniqueName: \"kubernetes.io/projected/3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e-kube-api-access-4xh26\") pod \"simulator-w0-ed4c53-kserve-989f79bd5-z4lwd\" (UID: \"3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e\") " pod="e2e-models-e2e-worker-w0-ed4c53/simulator-w0-ed4c53-kserve-989f79bd5-z4lwd"
Sep 18 13:57:38.714735 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:38.714634    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ed4c53/simulator-w0-ed4c53-kserve-989f79bd5-z4lwd"
Sep 18 13:57:38.785425 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:38.785398    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:57:38.995494 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:38.995460    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ed4c53/simulator-w0-ed4c53-kserve-989f79bd5-z4lwd"]
Sep 18 13:57:38.997753 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:57:38.997718    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c6a9f4b_ddd1_4ff7_a981_36f74c8e941e.slice/crio-4efcde98fa0633697d78afb9f3f4ed8d62f2cc2bc0a5ef1e5c483bc9b109ffeb WatchSource:0}: Error finding container 4efcde98fa0633697d78afb9f3f4ed8d62f2cc2bc0a5ef1e5c483bc9b109ffeb: Status 404 returned error can't find the container with id 4efcde98fa0633697d78afb9f3f4ed8d62f2cc2bc0a5ef1e5c483bc9b109ffeb
Sep 18 13:57:39.127519 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:39.127482    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ed4c53/simulator-w0-ed4c53-kserve-989f79bd5-z4lwd" event={"ID":"3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e","Type":"ContainerStarted","Data":"27d4b8142f62fac170c9503503fbb18a7469eaf10731424c23a6e5f4114031c8"}
Sep 18 13:57:39.127665 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:39.127525    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ed4c53/simulator-w0-ed4c53-kserve-989f79bd5-z4lwd" event={"ID":"3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e","Type":"ContainerStarted","Data":"4efcde98fa0633697d78afb9f3f4ed8d62f2cc2bc0a5ef1e5c483bc9b109ffeb"}
Sep 18 13:57:39.150713 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:39.150660    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-ed4c53/simulator-w0-ed4c53-kserve-989f79bd5-z4lwd" podStartSLOduration=1.150640795 podStartE2EDuration="1.150640795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:57:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:57:39.147479825 +0000 UTC m=+808.200139151" watchObservedRunningTime="2026-09-18 13:57:39.150640795 +0000 UTC m=+808.203300121"
Sep 18 13:57:48.122149 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:48.122118    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-feff2644-84b6cf597c-5rw8v"
Sep 18 13:57:48.715004 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:48.714962    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ed4c53/simulator-w0-ed4c53-kserve-989f79bd5-z4lwd"
Sep 18 13:57:48.715004 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:48.715012    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-ed4c53/simulator-w0-ed4c53-kserve-989f79bd5-z4lwd"
Sep 18 13:57:48.727380 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:48.727350    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-ed4c53/simulator-w0-ed4c53-kserve-989f79bd5-z4lwd"
Sep 18 13:57:49.195345 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:49.195318    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ed4c53/simulator-w0-ed4c53-kserve-989f79bd5-z4lwd"
Sep 18 13:57:50.079003 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:50.078926    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-70f06e/autores-model-70f06e-kserve-6cc99667d7-xktzd"]
Sep 18 13:57:50.082510 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:50.082494    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-70f06e/autores-model-70f06e-kserve-6cc99667d7-xktzd"
Sep 18 13:57:50.084788 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:50.084765    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-70f06e\"/\"openshift-service-ca.crt\""
Sep 18 13:57:50.085224 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:50.085210    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-70f06e\"/\"kube-root-ca.crt\""
Sep 18 13:57:50.085420 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:50.085400    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-70f06e\"/\"default-dockercfg-tm7lm\""
Sep 18 13:57:50.085486 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:50.085400    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-70f06e\"/\"autores-model-70f06e-kserve-self-signed-certs\""
Sep 18 13:57:50.091244 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:50.091223    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-70f06e/autores-model-70f06e-kserve-6cc99667d7-xktzd"]
Sep 18 13:57:50.186237 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:50.186209    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:57:50.250203 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:50.250172    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d8fb485-97ad-4126-9067-4f19179b51f5-tmp-dir\") pod \"autores-model-70f06e-kserve-6cc99667d7-xktzd\" (UID: \"1d8fb485-97ad-4126-9067-4f19179b51f5\") " pod="ai-tenant-e2e-autores-70f06e/autores-model-70f06e-kserve-6cc99667d7-xktzd"
Sep 18 13:57:50.250691 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:50.250228    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d8fb485-97ad-4126-9067-4f19179b51f5-tls-certs\") pod \"autores-model-70f06e-kserve-6cc99667d7-xktzd\" (UID: \"1d8fb485-97ad-4126-9067-4f19179b51f5\") " pod="ai-tenant-e2e-autores-70f06e/autores-model-70f06e-kserve-6cc99667d7-xktzd"
Sep 18 13:57:50.250691 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:50.250322    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spdzt\" (UniqueName: \"kubernetes.io/projected/1d8fb485-97ad-4126-9067-4f19179b51f5-kube-api-access-spdzt\") pod \"autores-model-70f06e-kserve-6cc99667d7-xktzd\" (UID: \"1d8fb485-97ad-4126-9067-4f19179b51f5\") " pod="ai-tenant-e2e-autores-70f06e/autores-model-70f06e-kserve-6cc99667d7-xktzd"
Sep 18 13:57:50.250691 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:50.250366    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d8fb485-97ad-4126-9067-4f19179b51f5-home\") pod \"autores-model-70f06e-kserve-6cc99667d7-xktzd\" (UID: \"1d8fb485-97ad-4126-9067-4f19179b51f5\") " pod="ai-tenant-e2e-autores-70f06e/autores-model-70f06e-kserve-6cc99667d7-xktzd"
Sep 18 13:57:50.250691 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:50.250478    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d8fb485-97ad-4126-9067-4f19179b51f5-dshm\") pod \"autores-model-70f06e-kserve-6cc99667d7-xktzd\" (UID: \"1d8fb485-97ad-4126-9067-4f19179b51f5\") " pod="ai-tenant-e2e-autores-70f06e/autores-model-70f06e-kserve-6cc99667d7-xktzd"
Sep 18 13:57:50.250691 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:50.250523    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d8fb485-97ad-4126-9067-4f19179b51f5-model-cache\") pod \"autores-model-70f06e-kserve-6cc99667d7-xktzd\" (UID: \"1d8fb485-97ad-4126-9067-4f19179b51f5\") " pod="ai-tenant-e2e-autores-70f06e/autores-model-70f06e-kserve-6cc99667d7-xktzd"
Sep 18 13:57:50.351918 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:50.351883    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d8fb485-97ad-4126-9067-4f19179b51f5-dshm\") pod \"autores-model-70f06e-kserve-6cc99667d7-xktzd\" (UID: \"1d8fb485-97ad-4126-9067-4f19179b51f5\") " pod="ai-tenant-e2e-autores-70f06e/autores-model-70f06e-kserve-6cc99667d7-xktzd"
Sep 18 13:57:50.352097 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:50.351928    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d8fb485-97ad-4126-9067-4f19179b51f5-model-cache\") pod \"autores-model-70f06e-kserve-6cc99667d7-xktzd\" (UID: \"1d8fb485-97ad-4126-9067-4f19179b51f5\") " pod="ai-tenant-e2e-autores-70f06e/autores-model-70f06e-kserve-6cc99667d7-xktzd"
Sep 18 13:57:50.352097 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:50.352034    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d8fb485-97ad-4126-9067-4f19179b51f5-tmp-dir\") pod \"autores-model-70f06e-kserve-6cc99667d7-xktzd\" (UID: \"1d8fb485-97ad-4126-9067-4f19179b51f5\") " pod="ai-tenant-e2e-autores-70f06e/autores-model-70f06e-kserve-6cc99667d7-xktzd"
Sep 18 13:57:50.352226 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:50.352157    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d8fb485-97ad-4126-9067-4f19179b51f5-tls-certs\") pod \"autores-model-70f06e-kserve-6cc99667d7-xktzd\" (UID: \"1d8fb485-97ad-4126-9067-4f19179b51f5\") " pod="ai-tenant-e2e-autores-70f06e/autores-model-70f06e-kserve-6cc99667d7-xktzd"
Sep 18 13:57:50.352287 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:50.352222    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spdzt\" (UniqueName: \"kubernetes.io/projected/1d8fb485-97ad-4126-9067-4f19179b51f5-kube-api-access-spdzt\") pod \"autores-model-70f06e-kserve-6cc99667d7-xktzd\" (UID: \"1d8fb485-97ad-4126-9067-4f19179b51f5\") " pod="ai-tenant-e2e-autores-70f06e/autores-model-70f06e-kserve-6cc99667d7-xktzd"
Sep 18 13:57:50.352287 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:50.352262    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d8fb485-97ad-4126-9067-4f19179b51f5-home\") pod \"autores-model-70f06e-kserve-6cc99667d7-xktzd\" (UID: \"1d8fb485-97ad-4126-9067-4f19179b51f5\") " pod="ai-tenant-e2e-autores-70f06e/autores-model-70f06e-kserve-6cc99667d7-xktzd"
Sep 18 13:57:50.352423 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:50.352398    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d8fb485-97ad-4126-9067-4f19179b51f5-tmp-dir\") pod \"autores-model-70f06e-kserve-6cc99667d7-xktzd\" (UID: \"1d8fb485-97ad-4126-9067-4f19179b51f5\") " pod="ai-tenant-e2e-autores-70f06e/autores-model-70f06e-kserve-6cc99667d7-xktzd"
Sep 18 13:57:50.352498 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:50.352420    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d8fb485-97ad-4126-9067-4f19179b51f5-model-cache\") pod \"autores-model-70f06e-kserve-6cc99667d7-xktzd\" (UID: \"1d8fb485-97ad-4126-9067-4f19179b51f5\") " pod="ai-tenant-e2e-autores-70f06e/autores-model-70f06e-kserve-6cc99667d7-xktzd"
Sep 18 13:57:50.352646 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:50.352624    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d8fb485-97ad-4126-9067-4f19179b51f5-home\") pod \"autores-model-70f06e-kserve-6cc99667d7-xktzd\" (UID: \"1d8fb485-97ad-4126-9067-4f19179b51f5\") " pod="ai-tenant-e2e-autores-70f06e/autores-model-70f06e-kserve-6cc99667d7-xktzd"
Sep 18 13:57:50.354638 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:50.354613    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d8fb485-97ad-4126-9067-4f19179b51f5-dshm\") pod \"autores-model-70f06e-kserve-6cc99667d7-xktzd\" (UID: \"1d8fb485-97ad-4126-9067-4f19179b51f5\") " pod="ai-tenant-e2e-autores-70f06e/autores-model-70f06e-kserve-6cc99667d7-xktzd"
Sep 18 13:57:50.354741 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:50.354616    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d8fb485-97ad-4126-9067-4f19179b51f5-tls-certs\") pod \"autores-model-70f06e-kserve-6cc99667d7-xktzd\" (UID: \"1d8fb485-97ad-4126-9067-4f19179b51f5\") " pod="ai-tenant-e2e-autores-70f06e/autores-model-70f06e-kserve-6cc99667d7-xktzd"
Sep 18 13:57:50.360484 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:50.360444    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-spdzt\" (UniqueName: \"kubernetes.io/projected/1d8fb485-97ad-4126-9067-4f19179b51f5-kube-api-access-spdzt\") pod \"autores-model-70f06e-kserve-6cc99667d7-xktzd\" (UID: \"1d8fb485-97ad-4126-9067-4f19179b51f5\") " pod="ai-tenant-e2e-autores-70f06e/autores-model-70f06e-kserve-6cc99667d7-xktzd"
Sep 18 13:57:50.392349 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:50.392313    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-70f06e/autores-model-70f06e-kserve-6cc99667d7-xktzd"
Sep 18 13:57:50.668275 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:50.668242    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-70f06e/autores-model-70f06e-kserve-6cc99667d7-xktzd"]
Sep 18 13:57:50.670337 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:57:50.670306    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1d8fb485_97ad_4126_9067_4f19179b51f5.slice/crio-623b36947181e239f795d605786f12c8eacd662e775147b014746032eec4152d WatchSource:0}: Error finding container 623b36947181e239f795d605786f12c8eacd662e775147b014746032eec4152d: Status 404 returned error can't find the container with id 623b36947181e239f795d605786f12c8eacd662e775147b014746032eec4152d
Sep 18 13:57:51.197346 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:51.197308    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-70f06e/autores-model-70f06e-kserve-6cc99667d7-xktzd" event={"ID":"1d8fb485-97ad-4126-9067-4f19179b51f5","Type":"ContainerStarted","Data":"9fb8707c99c755bb259ec0c6a7f9be1d9337e92f0c86fe86489ecef43f1a5e58"}
Sep 18 13:57:51.197530 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:51.197353    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-70f06e/autores-model-70f06e-kserve-6cc99667d7-xktzd" event={"ID":"1d8fb485-97ad-4126-9067-4f19179b51f5","Type":"ContainerStarted","Data":"623b36947181e239f795d605786f12c8eacd662e775147b014746032eec4152d"}
Sep 18 13:57:51.215762 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:51.215713    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-70f06e/autores-model-70f06e-kserve-6cc99667d7-xktzd" podStartSLOduration=1.215695534 podStartE2EDuration="1.215695534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:57:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:57:51.213888416 +0000 UTC m=+820.266547745" watchObservedRunningTime="2026-09-18 13:57:51.215695534 +0000 UTC m=+820.268354860"
Sep 18 13:57:52.404098 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:52.404058    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 13:57:56.837755 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:56.837700    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:57:57.676989 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:57.676952    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:57:58.366035 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:58.366000    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:57:58.743239 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:58.743138    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:57:59.238667 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:57:59.238637    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:00.033769 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:00.033726    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:00.393335 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:00.393298    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-70f06e/autores-model-70f06e-kserve-6cc99667d7-xktzd"
Sep 18 13:58:00.393604 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:00.393582    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-70f06e/autores-model-70f06e-kserve-6cc99667d7-xktzd"
Sep 18 13:58:00.405966 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:00.405941    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-70f06e/autores-model-70f06e-kserve-6cc99667d7-xktzd"
Sep 18 13:58:00.691551 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:00.691455    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:01.265477 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:01.265449    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-70f06e/autores-model-70f06e-kserve-6cc99667d7-xktzd"
Sep 18 13:58:01.433337 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:01.433294    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:02.332979 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:02.332932    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:03.053817 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:03.053771    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:03.855547 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:03.855510    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:04.496374 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:04.496332    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:04.529928 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:04.529889    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-feff2644-84b6cf597c-5rw8v"]
Sep 18 13:58:04.530201 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:04.530166    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-feff2644-84b6cf597c-5rw8v" podUID="8906d1c0-8d9d-47c6-bdeb-d53aca74499a" containerName="payload-processing" containerID="cri-o://91268599afb7e50dfbbad2e1d63eb73bd47ce527a3fd4dca76bc04b03280af67" gracePeriod=30
Sep 18 13:58:04.599646 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:04.599611    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:04.705807 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:04.705778    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:04.788330 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:04.788299    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-feff2644-84b6cf597c-5rw8v"
Sep 18 13:58:04.879251 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:04.879223    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:04.882347 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:04.882319    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8906d1c0-8d9d-47c6-bdeb-d53aca74499a-plugins-config-volume\") pod \"8906d1c0-8d9d-47c6-bdeb-d53aca74499a\" (UID: \"8906d1c0-8d9d-47c6-bdeb-d53aca74499a\") "
Sep 18 13:58:04.882491 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:04.882397    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2gr2p\" (UniqueName: \"kubernetes.io/projected/8906d1c0-8d9d-47c6-bdeb-d53aca74499a-kube-api-access-2gr2p\") pod \"8906d1c0-8d9d-47c6-bdeb-d53aca74499a\" (UID: \"8906d1c0-8d9d-47c6-bdeb-d53aca74499a\") "
Sep 18 13:58:04.882720 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:04.882695    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8906d1c0-8d9d-47c6-bdeb-d53aca74499a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8906d1c0-8d9d-47c6-bdeb-d53aca74499a" (UID: "8906d1c0-8d9d-47c6-bdeb-d53aca74499a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:58:04.884454 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:04.884430    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8906d1c0-8d9d-47c6-bdeb-d53aca74499a-kube-api-access-2gr2p" (OuterVolumeSpecName: "kube-api-access-2gr2p") pod "8906d1c0-8d9d-47c6-bdeb-d53aca74499a" (UID: "8906d1c0-8d9d-47c6-bdeb-d53aca74499a"). InnerVolumeSpecName "kube-api-access-2gr2p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:58:04.983992 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:04.983947    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gr2p\" (UniqueName: \"kubernetes.io/projected/8906d1c0-8d9d-47c6-bdeb-d53aca74499a-kube-api-access-2gr2p\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:58:04.983992 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:04.983983    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8906d1c0-8d9d-47c6-bdeb-d53aca74499a-plugins-config-volume\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:58:05.251365 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:05.251328    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:05.276553 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:05.276520    2577 generic.go:356] "Generic (PLEG): container finished" podID="8906d1c0-8d9d-47c6-bdeb-d53aca74499a" containerID="91268599afb7e50dfbbad2e1d63eb73bd47ce527a3fd4dca76bc04b03280af67" exitCode=0
Sep 18 13:58:05.276706 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:05.276568    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-feff2644-84b6cf597c-5rw8v"
Sep 18 13:58:05.276706 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:05.276602    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-feff2644-84b6cf597c-5rw8v" event={"ID":"8906d1c0-8d9d-47c6-bdeb-d53aca74499a","Type":"ContainerDied","Data":"91268599afb7e50dfbbad2e1d63eb73bd47ce527a3fd4dca76bc04b03280af67"}
Sep 18 13:58:05.276706 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:05.276655    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-feff2644-84b6cf597c-5rw8v" event={"ID":"8906d1c0-8d9d-47c6-bdeb-d53aca74499a","Type":"ContainerDied","Data":"f0259cdebf310b1e17e7be15b0e7662b13fbf40ef4ab5fbbb21c9f287a0cb0ee"}
Sep 18 13:58:05.276706 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:05.276678    2577 scope.go:117] "RemoveContainer" containerID="91268599afb7e50dfbbad2e1d63eb73bd47ce527a3fd4dca76bc04b03280af67"
Sep 18 13:58:05.285705 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:05.285687    2577 scope.go:117] "RemoveContainer" containerID="91268599afb7e50dfbbad2e1d63eb73bd47ce527a3fd4dca76bc04b03280af67"
Sep 18 13:58:05.286010 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:58:05.285992    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91268599afb7e50dfbbad2e1d63eb73bd47ce527a3fd4dca76bc04b03280af67\": container with ID starting with 91268599afb7e50dfbbad2e1d63eb73bd47ce527a3fd4dca76bc04b03280af67 not found: ID does not exist" containerID="91268599afb7e50dfbbad2e1d63eb73bd47ce527a3fd4dca76bc04b03280af67"
Sep 18 13:58:05.286082 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:05.286021    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91268599afb7e50dfbbad2e1d63eb73bd47ce527a3fd4dca76bc04b03280af67"} err="failed to get container status \"91268599afb7e50dfbbad2e1d63eb73bd47ce527a3fd4dca76bc04b03280af67\": rpc error: code = NotFound desc = could not find container \"91268599afb7e50dfbbad2e1d63eb73bd47ce527a3fd4dca76bc04b03280af67\": container with ID starting with 91268599afb7e50dfbbad2e1d63eb73bd47ce527a3fd4dca76bc04b03280af67 not found: ID does not exist"
Sep 18 13:58:05.292133 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:05.292109    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-feff2644-84b6cf597c-5rw8v"]
Sep 18 13:58:05.296776 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:05.296755    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-feff2644-84b6cf597c-5rw8v"]
Sep 18 13:58:05.877538 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:05.877498    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:05.978565 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:05.978525    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:06.290484 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:06.290399    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:06.600002 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:06.599967    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:06.800703 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:06.800661    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:07.108074 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:07.108040    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8906d1c0-8d9d-47c6-bdeb-d53aca74499a" path="/var/lib/kubelet/pods/8906d1c0-8d9d-47c6-bdeb-d53aca74499a/volumes"
Sep 18 13:58:07.300324 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:07.300289    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:07.533604 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:07.533520    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:08.241432 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:08.241401    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:09.021508 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:09.021474    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:09.573638 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:09.573601    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:09.661797 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:09.661761    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:09.826815 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:09.826723    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:10.647788 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:10.647757    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:11.393817 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:11.393787    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:11.695534 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:11.695450    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 13:58:12.224032 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:12.224003    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:12.334212 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:12.334172    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:12.865714 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:12.865680    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:13.538705 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:13.538594    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:13.639507 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:13.639473    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:13.739124 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:13.739094    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:14.008436 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:14.008395    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:14.087790 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:14.087758    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:21.169650 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:21.169609    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:21.268092 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:21.268043    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:23.186553 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:23.186510    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-70f06e-l4xw5"]
Sep 18 13:58:23.187077 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:23.187048    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8906d1c0-8d9d-47c6-bdeb-d53aca74499a" containerName="payload-processing"
Sep 18 13:58:23.187077 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:23.187069    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8906d1c0-8d9d-47c6-bdeb-d53aca74499a" containerName="payload-processing"
Sep 18 13:58:23.187226 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:23.187191    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8906d1c0-8d9d-47c6-bdeb-d53aca74499a" containerName="payload-processing"
Sep 18 13:58:23.191859 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:23.191814    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-70f06e-l4xw5"
Sep 18 13:58:23.194125 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:23.194095    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 13:58:23.194241 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:23.194205    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cxtb2\""
Sep 18 13:58:23.198041 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:23.198019    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-70f06e-l4xw5"]
Sep 18 13:58:23.248008 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:23.247973    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69528ad8-e882-4047-97dd-9ee30ae286a9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-70f06e-l4xw5\" (UID: \"69528ad8-e882-4047-97dd-9ee30ae286a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-70f06e-l4xw5"
Sep 18 13:58:23.349313 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:23.349270    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69528ad8-e882-4047-97dd-9ee30ae286a9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-70f06e-l4xw5\" (UID: \"69528ad8-e882-4047-97dd-9ee30ae286a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-70f06e-l4xw5"
Sep 18 13:58:23.349981 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:23.349957    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69528ad8-e882-4047-97dd-9ee30ae286a9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-70f06e-l4xw5\" (UID: \"69528ad8-e882-4047-97dd-9ee30ae286a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-70f06e-l4xw5"
Sep 18 13:58:23.504712 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:23.504608    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-70f06e-l4xw5"
Sep 18 13:58:23.509793 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:23.509750    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:23.783683 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:23.783661    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-70f06e-l4xw5"]
Sep 18 13:58:23.787259 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:58:23.787114    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69528ad8_e882_4047_97dd_9ee30ae286a9.slice/crio-5419b92fb533289d14ee2ffa730e67f3bbde607b4e14756dc90bdcafc157c609 WatchSource:0}: Error finding container 5419b92fb533289d14ee2ffa730e67f3bbde607b4e14756dc90bdcafc157c609: Status 404 returned error can't find the container with id 5419b92fb533289d14ee2ffa730e67f3bbde607b4e14756dc90bdcafc157c609
Sep 18 13:58:24.388445 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:24.388405    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-70f06e-l4xw5" event={"ID":"69528ad8-e882-4047-97dd-9ee30ae286a9","Type":"ContainerStarted","Data":"5419b92fb533289d14ee2ffa730e67f3bbde607b4e14756dc90bdcafc157c609"}
Sep 18 13:58:24.680062 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:24.679977    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:25.557654 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:25.557620    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:26.330179 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:26.330148    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:27.218237 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:27.218203    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:28.152635 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:28.152598    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:29.158796 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:29.158758    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:30.266882 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:30.266800    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:30.428146 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:30.428111    2577 generic.go:356] "Generic (PLEG): container finished" podID="69528ad8-e882-4047-97dd-9ee30ae286a9" containerID="6c4315fc1c2b0cb8f6fd8e80af0206e589f729002e5857938145177719e520e6" exitCode=0
Sep 18 13:58:30.428299 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:30.428203    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-70f06e-l4xw5" event={"ID":"69528ad8-e882-4047-97dd-9ee30ae286a9","Type":"ContainerDied","Data":"6c4315fc1c2b0cb8f6fd8e80af0206e589f729002e5857938145177719e520e6"}
Sep 18 13:58:31.243534 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:31.243499    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:31.308371 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:31.308333    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 13:58:31.569370 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:31.569343    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-70f06e-l4xw5"
Sep 18 13:58:31.640709 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:31.640673    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69528ad8-e882-4047-97dd-9ee30ae286a9-maas-api-service-ca\") pod \"69528ad8-e882-4047-97dd-9ee30ae286a9\" (UID: \"69528ad8-e882-4047-97dd-9ee30ae286a9\") "
Sep 18 13:58:31.641069 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:31.641046    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69528ad8-e882-4047-97dd-9ee30ae286a9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "69528ad8-e882-4047-97dd-9ee30ae286a9" (UID: "69528ad8-e882-4047-97dd-9ee30ae286a9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:58:31.742118 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:31.742077    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69528ad8-e882-4047-97dd-9ee30ae286a9-maas-api-service-ca\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:58:32.028748 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:32.028692    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:32.441128 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:32.441099    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-70f06e-l4xw5"
Sep 18 13:58:32.441128 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:32.441117    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-70f06e-l4xw5" event={"ID":"69528ad8-e882-4047-97dd-9ee30ae286a9","Type":"ContainerDied","Data":"5419b92fb533289d14ee2ffa730e67f3bbde607b4e14756dc90bdcafc157c609"}
Sep 18 13:58:32.441547 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:32.441164    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5419b92fb533289d14ee2ffa730e67f3bbde607b4e14756dc90bdcafc157c609"
Sep 18 13:58:32.965645 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:32.965602    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:33.858079 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:33.858046    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:34.777759 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:34.777719    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:35.526454 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:35.526421    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:36.125932 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:36.125903    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:36.150688 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:36.150653    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-70f06e-869f668c77-z9mx8"]
Sep 18 13:58:36.150911 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:36.150888    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-70f06e-869f668c77-z9mx8" podUID="b07a50e5-19e8-4821-bfa1-225d7160c9cc" containerName="payload-processing" containerID="cri-o://ba0b33db6de1b2ff9c74adb32575fe8bd0b574ae67d9b5510b06c63cdb53ca0a" gracePeriod=30
Sep 18 13:58:36.158207 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:36.158178    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-70f06e-8d6b74974-qshdd"]
Sep 18 13:58:36.158578 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:36.158428    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-70f06e-8d6b74974-qshdd" podUID="1730d85c-df98-4db2-99f9-951845ebbc56" containerName="payload-pre-processing" containerID="cri-o://fd708f8494d708935f9b1b11b1350404f357b567aaf9d4e37063d887aec0907a" gracePeriod=30
Sep 18 13:58:36.204674 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:36.204641    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:36.307752 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:36.307719    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:36.402768 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:36.402683    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-70f06e-l4xw5"]
Sep 18 13:58:36.411842 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:36.411796    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-70f06e-l4xw5"]
Sep 18 13:58:36.411990 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:36.411857    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:36.441154 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:36.441131    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-70f06e-869f668c77-z9mx8"
Sep 18 13:58:36.445448 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:36.445425    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-70f06e-8d6b74974-qshdd"
Sep 18 13:58:36.467956 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:36.467924    2577 generic.go:356] "Generic (PLEG): container finished" podID="1730d85c-df98-4db2-99f9-951845ebbc56" containerID="fd708f8494d708935f9b1b11b1350404f357b567aaf9d4e37063d887aec0907a" exitCode=0
Sep 18 13:58:36.468121 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:36.467971    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-70f06e-8d6b74974-qshdd"
Sep 18 13:58:36.468121 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:36.468001    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-70f06e-8d6b74974-qshdd" event={"ID":"1730d85c-df98-4db2-99f9-951845ebbc56","Type":"ContainerDied","Data":"fd708f8494d708935f9b1b11b1350404f357b567aaf9d4e37063d887aec0907a"}
Sep 18 13:58:36.468121 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:36.468055    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-70f06e-8d6b74974-qshdd" event={"ID":"1730d85c-df98-4db2-99f9-951845ebbc56","Type":"ContainerDied","Data":"c8b92f91e7ead6ad824860f6e5b4e4d042c0473bd973be66969bf50af3477b8e"}
Sep 18 13:58:36.468121 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:36.468080    2577 scope.go:117] "RemoveContainer" containerID="fd708f8494d708935f9b1b11b1350404f357b567aaf9d4e37063d887aec0907a"
Sep 18 13:58:36.469748 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:36.469724    2577 generic.go:356] "Generic (PLEG): container finished" podID="b07a50e5-19e8-4821-bfa1-225d7160c9cc" containerID="ba0b33db6de1b2ff9c74adb32575fe8bd0b574ae67d9b5510b06c63cdb53ca0a" exitCode=0
Sep 18 13:58:36.469862 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:36.469774    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-70f06e-869f668c77-z9mx8" event={"ID":"b07a50e5-19e8-4821-bfa1-225d7160c9cc","Type":"ContainerDied","Data":"ba0b33db6de1b2ff9c74adb32575fe8bd0b574ae67d9b5510b06c63cdb53ca0a"}
Sep 18 13:58:36.469862 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:36.469840    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-70f06e-869f668c77-z9mx8" event={"ID":"b07a50e5-19e8-4821-bfa1-225d7160c9cc","Type":"ContainerDied","Data":"91c4754da326ade58a6b5eec7f042d17fff8a7a1a3b0d11cccc86c4ef5851cc8"}
Sep 18 13:58:36.469943 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:36.469779    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-70f06e-869f668c77-z9mx8"
Sep 18 13:58:36.478719 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:36.478695    2577 scope.go:117] "RemoveContainer" containerID="fd708f8494d708935f9b1b11b1350404f357b567aaf9d4e37063d887aec0907a"
Sep 18 13:58:36.479108 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:58:36.479088    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd708f8494d708935f9b1b11b1350404f357b567aaf9d4e37063d887aec0907a\": container with ID starting with fd708f8494d708935f9b1b11b1350404f357b567aaf9d4e37063d887aec0907a not found: ID does not exist" containerID="fd708f8494d708935f9b1b11b1350404f357b567aaf9d4e37063d887aec0907a"
Sep 18 13:58:36.479176 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:36.479116    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd708f8494d708935f9b1b11b1350404f357b567aaf9d4e37063d887aec0907a"} err="failed to get container status \"fd708f8494d708935f9b1b11b1350404f357b567aaf9d4e37063d887aec0907a\": rpc error: code = NotFound desc = could not find container \"fd708f8494d708935f9b1b11b1350404f357b567aaf9d4e37063d887aec0907a\": container with ID starting with fd708f8494d708935f9b1b11b1350404f357b567aaf9d4e37063d887aec0907a not found: ID does not exist"
Sep 18 13:58:36.479176 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:36.479137    2577 scope.go:117] "RemoveContainer" containerID="ba0b33db6de1b2ff9c74adb32575fe8bd0b574ae67d9b5510b06c63cdb53ca0a"
Sep 18 13:58:36.491212 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:36.491188    2577 scope.go:117] "RemoveContainer" containerID="ba0b33db6de1b2ff9c74adb32575fe8bd0b574ae67d9b5510b06c63cdb53ca0a"
Sep 18 13:58:36.491524 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:58:36.491503    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba0b33db6de1b2ff9c74adb32575fe8bd0b574ae67d9b5510b06c63cdb53ca0a\": container with ID starting with ba0b33db6de1b2ff9c74adb32575fe8bd0b574ae67d9b5510b06c63cdb53ca0a not found: ID does not exist" containerID="ba0b33db6de1b2ff9c74adb32575fe8bd0b574ae67d9b5510b06c63cdb53ca0a"
Sep 18 13:58:36.491595 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:36.491531    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba0b33db6de1b2ff9c74adb32575fe8bd0b574ae67d9b5510b06c63cdb53ca0a"} err="failed to get container status \"ba0b33db6de1b2ff9c74adb32575fe8bd0b574ae67d9b5510b06c63cdb53ca0a\": rpc error: code = NotFound desc = could not find container \"ba0b33db6de1b2ff9c74adb32575fe8bd0b574ae67d9b5510b06c63cdb53ca0a\": container with ID starting with ba0b33db6de1b2ff9c74adb32575fe8bd0b574ae67d9b5510b06c63cdb53ca0a not found: ID does not exist"
Sep 18 13:58:36.501968 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:36.501938    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:36.584430 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:36.584399    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1730d85c-df98-4db2-99f9-951845ebbc56-plugins-config-volume\") pod \"1730d85c-df98-4db2-99f9-951845ebbc56\" (UID: \"1730d85c-df98-4db2-99f9-951845ebbc56\") "
Sep 18 13:58:36.584942 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:36.584438    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lc9zn\" (UniqueName: \"kubernetes.io/projected/1730d85c-df98-4db2-99f9-951845ebbc56-kube-api-access-lc9zn\") pod \"1730d85c-df98-4db2-99f9-951845ebbc56\" (UID: \"1730d85c-df98-4db2-99f9-951845ebbc56\") "
Sep 18 13:58:36.584942 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:36.584474    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b07a50e5-19e8-4821-bfa1-225d7160c9cc-plugins-config-volume\") pod \"b07a50e5-19e8-4821-bfa1-225d7160c9cc\" (UID: \"b07a50e5-19e8-4821-bfa1-225d7160c9cc\") "
Sep 18 13:58:36.584942 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:36.584500    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d7jg7\" (UniqueName: \"kubernetes.io/projected/b07a50e5-19e8-4821-bfa1-225d7160c9cc-kube-api-access-d7jg7\") pod \"b07a50e5-19e8-4821-bfa1-225d7160c9cc\" (UID: \"b07a50e5-19e8-4821-bfa1-225d7160c9cc\") "
Sep 18 13:58:36.584942 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:36.584903    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b07a50e5-19e8-4821-bfa1-225d7160c9cc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b07a50e5-19e8-4821-bfa1-225d7160c9cc" (UID: "b07a50e5-19e8-4821-bfa1-225d7160c9cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:58:36.585113 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:36.584915    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1730d85c-df98-4db2-99f9-951845ebbc56-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1730d85c-df98-4db2-99f9-951845ebbc56" (UID: "1730d85c-df98-4db2-99f9-951845ebbc56"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:58:36.586694 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:36.586668    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b07a50e5-19e8-4821-bfa1-225d7160c9cc-kube-api-access-d7jg7" (OuterVolumeSpecName: "kube-api-access-d7jg7") pod "b07a50e5-19e8-4821-bfa1-225d7160c9cc" (UID: "b07a50e5-19e8-4821-bfa1-225d7160c9cc"). InnerVolumeSpecName "kube-api-access-d7jg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:58:36.586783 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:36.586747    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1730d85c-df98-4db2-99f9-951845ebbc56-kube-api-access-lc9zn" (OuterVolumeSpecName: "kube-api-access-lc9zn") pod "1730d85c-df98-4db2-99f9-951845ebbc56" (UID: "1730d85c-df98-4db2-99f9-951845ebbc56"). InnerVolumeSpecName "kube-api-access-lc9zn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:58:36.685185 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:36.685098    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b07a50e5-19e8-4821-bfa1-225d7160c9cc-plugins-config-volume\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:58:36.685185 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:36.685126    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7jg7\" (UniqueName: \"kubernetes.io/projected/b07a50e5-19e8-4821-bfa1-225d7160c9cc-kube-api-access-d7jg7\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:58:36.685185 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:36.685137    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1730d85c-df98-4db2-99f9-951845ebbc56-plugins-config-volume\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:58:36.685185 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:36.685147    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc9zn\" (UniqueName: \"kubernetes.io/projected/1730d85c-df98-4db2-99f9-951845ebbc56-kube-api-access-lc9zn\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:58:36.737016 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:36.736978    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:36.793986 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:36.793955    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-70f06e-8d6b74974-qshdd"]
Sep 18 13:58:36.796424 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:36.796400    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-70f06e-8d6b74974-qshdd"]
Sep 18 13:58:36.806374 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:36.806344    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-70f06e-869f668c77-z9mx8"]
Sep 18 13:58:36.808304 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:36.808285    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-70f06e-869f668c77-z9mx8"]
Sep 18 13:58:37.107465 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:37.107436    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1730d85c-df98-4db2-99f9-951845ebbc56" path="/var/lib/kubelet/pods/1730d85c-df98-4db2-99f9-951845ebbc56/volumes"
Sep 18 13:58:37.107758 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:37.107745    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69528ad8-e882-4047-97dd-9ee30ae286a9" path="/var/lib/kubelet/pods/69528ad8-e882-4047-97dd-9ee30ae286a9/volumes"
Sep 18 13:58:37.108081 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:37.108069    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b07a50e5-19e8-4821-bfa1-225d7160c9cc" path="/var/lib/kubelet/pods/b07a50e5-19e8-4821-bfa1-225d7160c9cc/volumes"
Sep 18 13:58:37.471246 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:37.471157    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:38.473710 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:38.473674    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:39.381468 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:39.381437    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:40.248596 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:40.248562    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:41.060549 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:41.060517    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:41.707526 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:41.707493    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:41.754264 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:41.754228    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-70f06e/autores-model-70f06e-kserve-6cc99667d7-xktzd"]
Sep 18 13:58:41.790900 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:41.790868    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:41.878718 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:41.878668    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:41.963928 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:41.963808    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:42.037200 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:42.037167    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:42.155260 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:42.155214    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:42.550402 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:42.550357    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:42.788108 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:42.788074    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:42.889786 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:42.889752    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:42.987782 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:42.987751    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:43.387845 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:43.387790    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:44.224029 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:44.223983    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:44.330101 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:44.330065    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:45.087402 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:45.087370    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:45.793538 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:45.793502    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:50.963105 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:50.963061    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:51.075279 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:51.075240    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:51.164110 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:51.164073    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:51.264040 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:51.263954    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:51.638904 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:51.638872    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:58:52.801935 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:52.801898    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 13:58:56.781564 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:56.781477    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-70f06e/autores-model-70f06e-kserve-6cc99667d7-xktzd" podUID="1d8fb485-97ad-4126-9067-4f19179b51f5" containerName="main" containerID="cri-o://9fb8707c99c755bb259ec0c6a7f9be1d9337e92f0c86fe86489ecef43f1a5e58" gracePeriod=45
Sep 18 13:58:57.033564 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:57.033504    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-70f06e/autores-model-70f06e-kserve-6cc99667d7-xktzd"
Sep 18 13:58:57.058839 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:57.058799    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d8fb485-97ad-4126-9067-4f19179b51f5-model-cache\") pod \"1d8fb485-97ad-4126-9067-4f19179b51f5\" (UID: \"1d8fb485-97ad-4126-9067-4f19179b51f5\") "
Sep 18 13:58:57.058995 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:57.058972    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d8fb485-97ad-4126-9067-4f19179b51f5-tls-certs\") pod \"1d8fb485-97ad-4126-9067-4f19179b51f5\" (UID: \"1d8fb485-97ad-4126-9067-4f19179b51f5\") "
Sep 18 13:58:57.059057 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:57.059010    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-spdzt\" (UniqueName: \"kubernetes.io/projected/1d8fb485-97ad-4126-9067-4f19179b51f5-kube-api-access-spdzt\") pod \"1d8fb485-97ad-4126-9067-4f19179b51f5\" (UID: \"1d8fb485-97ad-4126-9067-4f19179b51f5\") "
Sep 18 13:58:57.059057 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:57.059042    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d8fb485-97ad-4126-9067-4f19179b51f5-dshm\") pod \"1d8fb485-97ad-4126-9067-4f19179b51f5\" (UID: \"1d8fb485-97ad-4126-9067-4f19179b51f5\") "
Sep 18 13:58:57.059147 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:57.059058    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d8fb485-97ad-4126-9067-4f19179b51f5-model-cache" (OuterVolumeSpecName: "model-cache") pod "1d8fb485-97ad-4126-9067-4f19179b51f5" (UID: "1d8fb485-97ad-4126-9067-4f19179b51f5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:58:57.059147 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:57.059091    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d8fb485-97ad-4126-9067-4f19179b51f5-tmp-dir\") pod \"1d8fb485-97ad-4126-9067-4f19179b51f5\" (UID: \"1d8fb485-97ad-4126-9067-4f19179b51f5\") "
Sep 18 13:58:57.059147 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:57.059136    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d8fb485-97ad-4126-9067-4f19179b51f5-home\") pod \"1d8fb485-97ad-4126-9067-4f19179b51f5\" (UID: \"1d8fb485-97ad-4126-9067-4f19179b51f5\") "
Sep 18 13:58:57.059365 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:57.059350    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d8fb485-97ad-4126-9067-4f19179b51f5-model-cache\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:58:57.059450 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:57.059388    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d8fb485-97ad-4126-9067-4f19179b51f5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1d8fb485-97ad-4126-9067-4f19179b51f5" (UID: "1d8fb485-97ad-4126-9067-4f19179b51f5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:58:57.059528 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:57.059458    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d8fb485-97ad-4126-9067-4f19179b51f5-home" (OuterVolumeSpecName: "home") pod "1d8fb485-97ad-4126-9067-4f19179b51f5" (UID: "1d8fb485-97ad-4126-9067-4f19179b51f5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:58:57.061584 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:57.061557    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d8fb485-97ad-4126-9067-4f19179b51f5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1d8fb485-97ad-4126-9067-4f19179b51f5" (UID: "1d8fb485-97ad-4126-9067-4f19179b51f5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:58:57.061744 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:57.061723    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d8fb485-97ad-4126-9067-4f19179b51f5-dshm" (OuterVolumeSpecName: "dshm") pod "1d8fb485-97ad-4126-9067-4f19179b51f5" (UID: "1d8fb485-97ad-4126-9067-4f19179b51f5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:58:57.061812 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:57.061767    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d8fb485-97ad-4126-9067-4f19179b51f5-kube-api-access-spdzt" (OuterVolumeSpecName: "kube-api-access-spdzt") pod "1d8fb485-97ad-4126-9067-4f19179b51f5" (UID: "1d8fb485-97ad-4126-9067-4f19179b51f5"). InnerVolumeSpecName "kube-api-access-spdzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:58:57.159773 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:57.159745    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d8fb485-97ad-4126-9067-4f19179b51f5-tls-certs\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:58:57.159773 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:57.159772    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spdzt\" (UniqueName: \"kubernetes.io/projected/1d8fb485-97ad-4126-9067-4f19179b51f5-kube-api-access-spdzt\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:58:57.159955 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:57.159782    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d8fb485-97ad-4126-9067-4f19179b51f5-dshm\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:58:57.159955 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:57.159792    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d8fb485-97ad-4126-9067-4f19179b51f5-tmp-dir\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:58:57.159955 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:57.159801    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d8fb485-97ad-4126-9067-4f19179b51f5-home\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:58:57.583069 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:57.583036    2577 generic.go:356] "Generic (PLEG): container finished" podID="1d8fb485-97ad-4126-9067-4f19179b51f5" containerID="9fb8707c99c755bb259ec0c6a7f9be1d9337e92f0c86fe86489ecef43f1a5e58" exitCode=0
Sep 18 13:58:57.583241 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:57.583087    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-70f06e/autores-model-70f06e-kserve-6cc99667d7-xktzd"
Sep 18 13:58:57.583241 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:57.583111    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-70f06e/autores-model-70f06e-kserve-6cc99667d7-xktzd" event={"ID":"1d8fb485-97ad-4126-9067-4f19179b51f5","Type":"ContainerDied","Data":"9fb8707c99c755bb259ec0c6a7f9be1d9337e92f0c86fe86489ecef43f1a5e58"}
Sep 18 13:58:57.583241 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:57.583156    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-70f06e/autores-model-70f06e-kserve-6cc99667d7-xktzd" event={"ID":"1d8fb485-97ad-4126-9067-4f19179b51f5","Type":"ContainerDied","Data":"623b36947181e239f795d605786f12c8eacd662e775147b014746032eec4152d"}
Sep 18 13:58:57.583241 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:57.583174    2577 scope.go:117] "RemoveContainer" containerID="9fb8707c99c755bb259ec0c6a7f9be1d9337e92f0c86fe86489ecef43f1a5e58"
Sep 18 13:58:57.592282 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:57.592266    2577 scope.go:117] "RemoveContainer" containerID="9fb8707c99c755bb259ec0c6a7f9be1d9337e92f0c86fe86489ecef43f1a5e58"
Sep 18 13:58:57.592544 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:58:57.592527    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fb8707c99c755bb259ec0c6a7f9be1d9337e92f0c86fe86489ecef43f1a5e58\": container with ID starting with 9fb8707c99c755bb259ec0c6a7f9be1d9337e92f0c86fe86489ecef43f1a5e58 not found: ID does not exist" containerID="9fb8707c99c755bb259ec0c6a7f9be1d9337e92f0c86fe86489ecef43f1a5e58"
Sep 18 13:58:57.592582 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:57.592553    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fb8707c99c755bb259ec0c6a7f9be1d9337e92f0c86fe86489ecef43f1a5e58"} err="failed to get container status \"9fb8707c99c755bb259ec0c6a7f9be1d9337e92f0c86fe86489ecef43f1a5e58\": rpc error: code = NotFound desc = could not find container \"9fb8707c99c755bb259ec0c6a7f9be1d9337e92f0c86fe86489ecef43f1a5e58\": container with ID starting with 9fb8707c99c755bb259ec0c6a7f9be1d9337e92f0c86fe86489ecef43f1a5e58 not found: ID does not exist"
Sep 18 13:58:57.600272 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:57.600243    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-70f06e/autores-model-70f06e-kserve-6cc99667d7-xktzd"]
Sep 18 13:58:57.603193 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:57.603168    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-70f06e/autores-model-70f06e-kserve-6cc99667d7-xktzd"]
Sep 18 13:58:59.107914 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:58:59.107872    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d8fb485-97ad-4126-9067-4f19179b51f5" path="/var/lib/kubelet/pods/1d8fb485-97ad-4126-9067-4f19179b51f5/volumes"
Sep 18 13:59:04.474364 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:04.474329    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:05.217323 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:05.217294    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:13.125151 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:13.125123    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-8tnxl_69d3178a-5769-4b5f-a243-30a1bedc5bd6/console-operator/1.log"
Sep 18 13:59:13.126653 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:13.126633    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-8tnxl_69d3178a-5769-4b5f-a243-30a1bedc5bd6/console-operator/1.log"
Sep 18 13:59:17.559271 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:17.559234    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:19.104020 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:19.103986    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:19.315656 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:19.315619    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c"]
Sep 18 13:59:19.315928 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:19.315901    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c" podUID="62972a5e-7296-4486-928b-29f00380ef52" containerName="istio-proxy" containerID="cri-o://c9a4a4db39fc559cc5b310136d64745f968bc98adf9bc459681072fd610774a9" gracePeriod=30
Sep 18 13:59:19.346979 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:19.346946    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:19.801024 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:19.800989    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:20.076631 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:20.076552    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd"]
Sep 18 13:59:20.076952 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:20.076938    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1730d85c-df98-4db2-99f9-951845ebbc56" containerName="payload-pre-processing"
Sep 18 13:59:20.077003 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:20.076954    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1730d85c-df98-4db2-99f9-951845ebbc56" containerName="payload-pre-processing"
Sep 18 13:59:20.077003 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:20.076969    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b07a50e5-19e8-4821-bfa1-225d7160c9cc" containerName="payload-processing"
Sep 18 13:59:20.077003 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:20.076974    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b07a50e5-19e8-4821-bfa1-225d7160c9cc" containerName="payload-processing"
Sep 18 13:59:20.077003 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:20.076985    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69528ad8-e882-4047-97dd-9ee30ae286a9" containerName="revoke-keys"
Sep 18 13:59:20.077003 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:20.076991    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="69528ad8-e882-4047-97dd-9ee30ae286a9" containerName="revoke-keys"
Sep 18 13:59:20.077003 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:20.076999    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d8fb485-97ad-4126-9067-4f19179b51f5" containerName="main"
Sep 18 13:59:20.077003 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:20.077005    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d8fb485-97ad-4126-9067-4f19179b51f5" containerName="main"
Sep 18 13:59:20.077220 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:20.077067    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1730d85c-df98-4db2-99f9-951845ebbc56" containerName="payload-pre-processing"
Sep 18 13:59:20.077220 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:20.077074    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b07a50e5-19e8-4821-bfa1-225d7160c9cc" containerName="payload-processing"
Sep 18 13:59:20.077220 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:20.077080    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d8fb485-97ad-4126-9067-4f19179b51f5" containerName="main"
Sep 18 13:59:20.077220 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:20.077086    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="69528ad8-e882-4047-97dd-9ee30ae286a9" containerName="revoke-keys"
Sep 18 13:59:20.084522 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:20.084496    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd"
Sep 18 13:59:20.088443 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:20.088415    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-0aac26-openshift-default-dockercfg-9c5jn\""
Sep 18 13:59:20.097601 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:20.097577    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd"]
Sep 18 13:59:20.098070 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:20.098047    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:20.162833 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:20.162791    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vbww\" (UniqueName: \"kubernetes.io/projected/f8d985f0-b51e-4a56-9840-179887b8f54d-kube-api-access-7vbww\") pod \"e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd\" (UID: \"f8d985f0-b51e-4a56-9840-179887b8f54d\") " pod="openshift-ingress/e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd"
Sep 18 13:59:20.163268 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:20.162868    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8d985f0-b51e-4a56-9840-179887b8f54d-istiod-ca-cert\") pod \"e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd\" (UID: \"f8d985f0-b51e-4a56-9840-179887b8f54d\") " pod="openshift-ingress/e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd"
Sep 18 13:59:20.163268 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:20.162939    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8d985f0-b51e-4a56-9840-179887b8f54d-workload-certs\") pod \"e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd\" (UID: \"f8d985f0-b51e-4a56-9840-179887b8f54d\") " pod="openshift-ingress/e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd"
Sep 18 13:59:20.163268 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:20.162989    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8d985f0-b51e-4a56-9840-179887b8f54d-workload-socket\") pod \"e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd\" (UID: \"f8d985f0-b51e-4a56-9840-179887b8f54d\") " pod="openshift-ingress/e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd"
Sep 18 13:59:20.163268 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:20.163020    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8d985f0-b51e-4a56-9840-179887b8f54d-istio-envoy\") pod \"e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd\" (UID: \"f8d985f0-b51e-4a56-9840-179887b8f54d\") " pod="openshift-ingress/e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd"
Sep 18 13:59:20.163268 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:20.163043    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8d985f0-b51e-4a56-9840-179887b8f54d-istio-podinfo\") pod \"e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd\" (UID: \"f8d985f0-b51e-4a56-9840-179887b8f54d\") " pod="openshift-ingress/e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd"
Sep 18 13:59:20.163268 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:20.163069    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8d985f0-b51e-4a56-9840-179887b8f54d-credential-socket\") pod \"e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd\" (UID: \"f8d985f0-b51e-4a56-9840-179887b8f54d\") " pod="openshift-ingress/e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd"
Sep 18 13:59:20.163268 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:20.163165    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8d985f0-b51e-4a56-9840-179887b8f54d-istio-data\") pod \"e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd\" (UID: \"f8d985f0-b51e-4a56-9840-179887b8f54d\") " pod="openshift-ingress/e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd"
Sep 18 13:59:20.163268 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:20.163185    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8d985f0-b51e-4a56-9840-179887b8f54d-istio-token\") pod \"e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd\" (UID: \"f8d985f0-b51e-4a56-9840-179887b8f54d\") " pod="openshift-ingress/e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd"
Sep 18 13:59:20.215103 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:20.215062    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:20.264267 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:20.264229    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8d985f0-b51e-4a56-9840-179887b8f54d-workload-socket\") pod \"e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd\" (UID: \"f8d985f0-b51e-4a56-9840-179887b8f54d\") " pod="openshift-ingress/e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd"
Sep 18 13:59:20.264267 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:20.264268    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8d985f0-b51e-4a56-9840-179887b8f54d-istio-envoy\") pod \"e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd\" (UID: \"f8d985f0-b51e-4a56-9840-179887b8f54d\") " pod="openshift-ingress/e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd"
Sep 18 13:59:20.264525 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:20.264283    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8d985f0-b51e-4a56-9840-179887b8f54d-istio-podinfo\") pod \"e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd\" (UID: \"f8d985f0-b51e-4a56-9840-179887b8f54d\") " pod="openshift-ingress/e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd"
Sep 18 13:59:20.264525 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:20.264305    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8d985f0-b51e-4a56-9840-179887b8f54d-credential-socket\") pod \"e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd\" (UID: \"f8d985f0-b51e-4a56-9840-179887b8f54d\") " pod="openshift-ingress/e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd"
Sep 18 13:59:20.264525 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:20.264358    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8d985f0-b51e-4a56-9840-179887b8f54d-istio-data\") pod \"e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd\" (UID: \"f8d985f0-b51e-4a56-9840-179887b8f54d\") " pod="openshift-ingress/e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd"
Sep 18 13:59:20.264525 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:20.264375    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8d985f0-b51e-4a56-9840-179887b8f54d-istio-token\") pod \"e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd\" (UID: \"f8d985f0-b51e-4a56-9840-179887b8f54d\") " pod="openshift-ingress/e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd"
Sep 18 13:59:20.264525 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:20.264396    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vbww\" (UniqueName: \"kubernetes.io/projected/f8d985f0-b51e-4a56-9840-179887b8f54d-kube-api-access-7vbww\") pod \"e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd\" (UID: \"f8d985f0-b51e-4a56-9840-179887b8f54d\") " pod="openshift-ingress/e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd"
Sep 18 13:59:20.264525 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:20.264436    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8d985f0-b51e-4a56-9840-179887b8f54d-istiod-ca-cert\") pod \"e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd\" (UID: \"f8d985f0-b51e-4a56-9840-179887b8f54d\") " pod="openshift-ingress/e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd"
Sep 18 13:59:20.264525 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:20.264479    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8d985f0-b51e-4a56-9840-179887b8f54d-workload-certs\") pod \"e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd\" (UID: \"f8d985f0-b51e-4a56-9840-179887b8f54d\") " pod="openshift-ingress/e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd"
Sep 18 13:59:20.264927 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:20.264858    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8d985f0-b51e-4a56-9840-179887b8f54d-workload-certs\") pod \"e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd\" (UID: \"f8d985f0-b51e-4a56-9840-179887b8f54d\") " pod="openshift-ingress/e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd"
Sep 18 13:59:20.264994 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:20.264927    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8d985f0-b51e-4a56-9840-179887b8f54d-credential-socket\") pod \"e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd\" (UID: \"f8d985f0-b51e-4a56-9840-179887b8f54d\") " pod="openshift-ingress/e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd"
Sep 18 13:59:20.265057 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:20.265013    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8d985f0-b51e-4a56-9840-179887b8f54d-istio-data\") pod \"e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd\" (UID: \"f8d985f0-b51e-4a56-9840-179887b8f54d\") " pod="openshift-ingress/e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd"
Sep 18 13:59:20.265057 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:20.265019    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8d985f0-b51e-4a56-9840-179887b8f54d-workload-socket\") pod \"e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd\" (UID: \"f8d985f0-b51e-4a56-9840-179887b8f54d\") " pod="openshift-ingress/e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd"
Sep 18 13:59:20.265403 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:20.265376    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8d985f0-b51e-4a56-9840-179887b8f54d-istiod-ca-cert\") pod \"e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd\" (UID: \"f8d985f0-b51e-4a56-9840-179887b8f54d\") " pod="openshift-ingress/e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd"
Sep 18 13:59:20.266583 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:20.266559    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8d985f0-b51e-4a56-9840-179887b8f54d-istio-envoy\") pod \"e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd\" (UID: \"f8d985f0-b51e-4a56-9840-179887b8f54d\") " pod="openshift-ingress/e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd"
Sep 18 13:59:20.267020 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:20.266999    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8d985f0-b51e-4a56-9840-179887b8f54d-istio-podinfo\") pod \"e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd\" (UID: \"f8d985f0-b51e-4a56-9840-179887b8f54d\") " pod="openshift-ingress/e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd"
Sep 18 13:59:20.272162 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:20.272131    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8d985f0-b51e-4a56-9840-179887b8f54d-istio-token\") pod \"e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd\" (UID: \"f8d985f0-b51e-4a56-9840-179887b8f54d\") " pod="openshift-ingress/e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd"
Sep 18 13:59:20.272303 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:20.272288    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vbww\" (UniqueName: \"kubernetes.io/projected/f8d985f0-b51e-4a56-9840-179887b8f54d-kube-api-access-7vbww\") pod \"e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd\" (UID: \"f8d985f0-b51e-4a56-9840-179887b8f54d\") " pod="openshift-ingress/e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd"
Sep 18 13:59:20.398940 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:20.398902    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd"
Sep 18 13:59:20.680484 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:20.680458    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd"]
Sep 18 13:59:20.682572 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:59:20.682545    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8d985f0_b51e_4a56_9840_179887b8f54d.slice/crio-cc0f786fe9e0bc3a92e57ee85329d51612ed6e355d0ed2d776f7fe5664d3069a WatchSource:0}: Error finding container cc0f786fe9e0bc3a92e57ee85329d51612ed6e355d0ed2d776f7fe5664d3069a: Status 404 returned error can't find the container with id cc0f786fe9e0bc3a92e57ee85329d51612ed6e355d0ed2d776f7fe5664d3069a
Sep 18 13:59:20.684859 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:20.684803    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 13:59:20.684935 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:20.684914    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 13:59:20.684986 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:20.684969    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 13:59:20.711755 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:20.711729    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd" event={"ID":"f8d985f0-b51e-4a56-9840-179887b8f54d","Type":"ContainerStarted","Data":"cc0f786fe9e0bc3a92e57ee85329d51612ed6e355d0ed2d776f7fe5664d3069a"}
Sep 18 13:59:21.718501 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:21.718468    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd" event={"ID":"f8d985f0-b51e-4a56-9840-179887b8f54d","Type":"ContainerStarted","Data":"af42580ad4adcd4beca73bc5031659a1465d2d9ce9c2fbbb02d964d4f5b800e2"}
Sep 18 13:59:21.739678 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:21.739626    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd" podStartSLOduration=1.73961055 podStartE2EDuration="1.73961055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:59:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:59:21.737309951 +0000 UTC m=+910.789969278" watchObservedRunningTime="2026-09-18 13:59:21.73961055 +0000 UTC m=+910.792269913"
Sep 18 13:59:22.400018 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:22.399979    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd"
Sep 18 13:59:22.405173 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:22.405149    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd"
Sep 18 13:59:22.724603 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:22.724521    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd"
Sep 18 13:59:22.725380 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:22.725360    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd"
Sep 18 13:59:22.921063 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:22.921035    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:23.270317 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:23.270275    2577 patch_prober.go:28] interesting pod/e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 18 13:59:23.270484 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:23.270339    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c" podUID="62972a5e-7296-4486-928b-29f00380ef52" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 13:59:23.606460 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:23.606422    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-377ca2/premium-w4-377ca2-kserve-759c89bc45-4r9k4"]
Sep 18 13:59:23.610439 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:23.610416    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-377ca2/premium-w4-377ca2-kserve-759c89bc45-4r9k4"
Sep 18 13:59:23.613414 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:23.613394    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-377ca2\"/\"premium-w4-377ca2-kserve-self-signed-certs\""
Sep 18 13:59:23.628652 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:23.628623    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-377ca2/premium-w4-377ca2-kserve-759c89bc45-4r9k4"]
Sep 18 13:59:23.630310 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:23.630274    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:23.696994 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:23.696953    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krlnk\" (UniqueName: \"kubernetes.io/projected/d81fc1dd-14e8-4f3c-9cfd-56731b661718-kube-api-access-krlnk\") pod \"premium-w4-377ca2-kserve-759c89bc45-4r9k4\" (UID: \"d81fc1dd-14e8-4f3c-9cfd-56731b661718\") " pod="e2e-models-e2e-worker-w4-377ca2/premium-w4-377ca2-kserve-759c89bc45-4r9k4"
Sep 18 13:59:23.697200 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:23.697000    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d81fc1dd-14e8-4f3c-9cfd-56731b661718-tls-certs\") pod \"premium-w4-377ca2-kserve-759c89bc45-4r9k4\" (UID: \"d81fc1dd-14e8-4f3c-9cfd-56731b661718\") " pod="e2e-models-e2e-worker-w4-377ca2/premium-w4-377ca2-kserve-759c89bc45-4r9k4"
Sep 18 13:59:23.697200 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:23.697022    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d81fc1dd-14e8-4f3c-9cfd-56731b661718-home\") pod \"premium-w4-377ca2-kserve-759c89bc45-4r9k4\" (UID: \"d81fc1dd-14e8-4f3c-9cfd-56731b661718\") " pod="e2e-models-e2e-worker-w4-377ca2/premium-w4-377ca2-kserve-759c89bc45-4r9k4"
Sep 18 13:59:23.697200 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:23.697038    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d81fc1dd-14e8-4f3c-9cfd-56731b661718-dshm\") pod \"premium-w4-377ca2-kserve-759c89bc45-4r9k4\" (UID: \"d81fc1dd-14e8-4f3c-9cfd-56731b661718\") " pod="e2e-models-e2e-worker-w4-377ca2/premium-w4-377ca2-kserve-759c89bc45-4r9k4"
Sep 18 13:59:23.697200 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:23.697125    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d81fc1dd-14e8-4f3c-9cfd-56731b661718-tmp-dir\") pod \"premium-w4-377ca2-kserve-759c89bc45-4r9k4\" (UID: \"d81fc1dd-14e8-4f3c-9cfd-56731b661718\") " pod="e2e-models-e2e-worker-w4-377ca2/premium-w4-377ca2-kserve-759c89bc45-4r9k4"
Sep 18 13:59:23.697200 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:23.697165    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d81fc1dd-14e8-4f3c-9cfd-56731b661718-model-cache\") pod \"premium-w4-377ca2-kserve-759c89bc45-4r9k4\" (UID: \"d81fc1dd-14e8-4f3c-9cfd-56731b661718\") " pod="e2e-models-e2e-worker-w4-377ca2/premium-w4-377ca2-kserve-759c89bc45-4r9k4"
Sep 18 13:59:23.721296 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:23.721267    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:23.797986 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:23.797932    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d81fc1dd-14e8-4f3c-9cfd-56731b661718-tls-certs\") pod \"premium-w4-377ca2-kserve-759c89bc45-4r9k4\" (UID: \"d81fc1dd-14e8-4f3c-9cfd-56731b661718\") " pod="e2e-models-e2e-worker-w4-377ca2/premium-w4-377ca2-kserve-759c89bc45-4r9k4"
Sep 18 13:59:23.798500 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:23.798017    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d81fc1dd-14e8-4f3c-9cfd-56731b661718-home\") pod \"premium-w4-377ca2-kserve-759c89bc45-4r9k4\" (UID: \"d81fc1dd-14e8-4f3c-9cfd-56731b661718\") " pod="e2e-models-e2e-worker-w4-377ca2/premium-w4-377ca2-kserve-759c89bc45-4r9k4"
Sep 18 13:59:23.798500 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:23.798054    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d81fc1dd-14e8-4f3c-9cfd-56731b661718-dshm\") pod \"premium-w4-377ca2-kserve-759c89bc45-4r9k4\" (UID: \"d81fc1dd-14e8-4f3c-9cfd-56731b661718\") " pod="e2e-models-e2e-worker-w4-377ca2/premium-w4-377ca2-kserve-759c89bc45-4r9k4"
Sep 18 13:59:23.798500 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:23.798103    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d81fc1dd-14e8-4f3c-9cfd-56731b661718-tmp-dir\") pod \"premium-w4-377ca2-kserve-759c89bc45-4r9k4\" (UID: \"d81fc1dd-14e8-4f3c-9cfd-56731b661718\") " pod="e2e-models-e2e-worker-w4-377ca2/premium-w4-377ca2-kserve-759c89bc45-4r9k4"
Sep 18 13:59:23.798500 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:23.798146    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d81fc1dd-14e8-4f3c-9cfd-56731b661718-model-cache\") pod \"premium-w4-377ca2-kserve-759c89bc45-4r9k4\" (UID: \"d81fc1dd-14e8-4f3c-9cfd-56731b661718\") " pod="e2e-models-e2e-worker-w4-377ca2/premium-w4-377ca2-kserve-759c89bc45-4r9k4"
Sep 18 13:59:23.798500 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:23.798334    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krlnk\" (UniqueName: \"kubernetes.io/projected/d81fc1dd-14e8-4f3c-9cfd-56731b661718-kube-api-access-krlnk\") pod \"premium-w4-377ca2-kserve-759c89bc45-4r9k4\" (UID: \"d81fc1dd-14e8-4f3c-9cfd-56731b661718\") " pod="e2e-models-e2e-worker-w4-377ca2/premium-w4-377ca2-kserve-759c89bc45-4r9k4"
Sep 18 13:59:23.798797 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:23.798515    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d81fc1dd-14e8-4f3c-9cfd-56731b661718-home\") pod \"premium-w4-377ca2-kserve-759c89bc45-4r9k4\" (UID: \"d81fc1dd-14e8-4f3c-9cfd-56731b661718\") " pod="e2e-models-e2e-worker-w4-377ca2/premium-w4-377ca2-kserve-759c89bc45-4r9k4"
Sep 18 13:59:23.798797 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:23.798565    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d81fc1dd-14e8-4f3c-9cfd-56731b661718-tmp-dir\") pod \"premium-w4-377ca2-kserve-759c89bc45-4r9k4\" (UID: \"d81fc1dd-14e8-4f3c-9cfd-56731b661718\") " pod="e2e-models-e2e-worker-w4-377ca2/premium-w4-377ca2-kserve-759c89bc45-4r9k4"
Sep 18 13:59:23.798941 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:23.798871    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d81fc1dd-14e8-4f3c-9cfd-56731b661718-model-cache\") pod \"premium-w4-377ca2-kserve-759c89bc45-4r9k4\" (UID: \"d81fc1dd-14e8-4f3c-9cfd-56731b661718\") " pod="e2e-models-e2e-worker-w4-377ca2/premium-w4-377ca2-kserve-759c89bc45-4r9k4"
Sep 18 13:59:23.800656 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:23.800634    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d81fc1dd-14e8-4f3c-9cfd-56731b661718-dshm\") pod \"premium-w4-377ca2-kserve-759c89bc45-4r9k4\" (UID: \"d81fc1dd-14e8-4f3c-9cfd-56731b661718\") " pod="e2e-models-e2e-worker-w4-377ca2/premium-w4-377ca2-kserve-759c89bc45-4r9k4"
Sep 18 13:59:23.800999 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:23.800980    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d81fc1dd-14e8-4f3c-9cfd-56731b661718-tls-certs\") pod \"premium-w4-377ca2-kserve-759c89bc45-4r9k4\" (UID: \"d81fc1dd-14e8-4f3c-9cfd-56731b661718\") " pod="e2e-models-e2e-worker-w4-377ca2/premium-w4-377ca2-kserve-759c89bc45-4r9k4"
Sep 18 13:59:23.806346 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:23.806320    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krlnk\" (UniqueName: \"kubernetes.io/projected/d81fc1dd-14e8-4f3c-9cfd-56731b661718-kube-api-access-krlnk\") pod \"premium-w4-377ca2-kserve-759c89bc45-4r9k4\" (UID: \"d81fc1dd-14e8-4f3c-9cfd-56731b661718\") " pod="e2e-models-e2e-worker-w4-377ca2/premium-w4-377ca2-kserve-759c89bc45-4r9k4"
Sep 18 13:59:23.919814 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:23.919727    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-377ca2/premium-w4-377ca2-kserve-759c89bc45-4r9k4"
Sep 18 13:59:24.202856 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:24.202816    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-377ca2/premium-w4-377ca2-kserve-759c89bc45-4r9k4"]
Sep 18 13:59:24.204811 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:59:24.204783    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd81fc1dd_14e8_4f3c_9cfd_56731b661718.slice/crio-582908eae9e74b73f8983caa5f5bcc1571877706115dc729e106d09e0ff06e1f WatchSource:0}: Error finding container 582908eae9e74b73f8983caa5f5bcc1571877706115dc729e106d09e0ff06e1f: Status 404 returned error can't find the container with id 582908eae9e74b73f8983caa5f5bcc1571877706115dc729e106d09e0ff06e1f
Sep 18 13:59:24.466585 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:24.466497    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:24.562840 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:24.562806    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c"
Sep 18 13:59:24.576054 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:24.576023    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:24.684170 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:24.684143    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:24.706355 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:24.706327    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvgpt\" (UniqueName: \"kubernetes.io/projected/62972a5e-7296-4486-928b-29f00380ef52-kube-api-access-jvgpt\") pod \"62972a5e-7296-4486-928b-29f00380ef52\" (UID: \"62972a5e-7296-4486-928b-29f00380ef52\") "
Sep 18 13:59:24.706462 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:24.706380    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62972a5e-7296-4486-928b-29f00380ef52-istiod-ca-cert\") pod \"62972a5e-7296-4486-928b-29f00380ef52\" (UID: \"62972a5e-7296-4486-928b-29f00380ef52\") "
Sep 18 13:59:24.706462 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:24.706414    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62972a5e-7296-4486-928b-29f00380ef52-istio-data\") pod \"62972a5e-7296-4486-928b-29f00380ef52\" (UID: \"62972a5e-7296-4486-928b-29f00380ef52\") "
Sep 18 13:59:24.706462 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:24.706432    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62972a5e-7296-4486-928b-29f00380ef52-istio-token\") pod \"62972a5e-7296-4486-928b-29f00380ef52\" (UID: \"62972a5e-7296-4486-928b-29f00380ef52\") "
Sep 18 13:59:24.706617 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:24.706525    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62972a5e-7296-4486-928b-29f00380ef52-workload-socket\") pod \"62972a5e-7296-4486-928b-29f00380ef52\" (UID: \"62972a5e-7296-4486-928b-29f00380ef52\") "
Sep 18 13:59:24.706617 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:24.706546    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62972a5e-7296-4486-928b-29f00380ef52-istio-envoy\") pod \"62972a5e-7296-4486-928b-29f00380ef52\" (UID: \"62972a5e-7296-4486-928b-29f00380ef52\") "
Sep 18 13:59:24.706720 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:24.706673    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62972a5e-7296-4486-928b-29f00380ef52-workload-certs\") pod \"62972a5e-7296-4486-928b-29f00380ef52\" (UID: \"62972a5e-7296-4486-928b-29f00380ef52\") "
Sep 18 13:59:24.706775 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:24.706721    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62972a5e-7296-4486-928b-29f00380ef52-credential-socket\") pod \"62972a5e-7296-4486-928b-29f00380ef52\" (UID: \"62972a5e-7296-4486-928b-29f00380ef52\") "
Sep 18 13:59:24.706849 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:24.706770    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62972a5e-7296-4486-928b-29f00380ef52-istio-data" (OuterVolumeSpecName: "istio-data") pod "62972a5e-7296-4486-928b-29f00380ef52" (UID: "62972a5e-7296-4486-928b-29f00380ef52"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:59:24.706849 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:24.706794    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62972a5e-7296-4486-928b-29f00380ef52-istio-podinfo\") pod \"62972a5e-7296-4486-928b-29f00380ef52\" (UID: \"62972a5e-7296-4486-928b-29f00380ef52\") "
Sep 18 13:59:24.706990 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:24.706968    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62972a5e-7296-4486-928b-29f00380ef52-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "62972a5e-7296-4486-928b-29f00380ef52" (UID: "62972a5e-7296-4486-928b-29f00380ef52"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:59:24.707071 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:24.707001    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62972a5e-7296-4486-928b-29f00380ef52-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "62972a5e-7296-4486-928b-29f00380ef52" (UID: "62972a5e-7296-4486-928b-29f00380ef52"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:59:24.707071 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:24.706920    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62972a5e-7296-4486-928b-29f00380ef52-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "62972a5e-7296-4486-928b-29f00380ef52" (UID: "62972a5e-7296-4486-928b-29f00380ef52"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:59:24.707179 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:24.707072    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62972a5e-7296-4486-928b-29f00380ef52-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "62972a5e-7296-4486-928b-29f00380ef52" (UID: "62972a5e-7296-4486-928b-29f00380ef52"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:59:24.707179 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:24.707157    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62972a5e-7296-4486-928b-29f00380ef52-workload-socket\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:59:24.707179 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:24.707174    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62972a5e-7296-4486-928b-29f00380ef52-workload-certs\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:59:24.707289 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:24.707188    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62972a5e-7296-4486-928b-29f00380ef52-credential-socket\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:59:24.707289 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:24.707201    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62972a5e-7296-4486-928b-29f00380ef52-istiod-ca-cert\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:59:24.707289 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:24.707213    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62972a5e-7296-4486-928b-29f00380ef52-istio-data\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:59:24.708942 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:24.708915    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62972a5e-7296-4486-928b-29f00380ef52-istio-token" (OuterVolumeSpecName: "istio-token") pod "62972a5e-7296-4486-928b-29f00380ef52" (UID: "62972a5e-7296-4486-928b-29f00380ef52"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:59:24.708942 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:24.708918    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62972a5e-7296-4486-928b-29f00380ef52-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "62972a5e-7296-4486-928b-29f00380ef52" (UID: "62972a5e-7296-4486-928b-29f00380ef52"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:59:24.709191 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:24.709164    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62972a5e-7296-4486-928b-29f00380ef52-kube-api-access-jvgpt" (OuterVolumeSpecName: "kube-api-access-jvgpt") pod "62972a5e-7296-4486-928b-29f00380ef52" (UID: "62972a5e-7296-4486-928b-29f00380ef52"). InnerVolumeSpecName "kube-api-access-jvgpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:59:24.709247 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:24.709231    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/62972a5e-7296-4486-928b-29f00380ef52-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "62972a5e-7296-4486-928b-29f00380ef52" (UID: "62972a5e-7296-4486-928b-29f00380ef52"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 13:59:24.737319 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:24.737225    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-377ca2/premium-w4-377ca2-kserve-759c89bc45-4r9k4" event={"ID":"d81fc1dd-14e8-4f3c-9cfd-56731b661718","Type":"ContainerStarted","Data":"4788c5e924a73b728097dd08d088131ceb7dcd156be8470d919aa3be3918ac78"}
Sep 18 13:59:24.737319 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:24.737271    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-377ca2/premium-w4-377ca2-kserve-759c89bc45-4r9k4" event={"ID":"d81fc1dd-14e8-4f3c-9cfd-56731b661718","Type":"ContainerStarted","Data":"582908eae9e74b73f8983caa5f5bcc1571877706115dc729e106d09e0ff06e1f"}
Sep 18 13:59:24.739038 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:24.739004    2577 generic.go:356] "Generic (PLEG): container finished" podID="62972a5e-7296-4486-928b-29f00380ef52" containerID="c9a4a4db39fc559cc5b310136d64745f968bc98adf9bc459681072fd610774a9" exitCode=0
Sep 18 13:59:24.739171 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:24.739048    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c"
Sep 18 13:59:24.739171 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:24.739054    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c" event={"ID":"62972a5e-7296-4486-928b-29f00380ef52","Type":"ContainerDied","Data":"c9a4a4db39fc559cc5b310136d64745f968bc98adf9bc459681072fd610774a9"}
Sep 18 13:59:24.739171 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:24.739109    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c" event={"ID":"62972a5e-7296-4486-928b-29f00380ef52","Type":"ContainerDied","Data":"3c15f8303ed621b3eeafaf03ab3057990525c78cee3bc3b548437d04444cac20"}
Sep 18 13:59:24.739171 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:24.739135    2577 scope.go:117] "RemoveContainer" containerID="c9a4a4db39fc559cc5b310136d64745f968bc98adf9bc459681072fd610774a9"
Sep 18 13:59:24.748597 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:24.748578    2577 scope.go:117] "RemoveContainer" containerID="8e684556d7d790083aca3b8cbc784dda56d26cdd45d309249f95b4d9b7fa8ba3"
Sep 18 13:59:24.757915 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:24.757891    2577 scope.go:117] "RemoveContainer" containerID="c9a4a4db39fc559cc5b310136d64745f968bc98adf9bc459681072fd610774a9"
Sep 18 13:59:24.758237 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:59:24.758216    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9a4a4db39fc559cc5b310136d64745f968bc98adf9bc459681072fd610774a9\": container with ID starting with c9a4a4db39fc559cc5b310136d64745f968bc98adf9bc459681072fd610774a9 not found: ID does not exist" containerID="c9a4a4db39fc559cc5b310136d64745f968bc98adf9bc459681072fd610774a9"
Sep 18 13:59:24.758305 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:24.758248    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9a4a4db39fc559cc5b310136d64745f968bc98adf9bc459681072fd610774a9"} err="failed to get container status \"c9a4a4db39fc559cc5b310136d64745f968bc98adf9bc459681072fd610774a9\": rpc error: code = NotFound desc = could not find container \"c9a4a4db39fc559cc5b310136d64745f968bc98adf9bc459681072fd610774a9\": container with ID starting with c9a4a4db39fc559cc5b310136d64745f968bc98adf9bc459681072fd610774a9 not found: ID does not exist"
Sep 18 13:59:24.758305 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:24.758265    2577 scope.go:117] "RemoveContainer" containerID="8e684556d7d790083aca3b8cbc784dda56d26cdd45d309249f95b4d9b7fa8ba3"
Sep 18 13:59:24.758531 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:59:24.758510    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e684556d7d790083aca3b8cbc784dda56d26cdd45d309249f95b4d9b7fa8ba3\": container with ID starting with 8e684556d7d790083aca3b8cbc784dda56d26cdd45d309249f95b4d9b7fa8ba3 not found: ID does not exist" containerID="8e684556d7d790083aca3b8cbc784dda56d26cdd45d309249f95b4d9b7fa8ba3"
Sep 18 13:59:24.758597 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:24.758539    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e684556d7d790083aca3b8cbc784dda56d26cdd45d309249f95b4d9b7fa8ba3"} err="failed to get container status \"8e684556d7d790083aca3b8cbc784dda56d26cdd45d309249f95b4d9b7fa8ba3\": rpc error: code = NotFound desc = could not find container \"8e684556d7d790083aca3b8cbc784dda56d26cdd45d309249f95b4d9b7fa8ba3\": container with ID starting with 8e684556d7d790083aca3b8cbc784dda56d26cdd45d309249f95b4d9b7fa8ba3 not found: ID does not exist"
Sep 18 13:59:24.777799 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:24.777744    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-377ca2/premium-w4-377ca2-kserve-759c89bc45-4r9k4" podStartSLOduration=1.777727896 podStartE2EDuration="1.777727896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:59:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:59:24.775235754 +0000 UTC m=+913.827895080" watchObservedRunningTime="2026-09-18 13:59:24.777727896 +0000 UTC m=+913.830387227"
Sep 18 13:59:24.808548 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:24.808508    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62972a5e-7296-4486-928b-29f00380ef52-istio-envoy\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:59:24.808548 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:24.808538    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62972a5e-7296-4486-928b-29f00380ef52-istio-podinfo\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:59:24.808548 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:24.808553    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvgpt\" (UniqueName: \"kubernetes.io/projected/62972a5e-7296-4486-928b-29f00380ef52-kube-api-access-jvgpt\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:59:24.809174 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:24.808570    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62972a5e-7296-4486-928b-29f00380ef52-istio-token\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 13:59:24.814072 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:24.814046    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c"]
Sep 18 13:59:24.834163 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:24.834132    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-70f06e-openshift-default-7947c6dcd-4jl8c"]
Sep 18 13:59:25.108147 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:25.108115    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62972a5e-7296-4486-928b-29f00380ef52" path="/var/lib/kubelet/pods/62972a5e-7296-4486-928b-29f00380ef52/volumes"
Sep 18 13:59:25.408224 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:25.408137    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:26.138702 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:26.138667    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:26.845956 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:26.845924    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-0aac26-66c8b4f78c-5p5ns"]
Sep 18 13:59:26.846316 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:26.846291    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62972a5e-7296-4486-928b-29f00380ef52" containerName="istio-proxy"
Sep 18 13:59:26.846316 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:26.846316    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="62972a5e-7296-4486-928b-29f00380ef52" containerName="istio-proxy"
Sep 18 13:59:26.846449 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:26.846326    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62972a5e-7296-4486-928b-29f00380ef52" containerName="istio-proxy"
Sep 18 13:59:26.846449 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:26.846332    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="62972a5e-7296-4486-928b-29f00380ef52" containerName="istio-proxy"
Sep 18 13:59:26.846449 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:26.846413    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="62972a5e-7296-4486-928b-29f00380ef52" containerName="istio-proxy"
Sep 18 13:59:26.851023 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:26.851001    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0aac26-66c8b4f78c-5p5ns"
Sep 18 13:59:26.855667 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:26.855645    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-0aac26-serving-cert\""
Sep 18 13:59:26.856025 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:26.856004    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vx9xd\""
Sep 18 13:59:26.856132 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:26.856035    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 13:59:26.856132 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:26.856005    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 18 13:59:26.856132 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:26.856005    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 18 13:59:26.863626 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:26.863597    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-0aac26-66c8b4f78c-5p5ns"]
Sep 18 13:59:26.890118 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:26.890087    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:26.929597 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:26.929551    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm458\" (UniqueName: \"kubernetes.io/projected/e96148d8-4aea-49f4-9664-1298b3d56af4-kube-api-access-sm458\") pod \"maas-api-e2e-explicit-0aac26-66c8b4f78c-5p5ns\" (UID: \"e96148d8-4aea-49f4-9664-1298b3d56af4\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0aac26-66c8b4f78c-5p5ns"
Sep 18 13:59:26.929785 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:26.929653    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e96148d8-4aea-49f4-9664-1298b3d56af4-metrics-certs\") pod \"maas-api-e2e-explicit-0aac26-66c8b4f78c-5p5ns\" (UID: \"e96148d8-4aea-49f4-9664-1298b3d56af4\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0aac26-66c8b4f78c-5p5ns"
Sep 18 13:59:26.929785 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:26.929683    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e96148d8-4aea-49f4-9664-1298b3d56af4-openshift-service-ca\") pod \"maas-api-e2e-explicit-0aac26-66c8b4f78c-5p5ns\" (UID: \"e96148d8-4aea-49f4-9664-1298b3d56af4\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0aac26-66c8b4f78c-5p5ns"
Sep 18 13:59:26.929785 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:26.929729    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e96148d8-4aea-49f4-9664-1298b3d56af4-maas-api-tls\") pod \"maas-api-e2e-explicit-0aac26-66c8b4f78c-5p5ns\" (UID: \"e96148d8-4aea-49f4-9664-1298b3d56af4\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0aac26-66c8b4f78c-5p5ns"
Sep 18 13:59:26.956704 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:26.956672    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:27.031098 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:27.031063    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm458\" (UniqueName: \"kubernetes.io/projected/e96148d8-4aea-49f4-9664-1298b3d56af4-kube-api-access-sm458\") pod \"maas-api-e2e-explicit-0aac26-66c8b4f78c-5p5ns\" (UID: \"e96148d8-4aea-49f4-9664-1298b3d56af4\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0aac26-66c8b4f78c-5p5ns"
Sep 18 13:59:27.031272 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:27.031148    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e96148d8-4aea-49f4-9664-1298b3d56af4-metrics-certs\") pod \"maas-api-e2e-explicit-0aac26-66c8b4f78c-5p5ns\" (UID: \"e96148d8-4aea-49f4-9664-1298b3d56af4\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0aac26-66c8b4f78c-5p5ns"
Sep 18 13:59:27.031272 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:27.031178    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e96148d8-4aea-49f4-9664-1298b3d56af4-openshift-service-ca\") pod \"maas-api-e2e-explicit-0aac26-66c8b4f78c-5p5ns\" (UID: \"e96148d8-4aea-49f4-9664-1298b3d56af4\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0aac26-66c8b4f78c-5p5ns"
Sep 18 13:59:27.031272 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:27.031224    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e96148d8-4aea-49f4-9664-1298b3d56af4-maas-api-tls\") pod \"maas-api-e2e-explicit-0aac26-66c8b4f78c-5p5ns\" (UID: \"e96148d8-4aea-49f4-9664-1298b3d56af4\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0aac26-66c8b4f78c-5p5ns"
Sep 18 13:59:27.031497 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:59:27.031371    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-0aac26-serving-cert: secret "maas-api-e2e-explicit-0aac26-serving-cert" not found
Sep 18 13:59:27.031553 ip-10-0-141-222 kubenswrapper[2577]: E0918 13:59:27.031497    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e96148d8-4aea-49f4-9664-1298b3d56af4-maas-api-tls podName:e96148d8-4aea-49f4-9664-1298b3d56af4 nodeName:}" failed. No retries permitted until 2026-09-18 13:59:27.531478296 +0000 UTC m=+916.584137618 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e96148d8-4aea-49f4-9664-1298b3d56af4-maas-api-tls") pod "maas-api-e2e-explicit-0aac26-66c8b4f78c-5p5ns" (UID: "e96148d8-4aea-49f4-9664-1298b3d56af4") : secret "maas-api-e2e-explicit-0aac26-serving-cert" not found
Sep 18 13:59:27.031895 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:27.031863    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e96148d8-4aea-49f4-9664-1298b3d56af4-openshift-service-ca\") pod \"maas-api-e2e-explicit-0aac26-66c8b4f78c-5p5ns\" (UID: \"e96148d8-4aea-49f4-9664-1298b3d56af4\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0aac26-66c8b4f78c-5p5ns"
Sep 18 13:59:27.033684 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:27.033659    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e96148d8-4aea-49f4-9664-1298b3d56af4-metrics-certs\") pod \"maas-api-e2e-explicit-0aac26-66c8b4f78c-5p5ns\" (UID: \"e96148d8-4aea-49f4-9664-1298b3d56af4\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0aac26-66c8b4f78c-5p5ns"
Sep 18 13:59:27.040838 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:27.040798    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm458\" (UniqueName: \"kubernetes.io/projected/e96148d8-4aea-49f4-9664-1298b3d56af4-kube-api-access-sm458\") pod \"maas-api-e2e-explicit-0aac26-66c8b4f78c-5p5ns\" (UID: \"e96148d8-4aea-49f4-9664-1298b3d56af4\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0aac26-66c8b4f78c-5p5ns"
Sep 18 13:59:27.236128 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:27.236039    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:27.470150 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:27.470113    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:27.536902 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:27.536785    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e96148d8-4aea-49f4-9664-1298b3d56af4-maas-api-tls\") pod \"maas-api-e2e-explicit-0aac26-66c8b4f78c-5p5ns\" (UID: \"e96148d8-4aea-49f4-9664-1298b3d56af4\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0aac26-66c8b4f78c-5p5ns"
Sep 18 13:59:27.539296 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:27.539273    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e96148d8-4aea-49f4-9664-1298b3d56af4-maas-api-tls\") pod \"maas-api-e2e-explicit-0aac26-66c8b4f78c-5p5ns\" (UID: \"e96148d8-4aea-49f4-9664-1298b3d56af4\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0aac26-66c8b4f78c-5p5ns"
Sep 18 13:59:27.762033 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:27.761997    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0aac26-66c8b4f78c-5p5ns"
Sep 18 13:59:28.038125 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:28.038094    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-0aac26-66c8b4f78c-5p5ns"]
Sep 18 13:59:28.039606 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:59:28.039573    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode96148d8_4aea_49f4_9664_1298b3d56af4.slice/crio-46964adc273ae519b1e21c060ccab66f1abb9768d585653212bda9df36ba9a1f WatchSource:0}: Error finding container 46964adc273ae519b1e21c060ccab66f1abb9768d585653212bda9df36ba9a1f: Status 404 returned error can't find the container with id 46964adc273ae519b1e21c060ccab66f1abb9768d585653212bda9df36ba9a1f
Sep 18 13:59:28.502092 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:28.502062    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:28.770077 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:28.769978    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0aac26-66c8b4f78c-5p5ns" event={"ID":"e96148d8-4aea-49f4-9664-1298b3d56af4","Type":"ContainerStarted","Data":"c02db53c6778240d874b031ba97d777d3ed38b01a1af6402db666928d4463ce0"}
Sep 18 13:59:28.770077 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:28.770061    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0aac26-66c8b4f78c-5p5ns" event={"ID":"e96148d8-4aea-49f4-9664-1298b3d56af4","Type":"ContainerStarted","Data":"46964adc273ae519b1e21c060ccab66f1abb9768d585653212bda9df36ba9a1f"}
Sep 18 13:59:28.770280 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:28.770177    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0aac26-66c8b4f78c-5p5ns"
Sep 18 13:59:28.791334 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:28.791247    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0aac26-66c8b4f78c-5p5ns" podStartSLOduration=2.35384557 podStartE2EDuration="2.791228792s" totalImagesPullingTime="437.383222ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:59:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:59:28.789685363 +0000 UTC m=+917.842344686" watchObservedRunningTime="2026-09-18 13:59:28.791228792 +0000 UTC m=+917.843888118"
Sep 18 13:59:33.915384 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:33.915346    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-ed4c53/premium-w0-ed4c53-kserve-565c6b654d-mcfqz"]
Sep 18 13:59:33.916014 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:33.915998    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="62972a5e-7296-4486-928b-29f00380ef52" containerName="istio-proxy"
Sep 18 13:59:33.919630 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:33.919612    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ed4c53/premium-w0-ed4c53-kserve-565c6b654d-mcfqz"
Sep 18 13:59:33.919950 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:33.919915    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-377ca2/premium-w4-377ca2-kserve-759c89bc45-4r9k4"
Sep 18 13:59:33.920068 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:33.919958    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-377ca2/premium-w4-377ca2-kserve-759c89bc45-4r9k4"
Sep 18 13:59:33.922199 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:33.922180    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ed4c53\"/\"premium-w0-ed4c53-kserve-self-signed-certs\""
Sep 18 13:59:33.931860 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:33.931838    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ed4c53/premium-w0-ed4c53-kserve-565c6b654d-mcfqz"]
Sep 18 13:59:33.938781 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:33.938756    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-377ca2/premium-w4-377ca2-kserve-759c89bc45-4r9k4"
Sep 18 13:59:33.961614 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:33.961584    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:34.097214 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:34.097178    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66e829fe-01fb-48bc-a708-f1d35e050228-tls-certs\") pod \"premium-w0-ed4c53-kserve-565c6b654d-mcfqz\" (UID: \"66e829fe-01fb-48bc-a708-f1d35e050228\") " pod="e2e-models-e2e-worker-w0-ed4c53/premium-w0-ed4c53-kserve-565c6b654d-mcfqz"
Sep 18 13:59:34.097392 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:34.097262    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66e829fe-01fb-48bc-a708-f1d35e050228-tmp-dir\") pod \"premium-w0-ed4c53-kserve-565c6b654d-mcfqz\" (UID: \"66e829fe-01fb-48bc-a708-f1d35e050228\") " pod="e2e-models-e2e-worker-w0-ed4c53/premium-w0-ed4c53-kserve-565c6b654d-mcfqz"
Sep 18 13:59:34.097462 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:34.097422    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66e829fe-01fb-48bc-a708-f1d35e050228-home\") pod \"premium-w0-ed4c53-kserve-565c6b654d-mcfqz\" (UID: \"66e829fe-01fb-48bc-a708-f1d35e050228\") " pod="e2e-models-e2e-worker-w0-ed4c53/premium-w0-ed4c53-kserve-565c6b654d-mcfqz"
Sep 18 13:59:34.097522 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:34.097467    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66e829fe-01fb-48bc-a708-f1d35e050228-dshm\") pod \"premium-w0-ed4c53-kserve-565c6b654d-mcfqz\" (UID: \"66e829fe-01fb-48bc-a708-f1d35e050228\") " pod="e2e-models-e2e-worker-w0-ed4c53/premium-w0-ed4c53-kserve-565c6b654d-mcfqz"
Sep 18 13:59:34.097522 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:34.097495    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66e829fe-01fb-48bc-a708-f1d35e050228-model-cache\") pod \"premium-w0-ed4c53-kserve-565c6b654d-mcfqz\" (UID: \"66e829fe-01fb-48bc-a708-f1d35e050228\") " pod="e2e-models-e2e-worker-w0-ed4c53/premium-w0-ed4c53-kserve-565c6b654d-mcfqz"
Sep 18 13:59:34.097634 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:34.097589    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jb7g\" (UniqueName: \"kubernetes.io/projected/66e829fe-01fb-48bc-a708-f1d35e050228-kube-api-access-7jb7g\") pod \"premium-w0-ed4c53-kserve-565c6b654d-mcfqz\" (UID: \"66e829fe-01fb-48bc-a708-f1d35e050228\") " pod="e2e-models-e2e-worker-w0-ed4c53/premium-w0-ed4c53-kserve-565c6b654d-mcfqz"
Sep 18 13:59:34.198754 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:34.198671    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66e829fe-01fb-48bc-a708-f1d35e050228-home\") pod \"premium-w0-ed4c53-kserve-565c6b654d-mcfqz\" (UID: \"66e829fe-01fb-48bc-a708-f1d35e050228\") " pod="e2e-models-e2e-worker-w0-ed4c53/premium-w0-ed4c53-kserve-565c6b654d-mcfqz"
Sep 18 13:59:34.198754 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:34.198710    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66e829fe-01fb-48bc-a708-f1d35e050228-dshm\") pod \"premium-w0-ed4c53-kserve-565c6b654d-mcfqz\" (UID: \"66e829fe-01fb-48bc-a708-f1d35e050228\") " pod="e2e-models-e2e-worker-w0-ed4c53/premium-w0-ed4c53-kserve-565c6b654d-mcfqz"
Sep 18 13:59:34.198754 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:34.198725    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66e829fe-01fb-48bc-a708-f1d35e050228-model-cache\") pod \"premium-w0-ed4c53-kserve-565c6b654d-mcfqz\" (UID: \"66e829fe-01fb-48bc-a708-f1d35e050228\") " pod="e2e-models-e2e-worker-w0-ed4c53/premium-w0-ed4c53-kserve-565c6b654d-mcfqz"
Sep 18 13:59:34.199073 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:34.198760    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jb7g\" (UniqueName: \"kubernetes.io/projected/66e829fe-01fb-48bc-a708-f1d35e050228-kube-api-access-7jb7g\") pod \"premium-w0-ed4c53-kserve-565c6b654d-mcfqz\" (UID: \"66e829fe-01fb-48bc-a708-f1d35e050228\") " pod="e2e-models-e2e-worker-w0-ed4c53/premium-w0-ed4c53-kserve-565c6b654d-mcfqz"
Sep 18 13:59:34.199073 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:34.198806    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66e829fe-01fb-48bc-a708-f1d35e050228-tls-certs\") pod \"premium-w0-ed4c53-kserve-565c6b654d-mcfqz\" (UID: \"66e829fe-01fb-48bc-a708-f1d35e050228\") " pod="e2e-models-e2e-worker-w0-ed4c53/premium-w0-ed4c53-kserve-565c6b654d-mcfqz"
Sep 18 13:59:34.199073 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:34.198962    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66e829fe-01fb-48bc-a708-f1d35e050228-tmp-dir\") pod \"premium-w0-ed4c53-kserve-565c6b654d-mcfqz\" (UID: \"66e829fe-01fb-48bc-a708-f1d35e050228\") " pod="e2e-models-e2e-worker-w0-ed4c53/premium-w0-ed4c53-kserve-565c6b654d-mcfqz"
Sep 18 13:59:34.199231 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:34.199157    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66e829fe-01fb-48bc-a708-f1d35e050228-home\") pod \"premium-w0-ed4c53-kserve-565c6b654d-mcfqz\" (UID: \"66e829fe-01fb-48bc-a708-f1d35e050228\") " pod="e2e-models-e2e-worker-w0-ed4c53/premium-w0-ed4c53-kserve-565c6b654d-mcfqz"
Sep 18 13:59:34.199231 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:34.199204    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66e829fe-01fb-48bc-a708-f1d35e050228-model-cache\") pod \"premium-w0-ed4c53-kserve-565c6b654d-mcfqz\" (UID: \"66e829fe-01fb-48bc-a708-f1d35e050228\") " pod="e2e-models-e2e-worker-w0-ed4c53/premium-w0-ed4c53-kserve-565c6b654d-mcfqz"
Sep 18 13:59:34.199322 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:34.199304    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66e829fe-01fb-48bc-a708-f1d35e050228-tmp-dir\") pod \"premium-w0-ed4c53-kserve-565c6b654d-mcfqz\" (UID: \"66e829fe-01fb-48bc-a708-f1d35e050228\") " pod="e2e-models-e2e-worker-w0-ed4c53/premium-w0-ed4c53-kserve-565c6b654d-mcfqz"
Sep 18 13:59:34.201059 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:34.201029    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66e829fe-01fb-48bc-a708-f1d35e050228-dshm\") pod \"premium-w0-ed4c53-kserve-565c6b654d-mcfqz\" (UID: \"66e829fe-01fb-48bc-a708-f1d35e050228\") " pod="e2e-models-e2e-worker-w0-ed4c53/premium-w0-ed4c53-kserve-565c6b654d-mcfqz"
Sep 18 13:59:34.201298 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:34.201280    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66e829fe-01fb-48bc-a708-f1d35e050228-tls-certs\") pod \"premium-w0-ed4c53-kserve-565c6b654d-mcfqz\" (UID: \"66e829fe-01fb-48bc-a708-f1d35e050228\") " pod="e2e-models-e2e-worker-w0-ed4c53/premium-w0-ed4c53-kserve-565c6b654d-mcfqz"
Sep 18 13:59:34.206329 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:34.206298    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jb7g\" (UniqueName: \"kubernetes.io/projected/66e829fe-01fb-48bc-a708-f1d35e050228-kube-api-access-7jb7g\") pod \"premium-w0-ed4c53-kserve-565c6b654d-mcfqz\" (UID: \"66e829fe-01fb-48bc-a708-f1d35e050228\") " pod="e2e-models-e2e-worker-w0-ed4c53/premium-w0-ed4c53-kserve-565c6b654d-mcfqz"
Sep 18 13:59:34.229006 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:34.228980    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ed4c53/premium-w0-ed4c53-kserve-565c6b654d-mcfqz"
Sep 18 13:59:34.503699 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:34.503671    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ed4c53/premium-w0-ed4c53-kserve-565c6b654d-mcfqz"]
Sep 18 13:59:34.505862 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:59:34.505836    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod66e829fe_01fb_48bc_a708_f1d35e050228.slice/crio-24b5d99fd09a61502855b86a38f6a81b95099e406048f000451f6d7748e50adc WatchSource:0}: Error finding container 24b5d99fd09a61502855b86a38f6a81b95099e406048f000451f6d7748e50adc: Status 404 returned error can't find the container with id 24b5d99fd09a61502855b86a38f6a81b95099e406048f000451f6d7748e50adc
Sep 18 13:59:34.781601 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:34.781518    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0aac26-66c8b4f78c-5p5ns"
Sep 18 13:59:34.809587 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:34.809550    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ed4c53/premium-w0-ed4c53-kserve-565c6b654d-mcfqz" event={"ID":"66e829fe-01fb-48bc-a708-f1d35e050228","Type":"ContainerStarted","Data":"9ea591b988b67c650993843d8474dac58dd6ddfafee71cf40666ba2594fb8f96"}
Sep 18 13:59:34.809858 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:34.809837    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ed4c53/premium-w0-ed4c53-kserve-565c6b654d-mcfqz" event={"ID":"66e829fe-01fb-48bc-a708-f1d35e050228","Type":"ContainerStarted","Data":"24b5d99fd09a61502855b86a38f6a81b95099e406048f000451f6d7748e50adc"}
Sep 18 13:59:34.827442 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:34.827393    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-377ca2/premium-w4-377ca2-kserve-759c89bc45-4r9k4"
Sep 18 13:59:34.832458 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:34.832382    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-ed4c53/premium-w0-ed4c53-kserve-565c6b654d-mcfqz" podStartSLOduration=1.8323667160000001 podStartE2EDuration="1.832366716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:59:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:59:34.830002973 +0000 UTC m=+923.882662303" watchObservedRunningTime="2026-09-18 13:59:34.832366716 +0000 UTC m=+923.885026043"
Sep 18 13:59:38.403904 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:38.403864    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 13:59:40.544721 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:40.544672    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:44.229303 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:44.229269    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ed4c53/premium-w0-ed4c53-kserve-565c6b654d-mcfqz"
Sep 18 13:59:44.229721 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:44.229346    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-ed4c53/premium-w0-ed4c53-kserve-565c6b654d-mcfqz"
Sep 18 13:59:44.241766 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:44.241741    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-ed4c53/premium-w0-ed4c53-kserve-565c6b654d-mcfqz"
Sep 18 13:59:44.874028 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:44.873999    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ed4c53/premium-w0-ed4c53-kserve-565c6b654d-mcfqz"
Sep 18 13:59:46.777851 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:46.777788    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:46.900563 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:46.900527    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:47.039091 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:47.038993    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:48.888266 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:48.888172    2577 generic.go:356] "Generic (PLEG): container finished" podID="f8d985f0-b51e-4a56-9840-179887b8f54d" containerID="af42580ad4adcd4beca73bc5031659a1465d2d9ce9c2fbbb02d964d4f5b800e2" exitCode=0
Sep 18 13:59:48.888266 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:48.888244    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd" event={"ID":"f8d985f0-b51e-4a56-9840-179887b8f54d","Type":"ContainerDied","Data":"af42580ad4adcd4beca73bc5031659a1465d2d9ce9c2fbbb02d964d4f5b800e2"}
Sep 18 13:59:48.888702 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:48.888610    2577 scope.go:117] "RemoveContainer" containerID="af42580ad4adcd4beca73bc5031659a1465d2d9ce9c2fbbb02d964d4f5b800e2"
Sep 18 13:59:48.889761 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:48.889734    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 13:59:48.889854 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:48.889801    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 13:59:48.889901 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:48.889854    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 13:59:49.245711 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:49.245590    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:49.905043 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:49.905002    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd" event={"ID":"f8d985f0-b51e-4a56-9840-179887b8f54d","Type":"ContainerStarted","Data":"ceaafafddd169638b3ad757ba2a569f9c2bfdb5686bf4096a27e8de5c9ca896c"}
Sep 18 13:59:49.976708 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:49.976675    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:50.400066 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:50.399940    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd"
Sep 18 13:59:50.401940 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:50.401890    2577 patch_prober.go:28] interesting pod/e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 18 13:59:50.402063 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:50.401972    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd" podUID="f8d985f0-b51e-4a56-9840-179887b8f54d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 18 13:59:50.835932 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:50.835851    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:51.399616 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:51.399570    2577 patch_prober.go:28] interesting pod/e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 18 13:59:51.400151 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:51.399651    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd" podUID="f8d985f0-b51e-4a56-9840-179887b8f54d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 18 13:59:51.490105 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:51.490060    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:51.702224 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:51.702135    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-b63f26/test-model-b63f26-kserve-8589457995-qslk6"]
Sep 18 13:59:51.707854 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:51.707578    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b63f26/test-model-b63f26-kserve-8589457995-qslk6"
Sep 18 13:59:51.710019 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:51.709996    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-b63f26\"/\"test-model-b63f26-kserve-self-signed-certs\""
Sep 18 13:59:51.710791 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:51.710771    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-b63f26\"/\"kube-root-ca.crt\""
Sep 18 13:59:51.711127 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:51.711110    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-b63f26\"/\"default-dockercfg-hnzjg\""
Sep 18 13:59:51.711413 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:51.711395    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-b63f26\"/\"openshift-service-ca.crt\""
Sep 18 13:59:51.715918 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:51.715896    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-b63f26/test-model-b63f26-kserve-8589457995-qslk6"]
Sep 18 13:59:51.728193 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:51.728151    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:51.762072 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:51.762039    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/675617d1-fe1b-4ec5-9ee6-03ce4e0859bf-dshm\") pod \"test-model-b63f26-kserve-8589457995-qslk6\" (UID: \"675617d1-fe1b-4ec5-9ee6-03ce4e0859bf\") " pod="ai-tenant-e2e-inf-b-b63f26/test-model-b63f26-kserve-8589457995-qslk6"
Sep 18 13:59:51.762197 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:51.762101    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/675617d1-fe1b-4ec5-9ee6-03ce4e0859bf-model-cache\") pod \"test-model-b63f26-kserve-8589457995-qslk6\" (UID: \"675617d1-fe1b-4ec5-9ee6-03ce4e0859bf\") " pod="ai-tenant-e2e-inf-b-b63f26/test-model-b63f26-kserve-8589457995-qslk6"
Sep 18 13:59:51.762197 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:51.762138    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/675617d1-fe1b-4ec5-9ee6-03ce4e0859bf-home\") pod \"test-model-b63f26-kserve-8589457995-qslk6\" (UID: \"675617d1-fe1b-4ec5-9ee6-03ce4e0859bf\") " pod="ai-tenant-e2e-inf-b-b63f26/test-model-b63f26-kserve-8589457995-qslk6"
Sep 18 13:59:51.762197 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:51.762165    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/675617d1-fe1b-4ec5-9ee6-03ce4e0859bf-tls-certs\") pod \"test-model-b63f26-kserve-8589457995-qslk6\" (UID: \"675617d1-fe1b-4ec5-9ee6-03ce4e0859bf\") " pod="ai-tenant-e2e-inf-b-b63f26/test-model-b63f26-kserve-8589457995-qslk6"
Sep 18 13:59:51.762316 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:51.762245    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs2gv\" (UniqueName: \"kubernetes.io/projected/675617d1-fe1b-4ec5-9ee6-03ce4e0859bf-kube-api-access-zs2gv\") pod \"test-model-b63f26-kserve-8589457995-qslk6\" (UID: \"675617d1-fe1b-4ec5-9ee6-03ce4e0859bf\") " pod="ai-tenant-e2e-inf-b-b63f26/test-model-b63f26-kserve-8589457995-qslk6"
Sep 18 13:59:51.762316 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:51.762277    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/675617d1-fe1b-4ec5-9ee6-03ce4e0859bf-tmp-dir\") pod \"test-model-b63f26-kserve-8589457995-qslk6\" (UID: \"675617d1-fe1b-4ec5-9ee6-03ce4e0859bf\") " pod="ai-tenant-e2e-inf-b-b63f26/test-model-b63f26-kserve-8589457995-qslk6"
Sep 18 13:59:51.863034 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:51.862998    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/675617d1-fe1b-4ec5-9ee6-03ce4e0859bf-home\") pod \"test-model-b63f26-kserve-8589457995-qslk6\" (UID: \"675617d1-fe1b-4ec5-9ee6-03ce4e0859bf\") " pod="ai-tenant-e2e-inf-b-b63f26/test-model-b63f26-kserve-8589457995-qslk6"
Sep 18 13:59:51.863223 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:51.863043    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/675617d1-fe1b-4ec5-9ee6-03ce4e0859bf-tls-certs\") pod \"test-model-b63f26-kserve-8589457995-qslk6\" (UID: \"675617d1-fe1b-4ec5-9ee6-03ce4e0859bf\") " pod="ai-tenant-e2e-inf-b-b63f26/test-model-b63f26-kserve-8589457995-qslk6"
Sep 18 13:59:51.863223 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:51.863085    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs2gv\" (UniqueName: \"kubernetes.io/projected/675617d1-fe1b-4ec5-9ee6-03ce4e0859bf-kube-api-access-zs2gv\") pod \"test-model-b63f26-kserve-8589457995-qslk6\" (UID: \"675617d1-fe1b-4ec5-9ee6-03ce4e0859bf\") " pod="ai-tenant-e2e-inf-b-b63f26/test-model-b63f26-kserve-8589457995-qslk6"
Sep 18 13:59:51.863345 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:51.863225    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/675617d1-fe1b-4ec5-9ee6-03ce4e0859bf-tmp-dir\") pod \"test-model-b63f26-kserve-8589457995-qslk6\" (UID: \"675617d1-fe1b-4ec5-9ee6-03ce4e0859bf\") " pod="ai-tenant-e2e-inf-b-b63f26/test-model-b63f26-kserve-8589457995-qslk6"
Sep 18 13:59:51.863345 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:51.863283    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/675617d1-fe1b-4ec5-9ee6-03ce4e0859bf-dshm\") pod \"test-model-b63f26-kserve-8589457995-qslk6\" (UID: \"675617d1-fe1b-4ec5-9ee6-03ce4e0859bf\") " pod="ai-tenant-e2e-inf-b-b63f26/test-model-b63f26-kserve-8589457995-qslk6"
Sep 18 13:59:51.863452 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:51.863353    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/675617d1-fe1b-4ec5-9ee6-03ce4e0859bf-model-cache\") pod \"test-model-b63f26-kserve-8589457995-qslk6\" (UID: \"675617d1-fe1b-4ec5-9ee6-03ce4e0859bf\") " pod="ai-tenant-e2e-inf-b-b63f26/test-model-b63f26-kserve-8589457995-qslk6"
Sep 18 13:59:51.863506 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:51.863463    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/675617d1-fe1b-4ec5-9ee6-03ce4e0859bf-home\") pod \"test-model-b63f26-kserve-8589457995-qslk6\" (UID: \"675617d1-fe1b-4ec5-9ee6-03ce4e0859bf\") " pod="ai-tenant-e2e-inf-b-b63f26/test-model-b63f26-kserve-8589457995-qslk6"
Sep 18 13:59:51.863573 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:51.863555    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/675617d1-fe1b-4ec5-9ee6-03ce4e0859bf-tmp-dir\") pod \"test-model-b63f26-kserve-8589457995-qslk6\" (UID: \"675617d1-fe1b-4ec5-9ee6-03ce4e0859bf\") " pod="ai-tenant-e2e-inf-b-b63f26/test-model-b63f26-kserve-8589457995-qslk6"
Sep 18 13:59:51.863724 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:51.863695    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/675617d1-fe1b-4ec5-9ee6-03ce4e0859bf-model-cache\") pod \"test-model-b63f26-kserve-8589457995-qslk6\" (UID: \"675617d1-fe1b-4ec5-9ee6-03ce4e0859bf\") " pod="ai-tenant-e2e-inf-b-b63f26/test-model-b63f26-kserve-8589457995-qslk6"
Sep 18 13:59:51.865578 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:51.865555    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/675617d1-fe1b-4ec5-9ee6-03ce4e0859bf-dshm\") pod \"test-model-b63f26-kserve-8589457995-qslk6\" (UID: \"675617d1-fe1b-4ec5-9ee6-03ce4e0859bf\") " pod="ai-tenant-e2e-inf-b-b63f26/test-model-b63f26-kserve-8589457995-qslk6"
Sep 18 13:59:51.865777 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:51.865759    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/675617d1-fe1b-4ec5-9ee6-03ce4e0859bf-tls-certs\") pod \"test-model-b63f26-kserve-8589457995-qslk6\" (UID: \"675617d1-fe1b-4ec5-9ee6-03ce4e0859bf\") " pod="ai-tenant-e2e-inf-b-b63f26/test-model-b63f26-kserve-8589457995-qslk6"
Sep 18 13:59:51.872105 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:51.872080    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs2gv\" (UniqueName: \"kubernetes.io/projected/675617d1-fe1b-4ec5-9ee6-03ce4e0859bf-kube-api-access-zs2gv\") pod \"test-model-b63f26-kserve-8589457995-qslk6\" (UID: \"675617d1-fe1b-4ec5-9ee6-03ce4e0859bf\") " pod="ai-tenant-e2e-inf-b-b63f26/test-model-b63f26-kserve-8589457995-qslk6"
Sep 18 13:59:52.020512 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:52.020428    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b63f26/test-model-b63f26-kserve-8589457995-qslk6"
Sep 18 13:59:52.268229 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:52.268173    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:52.302756 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:52.302643    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-b63f26/test-model-b63f26-kserve-8589457995-qslk6"]
Sep 18 13:59:52.306430 ip-10-0-141-222 kubenswrapper[2577]: W0918 13:59:52.306396    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod675617d1_fe1b_4ec5_9ee6_03ce4e0859bf.slice/crio-a0538a68d214741bf2cacbe1e9b7109a6043d6acdd3e14acf5a12e33911101e1 WatchSource:0}: Error finding container a0538a68d214741bf2cacbe1e9b7109a6043d6acdd3e14acf5a12e33911101e1: Status 404 returned error can't find the container with id a0538a68d214741bf2cacbe1e9b7109a6043d6acdd3e14acf5a12e33911101e1
Sep 18 13:59:52.399932 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:52.399893    2577 patch_prober.go:28] interesting pod/e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 18 13:59:52.400934 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:52.400482    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd" podUID="f8d985f0-b51e-4a56-9840-179887b8f54d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 18 13:59:52.724737 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:52.724700    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd"
Sep 18 13:59:52.923948 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:52.923909    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b63f26/test-model-b63f26-kserve-8589457995-qslk6" event={"ID":"675617d1-fe1b-4ec5-9ee6-03ce4e0859bf","Type":"ContainerStarted","Data":"eab18aa61409a2b1a849df370bbeadc6e0d37fd3f9e94878109da31ca218c6f1"}
Sep 18 13:59:52.924101 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:52.923959    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b63f26/test-model-b63f26-kserve-8589457995-qslk6" event={"ID":"675617d1-fe1b-4ec5-9ee6-03ce4e0859bf","Type":"ContainerStarted","Data":"a0538a68d214741bf2cacbe1e9b7109a6043d6acdd3e14acf5a12e33911101e1"}
Sep 18 13:59:52.941555 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:52.941492    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-b63f26/test-model-b63f26-kserve-8589457995-qslk6" podStartSLOduration=1.941470898 podStartE2EDuration="1.941470898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:59:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:59:52.940543849 +0000 UTC m=+941.993203177" watchObservedRunningTime="2026-09-18 13:59:52.941470898 +0000 UTC m=+941.994130226"
Sep 18 13:59:53.142940 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:53.142909    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:53.400078 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:53.399972    2577 patch_prober.go:28] interesting pod/e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 18 13:59:53.400078 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:53.400056    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd" podUID="f8d985f0-b51e-4a56-9840-179887b8f54d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 18 13:59:53.678176 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:53.678047    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:54.400370 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:54.400327    2577 patch_prober.go:28] interesting pod/e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 18 13:59:54.401130 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:54.401098    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd" podUID="f8d985f0-b51e-4a56-9840-179887b8f54d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 18 13:59:55.403551 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:55.403514    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd"
Sep 18 13:59:55.404763 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:55.404743    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd"
Sep 18 13:59:57.529500 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:57.529468    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:58.336499 ip-10-0-141-222 kubenswrapper[2577]: I0918 13:59:58.336466    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:00.182990 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:00.182948    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-377ca2-29829000-rlwmp"]
Sep 18 14:00:00.186368 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:00.186349    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-377ca2-29829000-rlwmp"
Sep 18 14:00:00.189057 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:00.189032    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cxtb2\""
Sep 18 14:00:00.196488 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:00.196466    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-377ca2-29829000-rlwmp"
Sep 18 14:00:00.204352 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:00.204329    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-df2eb8-29829000-729fw"]
Sep 18 14:00:00.208540 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:00.208523    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-df2eb8-29829000-729fw"
Sep 18 14:00:00.237029 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:00.237001    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-df2eb8-29829000-729fw"
Sep 18 14:00:00.242063 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:00.242035    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-377ca2-29829000-rlwmp"]
Sep 18 14:00:00.252032 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:00.251995    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-df2eb8-29829000-729fw"]
Sep 18 14:00:00.484178 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:00.484151    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-377ca2-29829000-rlwmp"]
Sep 18 14:00:00.486665 ip-10-0-141-222 kubenswrapper[2577]: W0918 14:00:00.486641    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8df24333_37c6_435f_b858_de1708adbf0f.slice/crio-19fe49ef09d6fcbccc3abe10f819fd2b86bff9fb9b91bfadd3613b324ae1ce82 WatchSource:0}: Error finding container 19fe49ef09d6fcbccc3abe10f819fd2b86bff9fb9b91bfadd3613b324ae1ce82: Status 404 returned error can't find the container with id 19fe49ef09d6fcbccc3abe10f819fd2b86bff9fb9b91bfadd3613b324ae1ce82
Sep 18 14:00:00.545520 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:00.545478    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-df2eb8-29829000-729fw"]
Sep 18 14:00:00.550522 ip-10-0-141-222 kubenswrapper[2577]: W0918 14:00:00.550487    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1abe0e88_3fae_4d49_82d0_ad9d4fc91be3.slice/crio-9c64b15c9fb2519dd99f92fbb4716966d7412f5083b1052c813e956c758d32bb WatchSource:0}: Error finding container 9c64b15c9fb2519dd99f92fbb4716966d7412f5083b1052c813e956c758d32bb: Status 404 returned error can't find the container with id 9c64b15c9fb2519dd99f92fbb4716966d7412f5083b1052c813e956c758d32bb
Sep 18 14:00:00.971291 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:00.971241    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-df2eb8-29829000-729fw" event={"ID":"1abe0e88-3fae-4d49-82d0-ad9d4fc91be3","Type":"ContainerStarted","Data":"4e04e952b9e9b27d6cccb0e1693e6c5f1b45ebe3cb0959ec95bdba6b37b18177"}
Sep 18 14:00:00.971484 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:00.971311    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-df2eb8-29829000-729fw" event={"ID":"1abe0e88-3fae-4d49-82d0-ad9d4fc91be3","Type":"ContainerStarted","Data":"9c64b15c9fb2519dd99f92fbb4716966d7412f5083b1052c813e956c758d32bb"}
Sep 18 14:00:00.972878 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:00.972848    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-377ca2-29829000-rlwmp" event={"ID":"8df24333-37c6-435f-b858-de1708adbf0f","Type":"ContainerStarted","Data":"a6029672d44e4c1954bbd60e36c6546a24ad26322c8f547b948d9e4b8029bba1"}
Sep 18 14:00:00.972992 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:00.972895    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-377ca2-29829000-rlwmp" event={"ID":"8df24333-37c6-435f-b858-de1708adbf0f","Type":"ContainerStarted","Data":"19fe49ef09d6fcbccc3abe10f819fd2b86bff9fb9b91bfadd3613b324ae1ce82"}
Sep 18 14:00:00.985995 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:00.985946    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-df2eb8-29829000-729fw" podStartSLOduration=0.985929539 podStartE2EDuration="985.929539ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:00:00.984565487 +0000 UTC m=+950.037224813" watchObservedRunningTime="2026-09-18 14:00:00.985929539 +0000 UTC m=+950.038588865"
Sep 18 14:00:00.999732 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:00.999682    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-377ca2-29829000-rlwmp" podStartSLOduration=0.999663264 podStartE2EDuration="999.663264ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:00:00.997804956 +0000 UTC m=+950.050464285" watchObservedRunningTime="2026-09-18 14:00:00.999663264 +0000 UTC m=+950.052322649"
Sep 18 14:00:01.980098 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:01.980062    2577 generic.go:356] "Generic (PLEG): container finished" podID="1abe0e88-3fae-4d49-82d0-ad9d4fc91be3" containerID="4e04e952b9e9b27d6cccb0e1693e6c5f1b45ebe3cb0959ec95bdba6b37b18177" exitCode=0
Sep 18 14:00:01.980528 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:01.980148    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-df2eb8-29829000-729fw" event={"ID":"1abe0e88-3fae-4d49-82d0-ad9d4fc91be3","Type":"ContainerDied","Data":"4e04e952b9e9b27d6cccb0e1693e6c5f1b45ebe3cb0959ec95bdba6b37b18177"}
Sep 18 14:00:01.981713 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:01.981687    2577 generic.go:356] "Generic (PLEG): container finished" podID="8df24333-37c6-435f-b858-de1708adbf0f" containerID="a6029672d44e4c1954bbd60e36c6546a24ad26322c8f547b948d9e4b8029bba1" exitCode=0
Sep 18 14:00:01.981809 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:01.981736    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-377ca2-29829000-rlwmp" event={"ID":"8df24333-37c6-435f-b858-de1708adbf0f","Type":"ContainerDied","Data":"a6029672d44e4c1954bbd60e36c6546a24ad26322c8f547b948d9e4b8029bba1"}
Sep 18 14:00:02.020967 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:02.020928    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-b63f26/test-model-b63f26-kserve-8589457995-qslk6"
Sep 18 14:00:02.021178 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:02.021008    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-b63f26/test-model-b63f26-kserve-8589457995-qslk6"
Sep 18 14:00:02.033747 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:02.033718    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-b63f26/test-model-b63f26-kserve-8589457995-qslk6"
Sep 18 14:00:02.405436 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:02.405399    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 14:00:03.002140 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:03.002108    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-b63f26/test-model-b63f26-kserve-8589457995-qslk6"
Sep 18 14:00:03.147027 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:03.147001    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-377ca2-29829000-rlwmp"
Sep 18 14:00:03.175278 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:03.175252    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-df2eb8-29829000-729fw"
Sep 18 14:00:03.997055 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:03.997023    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-df2eb8-29829000-729fw"
Sep 18 14:00:03.997055 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:03.997045    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-df2eb8-29829000-729fw" event={"ID":"1abe0e88-3fae-4d49-82d0-ad9d4fc91be3","Type":"ContainerDied","Data":"9c64b15c9fb2519dd99f92fbb4716966d7412f5083b1052c813e956c758d32bb"}
Sep 18 14:00:03.997274 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:03.997091    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c64b15c9fb2519dd99f92fbb4716966d7412f5083b1052c813e956c758d32bb"
Sep 18 14:00:03.998422 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:03.998405    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-377ca2-29829000-rlwmp"
Sep 18 14:00:03.998487 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:03.998441    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-377ca2-29829000-rlwmp" event={"ID":"8df24333-37c6-435f-b858-de1708adbf0f","Type":"ContainerDied","Data":"19fe49ef09d6fcbccc3abe10f819fd2b86bff9fb9b91bfadd3613b324ae1ce82"}
Sep 18 14:00:03.998540 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:03.998486    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19fe49ef09d6fcbccc3abe10f819fd2b86bff9fb9b91bfadd3613b324ae1ce82"
Sep 18 14:00:05.295192 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:05.295149    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:05.420564 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:05.420532    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:06.214624 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:06.214586    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:06.316904 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:06.316862    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:07.768560 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:07.768528    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:14.429217 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:14.429172    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:14.530218 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:14.530185    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:20.876061 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:20.876032    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:20.976157 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:20.976124    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:21.097384 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:21.097343    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:21.431835 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:21.431791    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:21.694573 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:21.694489    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:27.039493 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:27.039402    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:27.832342 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:27.832309    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:27.970024 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:27.969992    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:28.063776 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:28.063748    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:28.913871 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:28.913808    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:30.013804 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:30.013762    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:30.615329 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:30.615292    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 14:00:30.749141 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:30.749110    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:31.461879 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:31.461841    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:32.297887 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:32.297851    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:32.392673 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:32.392632    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:32.493009 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:32.492968    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:32.827861 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:32.827808    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:32.928624 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:32.928579    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:54.552588 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:54.552551    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:55.308419 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:55.308384    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:55.490619 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:55.490585    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-0aac26/explicit-model-0aac26-kserve-659b975947-p4n4t"]
Sep 18 14:00:55.490996 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:55.490982    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1abe0e88-3fae-4d49-82d0-ad9d4fc91be3" containerName="cleanup"
Sep 18 14:00:55.491046 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:55.491001    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1abe0e88-3fae-4d49-82d0-ad9d4fc91be3" containerName="cleanup"
Sep 18 14:00:55.491046 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:55.491021    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8df24333-37c6-435f-b858-de1708adbf0f" containerName="cleanup"
Sep 18 14:00:55.491046 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:55.491027    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8df24333-37c6-435f-b858-de1708adbf0f" containerName="cleanup"
Sep 18 14:00:55.491149 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:55.491089    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8df24333-37c6-435f-b858-de1708adbf0f" containerName="cleanup"
Sep 18 14:00:55.491149 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:55.491098    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1abe0e88-3fae-4d49-82d0-ad9d4fc91be3" containerName="cleanup"
Sep 18 14:00:55.501453 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:55.501415    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-0aac26/explicit-model-0aac26-kserve-659b975947-p4n4t"
Sep 18 14:00:55.504262 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:55.504220    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-0aac26\"/\"kube-root-ca.crt\""
Sep 18 14:00:55.504262 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:55.504248    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-0aac26\"/\"explicit-model-0aac26-kserve-self-signed-certs\""
Sep 18 14:00:55.504463 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:55.504275    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-0aac26\"/\"openshift-service-ca.crt\""
Sep 18 14:00:55.504463 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:55.504228    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-0aac26\"/\"default-dockercfg-f5pmh\""
Sep 18 14:00:55.504768 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:55.504747    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-0aac26/explicit-model-0aac26-kserve-659b975947-p4n4t"]
Sep 18 14:00:55.510159 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:55.510104    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:55.653196 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:55.653163    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/987a4375-4d81-4073-8228-c7ea5249551e-home\") pod \"explicit-model-0aac26-kserve-659b975947-p4n4t\" (UID: \"987a4375-4d81-4073-8228-c7ea5249551e\") " pod="ai-tenant-e2e-explicit-0aac26/explicit-model-0aac26-kserve-659b975947-p4n4t"
Sep 18 14:00:55.653645 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:55.653207    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/987a4375-4d81-4073-8228-c7ea5249551e-tls-certs\") pod \"explicit-model-0aac26-kserve-659b975947-p4n4t\" (UID: \"987a4375-4d81-4073-8228-c7ea5249551e\") " pod="ai-tenant-e2e-explicit-0aac26/explicit-model-0aac26-kserve-659b975947-p4n4t"
Sep 18 14:00:55.653645 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:55.653236    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj5k7\" (UniqueName: \"kubernetes.io/projected/987a4375-4d81-4073-8228-c7ea5249551e-kube-api-access-pj5k7\") pod \"explicit-model-0aac26-kserve-659b975947-p4n4t\" (UID: \"987a4375-4d81-4073-8228-c7ea5249551e\") " pod="ai-tenant-e2e-explicit-0aac26/explicit-model-0aac26-kserve-659b975947-p4n4t"
Sep 18 14:00:55.653645 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:55.653321    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/987a4375-4d81-4073-8228-c7ea5249551e-model-cache\") pod \"explicit-model-0aac26-kserve-659b975947-p4n4t\" (UID: \"987a4375-4d81-4073-8228-c7ea5249551e\") " pod="ai-tenant-e2e-explicit-0aac26/explicit-model-0aac26-kserve-659b975947-p4n4t"
Sep 18 14:00:55.653645 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:55.653384    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/987a4375-4d81-4073-8228-c7ea5249551e-dshm\") pod \"explicit-model-0aac26-kserve-659b975947-p4n4t\" (UID: \"987a4375-4d81-4073-8228-c7ea5249551e\") " pod="ai-tenant-e2e-explicit-0aac26/explicit-model-0aac26-kserve-659b975947-p4n4t"
Sep 18 14:00:55.653645 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:55.653416    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/987a4375-4d81-4073-8228-c7ea5249551e-tmp-dir\") pod \"explicit-model-0aac26-kserve-659b975947-p4n4t\" (UID: \"987a4375-4d81-4073-8228-c7ea5249551e\") " pod="ai-tenant-e2e-explicit-0aac26/explicit-model-0aac26-kserve-659b975947-p4n4t"
Sep 18 14:00:55.753962 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:55.753927    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/987a4375-4d81-4073-8228-c7ea5249551e-home\") pod \"explicit-model-0aac26-kserve-659b975947-p4n4t\" (UID: \"987a4375-4d81-4073-8228-c7ea5249551e\") " pod="ai-tenant-e2e-explicit-0aac26/explicit-model-0aac26-kserve-659b975947-p4n4t"
Sep 18 14:00:55.754145 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:55.753972    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/987a4375-4d81-4073-8228-c7ea5249551e-tls-certs\") pod \"explicit-model-0aac26-kserve-659b975947-p4n4t\" (UID: \"987a4375-4d81-4073-8228-c7ea5249551e\") " pod="ai-tenant-e2e-explicit-0aac26/explicit-model-0aac26-kserve-659b975947-p4n4t"
Sep 18 14:00:55.754145 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:55.754005    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj5k7\" (UniqueName: \"kubernetes.io/projected/987a4375-4d81-4073-8228-c7ea5249551e-kube-api-access-pj5k7\") pod \"explicit-model-0aac26-kserve-659b975947-p4n4t\" (UID: \"987a4375-4d81-4073-8228-c7ea5249551e\") " pod="ai-tenant-e2e-explicit-0aac26/explicit-model-0aac26-kserve-659b975947-p4n4t"
Sep 18 14:00:55.754145 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:55.754049    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/987a4375-4d81-4073-8228-c7ea5249551e-model-cache\") pod \"explicit-model-0aac26-kserve-659b975947-p4n4t\" (UID: \"987a4375-4d81-4073-8228-c7ea5249551e\") " pod="ai-tenant-e2e-explicit-0aac26/explicit-model-0aac26-kserve-659b975947-p4n4t"
Sep 18 14:00:55.754145 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:55.754089    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/987a4375-4d81-4073-8228-c7ea5249551e-dshm\") pod \"explicit-model-0aac26-kserve-659b975947-p4n4t\" (UID: \"987a4375-4d81-4073-8228-c7ea5249551e\") " pod="ai-tenant-e2e-explicit-0aac26/explicit-model-0aac26-kserve-659b975947-p4n4t"
Sep 18 14:00:55.754729 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:55.754210    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/987a4375-4d81-4073-8228-c7ea5249551e-tmp-dir\") pod \"explicit-model-0aac26-kserve-659b975947-p4n4t\" (UID: \"987a4375-4d81-4073-8228-c7ea5249551e\") " pod="ai-tenant-e2e-explicit-0aac26/explicit-model-0aac26-kserve-659b975947-p4n4t"
Sep 18 14:00:55.754729 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:55.754407    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/987a4375-4d81-4073-8228-c7ea5249551e-home\") pod \"explicit-model-0aac26-kserve-659b975947-p4n4t\" (UID: \"987a4375-4d81-4073-8228-c7ea5249551e\") " pod="ai-tenant-e2e-explicit-0aac26/explicit-model-0aac26-kserve-659b975947-p4n4t"
Sep 18 14:00:55.754729 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:55.754422    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/987a4375-4d81-4073-8228-c7ea5249551e-model-cache\") pod \"explicit-model-0aac26-kserve-659b975947-p4n4t\" (UID: \"987a4375-4d81-4073-8228-c7ea5249551e\") " pod="ai-tenant-e2e-explicit-0aac26/explicit-model-0aac26-kserve-659b975947-p4n4t"
Sep 18 14:00:55.754729 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:55.754627    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/987a4375-4d81-4073-8228-c7ea5249551e-tmp-dir\") pod \"explicit-model-0aac26-kserve-659b975947-p4n4t\" (UID: \"987a4375-4d81-4073-8228-c7ea5249551e\") " pod="ai-tenant-e2e-explicit-0aac26/explicit-model-0aac26-kserve-659b975947-p4n4t"
Sep 18 14:00:55.756607 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:55.756583    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/987a4375-4d81-4073-8228-c7ea5249551e-dshm\") pod \"explicit-model-0aac26-kserve-659b975947-p4n4t\" (UID: \"987a4375-4d81-4073-8228-c7ea5249551e\") " pod="ai-tenant-e2e-explicit-0aac26/explicit-model-0aac26-kserve-659b975947-p4n4t"
Sep 18 14:00:55.756728 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:55.756709    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/987a4375-4d81-4073-8228-c7ea5249551e-tls-certs\") pod \"explicit-model-0aac26-kserve-659b975947-p4n4t\" (UID: \"987a4375-4d81-4073-8228-c7ea5249551e\") " pod="ai-tenant-e2e-explicit-0aac26/explicit-model-0aac26-kserve-659b975947-p4n4t"
Sep 18 14:00:55.761642 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:55.761620    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj5k7\" (UniqueName: \"kubernetes.io/projected/987a4375-4d81-4073-8228-c7ea5249551e-kube-api-access-pj5k7\") pod \"explicit-model-0aac26-kserve-659b975947-p4n4t\" (UID: \"987a4375-4d81-4073-8228-c7ea5249551e\") " pod="ai-tenant-e2e-explicit-0aac26/explicit-model-0aac26-kserve-659b975947-p4n4t"
Sep 18 14:00:55.812336 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:55.812306    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-0aac26/explicit-model-0aac26-kserve-659b975947-p4n4t"
Sep 18 14:00:56.088312 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:56.088285    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-0aac26/explicit-model-0aac26-kserve-659b975947-p4n4t"]
Sep 18 14:00:56.091161 ip-10-0-141-222 kubenswrapper[2577]: W0918 14:00:56.091131    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod987a4375_4d81_4073_8228_c7ea5249551e.slice/crio-7ca5bf180da84a9655193cf1726b8b462f997aa20b817d61b8b5438bf429ccc0 WatchSource:0}: Error finding container 7ca5bf180da84a9655193cf1726b8b462f997aa20b817d61b8b5438bf429ccc0: Status 404 returned error can't find the container with id 7ca5bf180da84a9655193cf1726b8b462f997aa20b817d61b8b5438bf429ccc0
Sep 18 14:00:56.098964 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:56.098944    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:56.300944 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:56.300907    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-0aac26/explicit-model-0aac26-kserve-659b975947-p4n4t" event={"ID":"987a4375-4d81-4073-8228-c7ea5249551e","Type":"ContainerStarted","Data":"78401eb5cfcb7040ec1ce98b8812bafd8f1d6fad9bb1a4fa81699a15c7085083"}
Sep 18 14:00:56.300944 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:56.300951    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-0aac26/explicit-model-0aac26-kserve-659b975947-p4n4t" event={"ID":"987a4375-4d81-4073-8228-c7ea5249551e","Type":"ContainerStarted","Data":"7ca5bf180da84a9655193cf1726b8b462f997aa20b817d61b8b5438bf429ccc0"}
Sep 18 14:00:56.319009 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:56.318956    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-0aac26/explicit-model-0aac26-kserve-659b975947-p4n4t" podStartSLOduration=1.3189417429999999 podStartE2EDuration="1.318941743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:00:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:00:56.316945778 +0000 UTC m=+1005.369605106" watchObservedRunningTime="2026-09-18 14:00:56.318941743 +0000 UTC m=+1005.371601069"
Sep 18 14:00:56.752278 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:56.752243    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:57.643551 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:57.643523    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:58.025628 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:58.025524    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:58.436176 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:58.436149    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:58.541016 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:58.540980    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:58.637559 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:58.637513    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:58.983798 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:58.983757    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:59.363356 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:00:59.363326    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:03.307081 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:03.307046    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 14:01:04.897131 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:04.897100    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:04.968859 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:04.968807    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:05.812716 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:05.812679    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-0aac26/explicit-model-0aac26-kserve-659b975947-p4n4t"
Sep 18 14:01:05.812878 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:05.812740    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-0aac26/explicit-model-0aac26-kserve-659b975947-p4n4t"
Sep 18 14:01:05.825762 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:05.825738    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-0aac26/explicit-model-0aac26-kserve-659b975947-p4n4t"
Sep 18 14:01:06.082796 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:06.082715    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:06.369211 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:06.369173    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-0aac26/explicit-model-0aac26-kserve-659b975947-p4n4t"
Sep 18 14:01:06.861608 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:06.861577    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:07.642535 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:07.642499    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:08.435847 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:08.435795    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:08.839078 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:08.839044    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:09.348903 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:09.348871    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:10.051814 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:10.051777    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:10.718754 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:10.718718    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:11.977097 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:11.977062    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:12.048753 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:12.048717    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:15.828651 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:15.828616    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:15.930998 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:15.930962    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:16.018079 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:16.018049    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:16.092476 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:16.092426    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:16.403296 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:16.403212    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:16.477512 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:16.477473    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:17.584358 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:17.584320    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:18.568544 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:18.568508    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:19.731005 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:19.730968    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:20.812803 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:20.812768    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:21.528424 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:21.528394    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:23.017430 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:23.017392    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:23.118378 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:23.118344    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:23.256482 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:23.256451    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:23.630531 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:23.630497    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:23.725467 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:23.725435    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:23.824861 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:23.824810    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:30.028394 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:30.028360    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:30.132107 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:30.132073    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:30.230199 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:30.230156    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:30.404609 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:30.404575    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:31.732308 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:31.732274    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:32.460510 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:32.460474    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:32.930354 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:32.930316    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:33.346328 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:33.346294    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:34.197775 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:34.197732    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:34.923416 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:34.923364    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:35.004328 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:35.004296    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 14:01:35.788034 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:35.788003    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-0aac26/explicit-model-0aac26-kserve-659b975947-p4n4t"]
Sep 18 14:01:35.790676 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:35.790650    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:35.860145 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:35.860111    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:36.435709 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:36.435676    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:37.253055 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:37.253013    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:37.358000 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:37.357968    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:37.454496 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:37.454459    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:38.028115 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:38.028081    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:38.134708 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:38.134674    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:38.473283 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:38.473241    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:38.572024 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:38.571991    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:40.592253 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:40.592218    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:41.230325 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:41.230285    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:41.326204 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:41.326170    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:41.425276 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:41.425242    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:42.202677 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:42.202642    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:42.852596 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:42.852555    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-0aac26-66c8b4f78c-5p5ns"]
Sep 18 14:01:42.852814 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:42.852787    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0aac26-66c8b4f78c-5p5ns" podUID="e96148d8-4aea-49f4-9664-1298b3d56af4" containerName="maas-api" containerID="cri-o://c02db53c6778240d874b031ba97d777d3ed38b01a1af6402db666928d4463ce0" gracePeriod=30
Sep 18 14:01:42.882146 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:42.882097    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:42.961684 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:42.961642    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:43.029285 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:43.029243    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:43.114221 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:43.114158    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0aac26-66c8b4f78c-5p5ns"
Sep 18 14:01:43.130346 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:43.130314    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:43.196166 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:43.196130    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e96148d8-4aea-49f4-9664-1298b3d56af4-openshift-service-ca\") pod \"e96148d8-4aea-49f4-9664-1298b3d56af4\" (UID: \"e96148d8-4aea-49f4-9664-1298b3d56af4\") "
Sep 18 14:01:43.196373 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:43.196198    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sm458\" (UniqueName: \"kubernetes.io/projected/e96148d8-4aea-49f4-9664-1298b3d56af4-kube-api-access-sm458\") pod \"e96148d8-4aea-49f4-9664-1298b3d56af4\" (UID: \"e96148d8-4aea-49f4-9664-1298b3d56af4\") "
Sep 18 14:01:43.196373 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:43.196219    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e96148d8-4aea-49f4-9664-1298b3d56af4-maas-api-tls\") pod \"e96148d8-4aea-49f4-9664-1298b3d56af4\" (UID: \"e96148d8-4aea-49f4-9664-1298b3d56af4\") "
Sep 18 14:01:43.196373 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:43.196255    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e96148d8-4aea-49f4-9664-1298b3d56af4-metrics-certs\") pod \"e96148d8-4aea-49f4-9664-1298b3d56af4\" (UID: \"e96148d8-4aea-49f4-9664-1298b3d56af4\") "
Sep 18 14:01:43.196553 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:43.196524    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e96148d8-4aea-49f4-9664-1298b3d56af4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e96148d8-4aea-49f4-9664-1298b3d56af4" (UID: "e96148d8-4aea-49f4-9664-1298b3d56af4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:01:43.198615 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:43.198578    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e96148d8-4aea-49f4-9664-1298b3d56af4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e96148d8-4aea-49f4-9664-1298b3d56af4" (UID: "e96148d8-4aea-49f4-9664-1298b3d56af4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:01:43.198713 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:43.198638    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e96148d8-4aea-49f4-9664-1298b3d56af4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e96148d8-4aea-49f4-9664-1298b3d56af4" (UID: "e96148d8-4aea-49f4-9664-1298b3d56af4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:01:43.198806 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:43.198784    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e96148d8-4aea-49f4-9664-1298b3d56af4-kube-api-access-sm458" (OuterVolumeSpecName: "kube-api-access-sm458") pod "e96148d8-4aea-49f4-9664-1298b3d56af4" (UID: "e96148d8-4aea-49f4-9664-1298b3d56af4"). InnerVolumeSpecName "kube-api-access-sm458". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:01:43.211537 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:43.211509    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:43.297072 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:43.297031    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e96148d8-4aea-49f4-9664-1298b3d56af4-openshift-service-ca\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:01:43.297072 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:43.297063    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sm458\" (UniqueName: \"kubernetes.io/projected/e96148d8-4aea-49f4-9664-1298b3d56af4-kube-api-access-sm458\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:01:43.297072 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:43.297073    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e96148d8-4aea-49f4-9664-1298b3d56af4-maas-api-tls\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:01:43.297072 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:43.297081    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e96148d8-4aea-49f4-9664-1298b3d56af4-metrics-certs\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:01:43.301486 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:43.301462    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:43.575613 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:43.575523    2577 generic.go:356] "Generic (PLEG): container finished" podID="e96148d8-4aea-49f4-9664-1298b3d56af4" containerID="c02db53c6778240d874b031ba97d777d3ed38b01a1af6402db666928d4463ce0" exitCode=0
Sep 18 14:01:43.575613 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:43.575567    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0aac26-66c8b4f78c-5p5ns" event={"ID":"e96148d8-4aea-49f4-9664-1298b3d56af4","Type":"ContainerDied","Data":"c02db53c6778240d874b031ba97d777d3ed38b01a1af6402db666928d4463ce0"}
Sep 18 14:01:43.575613 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:43.575575    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0aac26-66c8b4f78c-5p5ns"
Sep 18 14:01:43.575613 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:43.575608    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0aac26-66c8b4f78c-5p5ns" event={"ID":"e96148d8-4aea-49f4-9664-1298b3d56af4","Type":"ContainerDied","Data":"46964adc273ae519b1e21c060ccab66f1abb9768d585653212bda9df36ba9a1f"}
Sep 18 14:01:43.575991 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:43.575630    2577 scope.go:117] "RemoveContainer" containerID="c02db53c6778240d874b031ba97d777d3ed38b01a1af6402db666928d4463ce0"
Sep 18 14:01:43.585994 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:43.585973    2577 scope.go:117] "RemoveContainer" containerID="c02db53c6778240d874b031ba97d777d3ed38b01a1af6402db666928d4463ce0"
Sep 18 14:01:43.586265 ip-10-0-141-222 kubenswrapper[2577]: E0918 14:01:43.586246    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c02db53c6778240d874b031ba97d777d3ed38b01a1af6402db666928d4463ce0\": container with ID starting with c02db53c6778240d874b031ba97d777d3ed38b01a1af6402db666928d4463ce0 not found: ID does not exist" containerID="c02db53c6778240d874b031ba97d777d3ed38b01a1af6402db666928d4463ce0"
Sep 18 14:01:43.586341 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:43.586283    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c02db53c6778240d874b031ba97d777d3ed38b01a1af6402db666928d4463ce0"} err="failed to get container status \"c02db53c6778240d874b031ba97d777d3ed38b01a1af6402db666928d4463ce0\": rpc error: code = NotFound desc = could not find container \"c02db53c6778240d874b031ba97d777d3ed38b01a1af6402db666928d4463ce0\": container with ID starting with c02db53c6778240d874b031ba97d777d3ed38b01a1af6402db666928d4463ce0 not found: ID does not exist"
Sep 18 14:01:43.591020 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:43.591000    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:43.599551 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:43.599530    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-0aac26-66c8b4f78c-5p5ns"]
Sep 18 14:01:43.603456 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:43.603433    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-0aac26-66c8b4f78c-5p5ns"]
Sep 18 14:01:44.019885 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:44.019844    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:44.746484 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:44.746454    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:44.845484 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:44.845436    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:44.961657 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:44.961629    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:45.108328 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:45.108295    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e96148d8-4aea-49f4-9664-1298b3d56af4" path="/var/lib/kubelet/pods/e96148d8-4aea-49f4-9664-1298b3d56af4/volumes"
Sep 18 14:01:45.232158 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:45.232122    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:45.319633 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:45.319602    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:45.367666 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:45.367582    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-e46fbf-7d645454dc-nmptd"]
Sep 18 14:01:45.368063 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:45.368044    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e96148d8-4aea-49f4-9664-1298b3d56af4" containerName="maas-api"
Sep 18 14:01:45.368163 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:45.368064    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e96148d8-4aea-49f4-9664-1298b3d56af4" containerName="maas-api"
Sep 18 14:01:45.368223 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:45.368197    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e96148d8-4aea-49f4-9664-1298b3d56af4" containerName="maas-api"
Sep 18 14:01:45.372471 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:45.372450    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e46fbf-7d645454dc-nmptd"
Sep 18 14:01:45.374491 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:45.374473    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-e46fbf-dockercfg-66pfs\""
Sep 18 14:01:45.374870 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:45.374853    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-e46fbf\""
Sep 18 14:01:45.381879 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:45.381856    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-e46fbf-7d645454dc-nmptd"]
Sep 18 14:01:45.515436 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:45.515402    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cb59069-38f5-4972-9df5-99f51514d2e4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-e46fbf-7d645454dc-nmptd\" (UID: \"7cb59069-38f5-4972-9df5-99f51514d2e4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e46fbf-7d645454dc-nmptd"
Sep 18 14:01:45.515618 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:45.515539    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvgtc\" (UniqueName: \"kubernetes.io/projected/7cb59069-38f5-4972-9df5-99f51514d2e4-kube-api-access-fvgtc\") pod \"payload-processing-e2e-shared-a-w2-e46fbf-7d645454dc-nmptd\" (UID: \"7cb59069-38f5-4972-9df5-99f51514d2e4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e46fbf-7d645454dc-nmptd"
Sep 18 14:01:45.616183 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:45.616137    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cb59069-38f5-4972-9df5-99f51514d2e4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-e46fbf-7d645454dc-nmptd\" (UID: \"7cb59069-38f5-4972-9df5-99f51514d2e4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e46fbf-7d645454dc-nmptd"
Sep 18 14:01:45.616392 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:45.616256    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvgtc\" (UniqueName: \"kubernetes.io/projected/7cb59069-38f5-4972-9df5-99f51514d2e4-kube-api-access-fvgtc\") pod \"payload-processing-e2e-shared-a-w2-e46fbf-7d645454dc-nmptd\" (UID: \"7cb59069-38f5-4972-9df5-99f51514d2e4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e46fbf-7d645454dc-nmptd"
Sep 18 14:01:45.616952 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:45.616930    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cb59069-38f5-4972-9df5-99f51514d2e4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-e46fbf-7d645454dc-nmptd\" (UID: \"7cb59069-38f5-4972-9df5-99f51514d2e4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e46fbf-7d645454dc-nmptd"
Sep 18 14:01:45.624246 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:45.624191    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvgtc\" (UniqueName: \"kubernetes.io/projected/7cb59069-38f5-4972-9df5-99f51514d2e4-kube-api-access-fvgtc\") pod \"payload-processing-e2e-shared-a-w2-e46fbf-7d645454dc-nmptd\" (UID: \"7cb59069-38f5-4972-9df5-99f51514d2e4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e46fbf-7d645454dc-nmptd"
Sep 18 14:01:45.635517 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:45.635493    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:45.683983 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:45.683954    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e46fbf-7d645454dc-nmptd"
Sep 18 14:01:45.960341 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:45.960315    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-e46fbf-7d645454dc-nmptd"]
Sep 18 14:01:45.962041 ip-10-0-141-222 kubenswrapper[2577]: W0918 14:01:45.962014    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cb59069_38f5_4972_9df5_99f51514d2e4.slice/crio-9a27e3d0ca7616921ff3f759b88f2e7c31c1834eda9b9c87c8297917bae7d936 WatchSource:0}: Error finding container 9a27e3d0ca7616921ff3f759b88f2e7c31c1834eda9b9c87c8297917bae7d936: Status 404 returned error can't find the container with id 9a27e3d0ca7616921ff3f759b88f2e7c31c1834eda9b9c87c8297917bae7d936
Sep 18 14:01:46.328056 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:46.327972    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:46.595430 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:46.595389    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e46fbf-7d645454dc-nmptd" event={"ID":"7cb59069-38f5-4972-9df5-99f51514d2e4","Type":"ContainerStarted","Data":"6d6f719b5eea9d083bf0c80081e18ac81b82c9ae7b4c002d6986c169f070727b"}
Sep 18 14:01:46.595627 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:46.595491    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e46fbf-7d645454dc-nmptd"
Sep 18 14:01:46.595627 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:46.595521    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e46fbf-7d645454dc-nmptd" event={"ID":"7cb59069-38f5-4972-9df5-99f51514d2e4","Type":"ContainerStarted","Data":"9a27e3d0ca7616921ff3f759b88f2e7c31c1834eda9b9c87c8297917bae7d936"}
Sep 18 14:01:46.615262 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:46.615203    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e46fbf-7d645454dc-nmptd" podStartSLOduration=1.6151834219999999 podStartE2EDuration="1.615183422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:01:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:01:46.612952173 +0000 UTC m=+1055.665611499" watchObservedRunningTime="2026-09-18 14:01:46.615183422 +0000 UTC m=+1055.667842749"
Sep 18 14:01:47.381638 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:47.381600    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:48.071562 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:48.071529    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:48.658846 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:48.658784    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:49.049555 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:49.049463    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:50.807679 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:50.807632    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-0aac26/explicit-model-0aac26-kserve-659b975947-p4n4t" podUID="987a4375-4d81-4073-8228-c7ea5249551e" containerName="main" containerID="cri-o://78401eb5cfcb7040ec1ce98b8812bafd8f1d6fad9bb1a4fa81699a15c7085083" gracePeriod=45
Sep 18 14:01:51.058664 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:51.058604    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-0aac26/explicit-model-0aac26-kserve-659b975947-p4n4t"
Sep 18 14:01:51.167371 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:51.167339    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pj5k7\" (UniqueName: \"kubernetes.io/projected/987a4375-4d81-4073-8228-c7ea5249551e-kube-api-access-pj5k7\") pod \"987a4375-4d81-4073-8228-c7ea5249551e\" (UID: \"987a4375-4d81-4073-8228-c7ea5249551e\") "
Sep 18 14:01:51.167542 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:51.167379    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/987a4375-4d81-4073-8228-c7ea5249551e-model-cache\") pod \"987a4375-4d81-4073-8228-c7ea5249551e\" (UID: \"987a4375-4d81-4073-8228-c7ea5249551e\") "
Sep 18 14:01:51.167542 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:51.167512    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/987a4375-4d81-4073-8228-c7ea5249551e-home\") pod \"987a4375-4d81-4073-8228-c7ea5249551e\" (UID: \"987a4375-4d81-4073-8228-c7ea5249551e\") "
Sep 18 14:01:51.167631 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:51.167543    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/987a4375-4d81-4073-8228-c7ea5249551e-tls-certs\") pod \"987a4375-4d81-4073-8228-c7ea5249551e\" (UID: \"987a4375-4d81-4073-8228-c7ea5249551e\") "
Sep 18 14:01:51.167631 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:51.167574    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/987a4375-4d81-4073-8228-c7ea5249551e-dshm\") pod \"987a4375-4d81-4073-8228-c7ea5249551e\" (UID: \"987a4375-4d81-4073-8228-c7ea5249551e\") "
Sep 18 14:01:51.167631 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:51.167604    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/987a4375-4d81-4073-8228-c7ea5249551e-tmp-dir\") pod \"987a4375-4d81-4073-8228-c7ea5249551e\" (UID: \"987a4375-4d81-4073-8228-c7ea5249551e\") "
Sep 18 14:01:51.167776 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:51.167658    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/987a4375-4d81-4073-8228-c7ea5249551e-model-cache" (OuterVolumeSpecName: "model-cache") pod "987a4375-4d81-4073-8228-c7ea5249551e" (UID: "987a4375-4d81-4073-8228-c7ea5249551e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:01:51.167776 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:51.167759    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/987a4375-4d81-4073-8228-c7ea5249551e-home" (OuterVolumeSpecName: "home") pod "987a4375-4d81-4073-8228-c7ea5249551e" (UID: "987a4375-4d81-4073-8228-c7ea5249551e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:01:51.167966 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:51.167941    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/987a4375-4d81-4073-8228-c7ea5249551e-model-cache\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:01:51.168042 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:51.167971    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/987a4375-4d81-4073-8228-c7ea5249551e-home\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:01:51.168042 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:51.167967    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/987a4375-4d81-4073-8228-c7ea5249551e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "987a4375-4d81-4073-8228-c7ea5249551e" (UID: "987a4375-4d81-4073-8228-c7ea5249551e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:01:51.169731 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:51.169700    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/987a4375-4d81-4073-8228-c7ea5249551e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "987a4375-4d81-4073-8228-c7ea5249551e" (UID: "987a4375-4d81-4073-8228-c7ea5249551e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:01:51.169862 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:51.169736    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/987a4375-4d81-4073-8228-c7ea5249551e-dshm" (OuterVolumeSpecName: "dshm") pod "987a4375-4d81-4073-8228-c7ea5249551e" (UID: "987a4375-4d81-4073-8228-c7ea5249551e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:01:51.169862 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:51.169783    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/987a4375-4d81-4073-8228-c7ea5249551e-kube-api-access-pj5k7" (OuterVolumeSpecName: "kube-api-access-pj5k7") pod "987a4375-4d81-4073-8228-c7ea5249551e" (UID: "987a4375-4d81-4073-8228-c7ea5249551e"). InnerVolumeSpecName "kube-api-access-pj5k7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:01:51.268610 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:51.268578    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/987a4375-4d81-4073-8228-c7ea5249551e-tls-certs\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:01:51.268610 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:51.268608    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/987a4375-4d81-4073-8228-c7ea5249551e-dshm\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:01:51.268610 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:51.268619    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/987a4375-4d81-4073-8228-c7ea5249551e-tmp-dir\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:01:51.268879 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:51.268629    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pj5k7\" (UniqueName: \"kubernetes.io/projected/987a4375-4d81-4073-8228-c7ea5249551e-kube-api-access-pj5k7\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:01:51.627770 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:51.627731    2577 generic.go:356] "Generic (PLEG): container finished" podID="987a4375-4d81-4073-8228-c7ea5249551e" containerID="78401eb5cfcb7040ec1ce98b8812bafd8f1d6fad9bb1a4fa81699a15c7085083" exitCode=0
Sep 18 14:01:51.628000 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:51.627782    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-0aac26/explicit-model-0aac26-kserve-659b975947-p4n4t"
Sep 18 14:01:51.628000 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:51.627811    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-0aac26/explicit-model-0aac26-kserve-659b975947-p4n4t" event={"ID":"987a4375-4d81-4073-8228-c7ea5249551e","Type":"ContainerDied","Data":"78401eb5cfcb7040ec1ce98b8812bafd8f1d6fad9bb1a4fa81699a15c7085083"}
Sep 18 14:01:51.628000 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:51.627873    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-0aac26/explicit-model-0aac26-kserve-659b975947-p4n4t" event={"ID":"987a4375-4d81-4073-8228-c7ea5249551e","Type":"ContainerDied","Data":"7ca5bf180da84a9655193cf1726b8b462f997aa20b817d61b8b5438bf429ccc0"}
Sep 18 14:01:51.628000 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:51.627894    2577 scope.go:117] "RemoveContainer" containerID="78401eb5cfcb7040ec1ce98b8812bafd8f1d6fad9bb1a4fa81699a15c7085083"
Sep 18 14:01:51.643932 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:51.643914    2577 scope.go:117] "RemoveContainer" containerID="78401eb5cfcb7040ec1ce98b8812bafd8f1d6fad9bb1a4fa81699a15c7085083"
Sep 18 14:01:51.644234 ip-10-0-141-222 kubenswrapper[2577]: E0918 14:01:51.644208    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78401eb5cfcb7040ec1ce98b8812bafd8f1d6fad9bb1a4fa81699a15c7085083\": container with ID starting with 78401eb5cfcb7040ec1ce98b8812bafd8f1d6fad9bb1a4fa81699a15c7085083 not found: ID does not exist" containerID="78401eb5cfcb7040ec1ce98b8812bafd8f1d6fad9bb1a4fa81699a15c7085083"
Sep 18 14:01:51.644308 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:51.644246    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78401eb5cfcb7040ec1ce98b8812bafd8f1d6fad9bb1a4fa81699a15c7085083"} err="failed to get container status \"78401eb5cfcb7040ec1ce98b8812bafd8f1d6fad9bb1a4fa81699a15c7085083\": rpc error: code = NotFound desc = could not find container \"78401eb5cfcb7040ec1ce98b8812bafd8f1d6fad9bb1a4fa81699a15c7085083\": container with ID starting with 78401eb5cfcb7040ec1ce98b8812bafd8f1d6fad9bb1a4fa81699a15c7085083 not found: ID does not exist"
Sep 18 14:01:51.656990 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:51.656967    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-0aac26/explicit-model-0aac26-kserve-659b975947-p4n4t"]
Sep 18 14:01:51.659298 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:51.659277    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-0aac26/explicit-model-0aac26-kserve-659b975947-p4n4t"]
Sep 18 14:01:53.108874 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:53.108840    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="987a4375-4d81-4073-8228-c7ea5249551e" path="/var/lib/kubelet/pods/987a4375-4d81-4073-8228-c7ea5249551e/volumes"
Sep 18 14:01:54.655078 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:54.655041    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:54.883569 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:54.883533    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd"]
Sep 18 14:01:54.883776 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:54.883743    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd" podUID="f8d985f0-b51e-4a56-9840-179887b8f54d" containerName="istio-proxy" containerID="cri-o://ceaafafddd169638b3ad757ba2a569f9c2bfdb5686bf4096a27e8de5c9ca896c" gracePeriod=30
Sep 18 14:01:54.926906 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:54.926785    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:55.358031 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:55.357991    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:55.405979 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:55.405941    2577 patch_prober.go:28] interesting pod/e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 18 14:01:55.406155 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:55.405999    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd" podUID="f8d985f0-b51e-4a56-9840-179887b8f54d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 14:01:55.720667 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:55.720516    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:55.825953 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:55.825921    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:55.921470 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:55.921440    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:56.758242 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:56.758206    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:56.910505 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:56.910471    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:57.110343 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:57.110302    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:57.604977 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:01:57.604950    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e46fbf-7d645454dc-nmptd"
Sep 18 14:02:00.141449 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:00.141415    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd"
Sep 18 14:02:00.252661 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:00.252551    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8d985f0-b51e-4a56-9840-179887b8f54d-workload-certs\") pod \"f8d985f0-b51e-4a56-9840-179887b8f54d\" (UID: \"f8d985f0-b51e-4a56-9840-179887b8f54d\") "
Sep 18 14:02:00.252661 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:00.252638    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8d985f0-b51e-4a56-9840-179887b8f54d-credential-socket\") pod \"f8d985f0-b51e-4a56-9840-179887b8f54d\" (UID: \"f8d985f0-b51e-4a56-9840-179887b8f54d\") "
Sep 18 14:02:00.252945 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:00.252671    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8d985f0-b51e-4a56-9840-179887b8f54d-istiod-ca-cert\") pod \"f8d985f0-b51e-4a56-9840-179887b8f54d\" (UID: \"f8d985f0-b51e-4a56-9840-179887b8f54d\") "
Sep 18 14:02:00.252945 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:00.252702    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8d985f0-b51e-4a56-9840-179887b8f54d-istio-token\") pod \"f8d985f0-b51e-4a56-9840-179887b8f54d\" (UID: \"f8d985f0-b51e-4a56-9840-179887b8f54d\") "
Sep 18 14:02:00.252945 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:00.252746    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8d985f0-b51e-4a56-9840-179887b8f54d-istio-data\") pod \"f8d985f0-b51e-4a56-9840-179887b8f54d\" (UID: \"f8d985f0-b51e-4a56-9840-179887b8f54d\") "
Sep 18 14:02:00.252945 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:00.252779    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7vbww\" (UniqueName: \"kubernetes.io/projected/f8d985f0-b51e-4a56-9840-179887b8f54d-kube-api-access-7vbww\") pod \"f8d985f0-b51e-4a56-9840-179887b8f54d\" (UID: \"f8d985f0-b51e-4a56-9840-179887b8f54d\") "
Sep 18 14:02:00.252945 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:00.252846    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8d985f0-b51e-4a56-9840-179887b8f54d-istio-podinfo\") pod \"f8d985f0-b51e-4a56-9840-179887b8f54d\" (UID: \"f8d985f0-b51e-4a56-9840-179887b8f54d\") "
Sep 18 14:02:00.252945 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:00.252872    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8d985f0-b51e-4a56-9840-179887b8f54d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f8d985f0-b51e-4a56-9840-179887b8f54d" (UID: "f8d985f0-b51e-4a56-9840-179887b8f54d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:02:00.252945 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:00.252922    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8d985f0-b51e-4a56-9840-179887b8f54d-istio-envoy\") pod \"f8d985f0-b51e-4a56-9840-179887b8f54d\" (UID: \"f8d985f0-b51e-4a56-9840-179887b8f54d\") "
Sep 18 14:02:00.253303 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:00.252962    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8d985f0-b51e-4a56-9840-179887b8f54d-workload-socket\") pod \"f8d985f0-b51e-4a56-9840-179887b8f54d\" (UID: \"f8d985f0-b51e-4a56-9840-179887b8f54d\") "
Sep 18 14:02:00.253303 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:00.252991    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8d985f0-b51e-4a56-9840-179887b8f54d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f8d985f0-b51e-4a56-9840-179887b8f54d" (UID: "f8d985f0-b51e-4a56-9840-179887b8f54d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:02:00.253303 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:00.253117    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8d985f0-b51e-4a56-9840-179887b8f54d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f8d985f0-b51e-4a56-9840-179887b8f54d" (UID: "f8d985f0-b51e-4a56-9840-179887b8f54d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:02:00.253303 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:00.253298    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8d985f0-b51e-4a56-9840-179887b8f54d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f8d985f0-b51e-4a56-9840-179887b8f54d" (UID: "f8d985f0-b51e-4a56-9840-179887b8f54d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:02:00.253511 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:00.253317    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8d985f0-b51e-4a56-9840-179887b8f54d-istio-data" (OuterVolumeSpecName: "istio-data") pod "f8d985f0-b51e-4a56-9840-179887b8f54d" (UID: "f8d985f0-b51e-4a56-9840-179887b8f54d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:02:00.253511 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:00.253399    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8d985f0-b51e-4a56-9840-179887b8f54d-workload-certs\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:02:00.253511 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:00.253420    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8d985f0-b51e-4a56-9840-179887b8f54d-credential-socket\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:02:00.253511 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:00.253439    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8d985f0-b51e-4a56-9840-179887b8f54d-istiod-ca-cert\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:02:00.255082 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:00.255052    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8d985f0-b51e-4a56-9840-179887b8f54d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f8d985f0-b51e-4a56-9840-179887b8f54d" (UID: "f8d985f0-b51e-4a56-9840-179887b8f54d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 14:02:00.255217 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:00.255089    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8d985f0-b51e-4a56-9840-179887b8f54d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f8d985f0-b51e-4a56-9840-179887b8f54d" (UID: "f8d985f0-b51e-4a56-9840-179887b8f54d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:02:00.255217 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:00.255145    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8d985f0-b51e-4a56-9840-179887b8f54d-istio-token" (OuterVolumeSpecName: "istio-token") pod "f8d985f0-b51e-4a56-9840-179887b8f54d" (UID: "f8d985f0-b51e-4a56-9840-179887b8f54d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:02:00.255217 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:00.255172    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8d985f0-b51e-4a56-9840-179887b8f54d-kube-api-access-7vbww" (OuterVolumeSpecName: "kube-api-access-7vbww") pod "f8d985f0-b51e-4a56-9840-179887b8f54d" (UID: "f8d985f0-b51e-4a56-9840-179887b8f54d"). InnerVolumeSpecName "kube-api-access-7vbww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:02:00.354477 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:00.354441    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8d985f0-b51e-4a56-9840-179887b8f54d-istio-token\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:02:00.354477 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:00.354472    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8d985f0-b51e-4a56-9840-179887b8f54d-istio-data\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:02:00.354477 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:00.354481    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vbww\" (UniqueName: \"kubernetes.io/projected/f8d985f0-b51e-4a56-9840-179887b8f54d-kube-api-access-7vbww\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:02:00.354477 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:00.354490    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8d985f0-b51e-4a56-9840-179887b8f54d-istio-podinfo\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:02:00.354764 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:00.354498    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8d985f0-b51e-4a56-9840-179887b8f54d-istio-envoy\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:02:00.354764 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:00.354507    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8d985f0-b51e-4a56-9840-179887b8f54d-workload-socket\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:02:00.683705 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:00.683662    2577 generic.go:356] "Generic (PLEG): container finished" podID="f8d985f0-b51e-4a56-9840-179887b8f54d" containerID="ceaafafddd169638b3ad757ba2a569f9c2bfdb5686bf4096a27e8de5c9ca896c" exitCode=0
Sep 18 14:02:00.683961 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:00.683720    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd"
Sep 18 14:02:00.683961 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:00.683748    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd" event={"ID":"f8d985f0-b51e-4a56-9840-179887b8f54d","Type":"ContainerDied","Data":"ceaafafddd169638b3ad757ba2a569f9c2bfdb5686bf4096a27e8de5c9ca896c"}
Sep 18 14:02:00.683961 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:00.683802    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd" event={"ID":"f8d985f0-b51e-4a56-9840-179887b8f54d","Type":"ContainerDied","Data":"cc0f786fe9e0bc3a92e57ee85329d51612ed6e355d0ed2d776f7fe5664d3069a"}
Sep 18 14:02:00.683961 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:00.683847    2577 scope.go:117] "RemoveContainer" containerID="ceaafafddd169638b3ad757ba2a569f9c2bfdb5686bf4096a27e8de5c9ca896c"
Sep 18 14:02:00.693692 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:00.693674    2577 scope.go:117] "RemoveContainer" containerID="af42580ad4adcd4beca73bc5031659a1465d2d9ce9c2fbbb02d964d4f5b800e2"
Sep 18 14:02:00.702368 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:00.702193    2577 scope.go:117] "RemoveContainer" containerID="ceaafafddd169638b3ad757ba2a569f9c2bfdb5686bf4096a27e8de5c9ca896c"
Sep 18 14:02:00.702688 ip-10-0-141-222 kubenswrapper[2577]: E0918 14:02:00.702668    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ceaafafddd169638b3ad757ba2a569f9c2bfdb5686bf4096a27e8de5c9ca896c\": container with ID starting with ceaafafddd169638b3ad757ba2a569f9c2bfdb5686bf4096a27e8de5c9ca896c not found: ID does not exist" containerID="ceaafafddd169638b3ad757ba2a569f9c2bfdb5686bf4096a27e8de5c9ca896c"
Sep 18 14:02:00.702732 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:00.702698    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ceaafafddd169638b3ad757ba2a569f9c2bfdb5686bf4096a27e8de5c9ca896c"} err="failed to get container status \"ceaafafddd169638b3ad757ba2a569f9c2bfdb5686bf4096a27e8de5c9ca896c\": rpc error: code = NotFound desc = could not find container \"ceaafafddd169638b3ad757ba2a569f9c2bfdb5686bf4096a27e8de5c9ca896c\": container with ID starting with ceaafafddd169638b3ad757ba2a569f9c2bfdb5686bf4096a27e8de5c9ca896c not found: ID does not exist"
Sep 18 14:02:00.702732 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:00.702716    2577 scope.go:117] "RemoveContainer" containerID="af42580ad4adcd4beca73bc5031659a1465d2d9ce9c2fbbb02d964d4f5b800e2"
Sep 18 14:02:00.703004 ip-10-0-141-222 kubenswrapper[2577]: E0918 14:02:00.702987    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af42580ad4adcd4beca73bc5031659a1465d2d9ce9c2fbbb02d964d4f5b800e2\": container with ID starting with af42580ad4adcd4beca73bc5031659a1465d2d9ce9c2fbbb02d964d4f5b800e2 not found: ID does not exist" containerID="af42580ad4adcd4beca73bc5031659a1465d2d9ce9c2fbbb02d964d4f5b800e2"
Sep 18 14:02:00.703055 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:00.703010    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af42580ad4adcd4beca73bc5031659a1465d2d9ce9c2fbbb02d964d4f5b800e2"} err="failed to get container status \"af42580ad4adcd4beca73bc5031659a1465d2d9ce9c2fbbb02d964d4f5b800e2\": rpc error: code = NotFound desc = could not find container \"af42580ad4adcd4beca73bc5031659a1465d2d9ce9c2fbbb02d964d4f5b800e2\": container with ID starting with af42580ad4adcd4beca73bc5031659a1465d2d9ce9c2fbbb02d964d4f5b800e2 not found: ID does not exist"
Sep 18 14:02:00.706940 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:00.706913    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd"]
Sep 18 14:02:00.711350 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:00.711329    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-0aac26-openshift-default-587dbf4d56-f2dqd"]
Sep 18 14:02:01.107806 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:01.107773    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8d985f0-b51e-4a56-9840-179887b8f54d" path="/var/lib/kubelet/pods/f8d985f0-b51e-4a56-9840-179887b8f54d/volumes"
Sep 18 14:02:04.197332 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:04.197291    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 14:02:05.701397 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:05.701361    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:05.802286 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:05.802248    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:07.146351 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:07.146313    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:07.918650 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:07.918613    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:08.753248 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:08.753213    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:09.565715 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:09.565681    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:10.489910 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:10.489878    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:11.325814 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:11.325776    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:11.508572 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:11.508538    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:11.612715 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:11.612682    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:11.906008 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:11.905931    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6"]
Sep 18 14:02:11.906317 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:11.906303    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8d985f0-b51e-4a56-9840-179887b8f54d" containerName="istio-proxy"
Sep 18 14:02:11.906367 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:11.906318    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8d985f0-b51e-4a56-9840-179887b8f54d" containerName="istio-proxy"
Sep 18 14:02:11.906367 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:11.906335    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="987a4375-4d81-4073-8228-c7ea5249551e" containerName="main"
Sep 18 14:02:11.906367 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:11.906341    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="987a4375-4d81-4073-8228-c7ea5249551e" containerName="main"
Sep 18 14:02:11.906367 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:11.906360    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8d985f0-b51e-4a56-9840-179887b8f54d" containerName="istio-proxy"
Sep 18 14:02:11.906367 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:11.906366    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8d985f0-b51e-4a56-9840-179887b8f54d" containerName="istio-proxy"
Sep 18 14:02:11.906520 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:11.906426    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8d985f0-b51e-4a56-9840-179887b8f54d" containerName="istio-proxy"
Sep 18 14:02:11.906520 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:11.906436    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="987a4375-4d81-4073-8228-c7ea5249551e" containerName="main"
Sep 18 14:02:11.911067 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:11.911044    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6"
Sep 18 14:02:11.913465 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:11.913443    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 18 14:02:11.913579 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:11.913533    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-55bd4d-openshift-default-dockercfg-9pkcv\""
Sep 18 14:02:11.925778 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:11.925752    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6"]
Sep 18 14:02:11.937785 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:11.937764    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:11.952758 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:11.952731    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b4e8091-639f-434e-8bf3-4411ef5b0b7e-istio-token\") pod \"e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6\" (UID: \"1b4e8091-639f-434e-8bf3-4411ef5b0b7e\") " pod="openshift-ingress/e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6"
Sep 18 14:02:11.952911 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:11.952768    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b4e8091-639f-434e-8bf3-4411ef5b0b7e-workload-socket\") pod \"e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6\" (UID: \"1b4e8091-639f-434e-8bf3-4411ef5b0b7e\") " pod="openshift-ingress/e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6"
Sep 18 14:02:11.952911 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:11.952787    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b4e8091-639f-434e-8bf3-4411ef5b0b7e-workload-certs\") pod \"e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6\" (UID: \"1b4e8091-639f-434e-8bf3-4411ef5b0b7e\") " pod="openshift-ingress/e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6"
Sep 18 14:02:11.952911 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:11.952852    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgb7k\" (UniqueName: \"kubernetes.io/projected/1b4e8091-639f-434e-8bf3-4411ef5b0b7e-kube-api-access-bgb7k\") pod \"e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6\" (UID: \"1b4e8091-639f-434e-8bf3-4411ef5b0b7e\") " pod="openshift-ingress/e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6"
Sep 18 14:02:11.952911 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:11.952887    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b4e8091-639f-434e-8bf3-4411ef5b0b7e-istiod-ca-cert\") pod \"e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6\" (UID: \"1b4e8091-639f-434e-8bf3-4411ef5b0b7e\") " pod="openshift-ingress/e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6"
Sep 18 14:02:11.953114 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:11.952937    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b4e8091-639f-434e-8bf3-4411ef5b0b7e-istio-envoy\") pod \"e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6\" (UID: \"1b4e8091-639f-434e-8bf3-4411ef5b0b7e\") " pod="openshift-ingress/e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6"
Sep 18 14:02:11.953114 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:11.953028    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b4e8091-639f-434e-8bf3-4411ef5b0b7e-credential-socket\") pod \"e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6\" (UID: \"1b4e8091-639f-434e-8bf3-4411ef5b0b7e\") " pod="openshift-ingress/e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6"
Sep 18 14:02:11.953114 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:11.953071    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b4e8091-639f-434e-8bf3-4411ef5b0b7e-istio-data\") pod \"e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6\" (UID: \"1b4e8091-639f-434e-8bf3-4411ef5b0b7e\") " pod="openshift-ingress/e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6"
Sep 18 14:02:11.953114 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:11.953091    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b4e8091-639f-434e-8bf3-4411ef5b0b7e-istio-podinfo\") pod \"e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6\" (UID: \"1b4e8091-639f-434e-8bf3-4411ef5b0b7e\") " pod="openshift-ingress/e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6"
Sep 18 14:02:12.042370 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:12.042333    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:12.053545 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:12.053518    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b4e8091-639f-434e-8bf3-4411ef5b0b7e-credential-socket\") pod \"e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6\" (UID: \"1b4e8091-639f-434e-8bf3-4411ef5b0b7e\") " pod="openshift-ingress/e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6"
Sep 18 14:02:12.053700 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:12.053555    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b4e8091-639f-434e-8bf3-4411ef5b0b7e-istio-data\") pod \"e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6\" (UID: \"1b4e8091-639f-434e-8bf3-4411ef5b0b7e\") " pod="openshift-ingress/e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6"
Sep 18 14:02:12.053700 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:12.053571    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b4e8091-639f-434e-8bf3-4411ef5b0b7e-istio-podinfo\") pod \"e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6\" (UID: \"1b4e8091-639f-434e-8bf3-4411ef5b0b7e\") " pod="openshift-ingress/e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6"
Sep 18 14:02:12.053700 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:12.053599    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b4e8091-639f-434e-8bf3-4411ef5b0b7e-istio-token\") pod \"e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6\" (UID: \"1b4e8091-639f-434e-8bf3-4411ef5b0b7e\") " pod="openshift-ingress/e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6"
Sep 18 14:02:12.053700 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:12.053633    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b4e8091-639f-434e-8bf3-4411ef5b0b7e-workload-socket\") pod \"e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6\" (UID: \"1b4e8091-639f-434e-8bf3-4411ef5b0b7e\") " pod="openshift-ingress/e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6"
Sep 18 14:02:12.053700 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:12.053657    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b4e8091-639f-434e-8bf3-4411ef5b0b7e-workload-certs\") pod \"e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6\" (UID: \"1b4e8091-639f-434e-8bf3-4411ef5b0b7e\") " pod="openshift-ingress/e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6"
Sep 18 14:02:12.053700 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:12.053685    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgb7k\" (UniqueName: \"kubernetes.io/projected/1b4e8091-639f-434e-8bf3-4411ef5b0b7e-kube-api-access-bgb7k\") pod \"e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6\" (UID: \"1b4e8091-639f-434e-8bf3-4411ef5b0b7e\") " pod="openshift-ingress/e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6"
Sep 18 14:02:12.054032 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:12.053711    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b4e8091-639f-434e-8bf3-4411ef5b0b7e-istiod-ca-cert\") pod \"e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6\" (UID: \"1b4e8091-639f-434e-8bf3-4411ef5b0b7e\") " pod="openshift-ingress/e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6"
Sep 18 14:02:12.054032 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:12.053741    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b4e8091-639f-434e-8bf3-4411ef5b0b7e-istio-envoy\") pod \"e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6\" (UID: \"1b4e8091-639f-434e-8bf3-4411ef5b0b7e\") " pod="openshift-ingress/e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6"
Sep 18 14:02:12.054144 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:12.054048    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b4e8091-639f-434e-8bf3-4411ef5b0b7e-istio-data\") pod \"e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6\" (UID: \"1b4e8091-639f-434e-8bf3-4411ef5b0b7e\") " pod="openshift-ingress/e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6"
Sep 18 14:02:12.054194 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:12.054140    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b4e8091-639f-434e-8bf3-4411ef5b0b7e-credential-socket\") pod \"e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6\" (UID: \"1b4e8091-639f-434e-8bf3-4411ef5b0b7e\") " pod="openshift-ingress/e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6"
Sep 18 14:02:12.054194 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:12.054178    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b4e8091-639f-434e-8bf3-4411ef5b0b7e-workload-socket\") pod \"e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6\" (UID: \"1b4e8091-639f-434e-8bf3-4411ef5b0b7e\") " pod="openshift-ingress/e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6"
Sep 18 14:02:12.054324 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:12.054303    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b4e8091-639f-434e-8bf3-4411ef5b0b7e-workload-certs\") pod \"e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6\" (UID: \"1b4e8091-639f-434e-8bf3-4411ef5b0b7e\") " pod="openshift-ingress/e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6"
Sep 18 14:02:12.054540 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:12.054521    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b4e8091-639f-434e-8bf3-4411ef5b0b7e-istiod-ca-cert\") pod \"e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6\" (UID: \"1b4e8091-639f-434e-8bf3-4411ef5b0b7e\") " pod="openshift-ingress/e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6"
Sep 18 14:02:12.056089 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:12.056069    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b4e8091-639f-434e-8bf3-4411ef5b0b7e-istio-envoy\") pod \"e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6\" (UID: \"1b4e8091-639f-434e-8bf3-4411ef5b0b7e\") " pod="openshift-ingress/e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6"
Sep 18 14:02:12.056191 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:12.056110    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b4e8091-639f-434e-8bf3-4411ef5b0b7e-istio-podinfo\") pod \"e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6\" (UID: \"1b4e8091-639f-434e-8bf3-4411ef5b0b7e\") " pod="openshift-ingress/e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6"
Sep 18 14:02:12.075871 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:12.075839    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b4e8091-639f-434e-8bf3-4411ef5b0b7e-istio-token\") pod \"e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6\" (UID: \"1b4e8091-639f-434e-8bf3-4411ef5b0b7e\") " pod="openshift-ingress/e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6"
Sep 18 14:02:12.076028 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:12.076009    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgb7k\" (UniqueName: \"kubernetes.io/projected/1b4e8091-639f-434e-8bf3-4411ef5b0b7e-kube-api-access-bgb7k\") pod \"e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6\" (UID: \"1b4e8091-639f-434e-8bf3-4411ef5b0b7e\") " pod="openshift-ingress/e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6"
Sep 18 14:02:12.221031 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:12.220942    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6"
Sep 18 14:02:12.506747 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:12.504492    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6"]
Sep 18 14:02:12.506747 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:12.506499    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:02:12.506747 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:12.506627    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:02:12.506747 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:12.506709    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:02:12.757592 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:12.757500    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6" event={"ID":"1b4e8091-639f-434e-8bf3-4411ef5b0b7e","Type":"ContainerStarted","Data":"d2238084965e842fce7e00c218dd42f2e36814d052f5374cf387c7890c11474e"}
Sep 18 14:02:12.757592 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:12.757564    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6" event={"ID":"1b4e8091-639f-434e-8bf3-4411ef5b0b7e","Type":"ContainerStarted","Data":"3c077352c8655533f7a4ba0fc8f2c83755dd8c02c66a33e46eb7b84408b6b062"}
Sep 18 14:02:12.780652 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:12.780594    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6" podStartSLOduration=1.780577887 podStartE2EDuration="1.780577887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:02:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:02:12.778382329 +0000 UTC m=+1081.831041658" watchObservedRunningTime="2026-09-18 14:02:12.780577887 +0000 UTC m=+1081.833237284"
Sep 18 14:02:13.221973 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:13.221946    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6"
Sep 18 14:02:13.228081 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:13.228057    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6"
Sep 18 14:02:13.498859 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:13.498762    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:13.600004 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:13.599972    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:13.763512 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:13.763417    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6"
Sep 18 14:02:13.764518 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:13.764499    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6"
Sep 18 14:02:14.030348 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:14.030257    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:14.717055 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:14.717014    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:14.813677 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:14.813640    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:14.914532 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:14.914495    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:15.689322 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:15.685686    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df2eb8-7lj72"]
Sep 18 14:02:15.689322 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:15.686695    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8d985f0-b51e-4a56-9840-179887b8f54d" containerName="istio-proxy"
Sep 18 14:02:15.692341 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:15.692307    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df2eb8-7lj72"
Sep 18 14:02:15.694670 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:15.694643    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cxtb2\""
Sep 18 14:02:15.694856 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:15.694695    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 14:02:15.695609 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:15.695584    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df2eb8-7lj72"]
Sep 18 14:02:15.713932 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:15.713904    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:15.789692 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:15.789657    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d262c787-030d-4330-81b0-7611adefc9ba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-df2eb8-7lj72\" (UID: \"d262c787-030d-4330-81b0-7611adefc9ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df2eb8-7lj72"
Sep 18 14:02:15.890702 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:15.890669    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d262c787-030d-4330-81b0-7611adefc9ba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-df2eb8-7lj72\" (UID: \"d262c787-030d-4330-81b0-7611adefc9ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df2eb8-7lj72"
Sep 18 14:02:15.891436 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:15.891412    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d262c787-030d-4330-81b0-7611adefc9ba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-df2eb8-7lj72\" (UID: \"d262c787-030d-4330-81b0-7611adefc9ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df2eb8-7lj72"
Sep 18 14:02:16.004462 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:16.004379    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df2eb8-7lj72"
Sep 18 14:02:16.281066 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:16.281037    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df2eb8-7lj72"]
Sep 18 14:02:16.283645 ip-10-0-141-222 kubenswrapper[2577]: W0918 14:02:16.283615    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd262c787_030d_4330_81b0_7611adefc9ba.slice/crio-5077b768e6c80d101447484bb407ab296644519e67310b0d1cf2a9c1a7c9d87a WatchSource:0}: Error finding container 5077b768e6c80d101447484bb407ab296644519e67310b0d1cf2a9c1a7c9d87a: Status 404 returned error can't find the container with id 5077b768e6c80d101447484bb407ab296644519e67310b0d1cf2a9c1a7c9d87a
Sep 18 14:02:16.598592 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:16.598564    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:16.782360 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:16.782314    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df2eb8-7lj72" event={"ID":"d262c787-030d-4330-81b0-7611adefc9ba","Type":"ContainerStarted","Data":"8b3581ba1551b1e9ec9f89df626d2eb839eefd84e404993f1ceede0584d4ba4b"}
Sep 18 14:02:16.782524 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:16.782371    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df2eb8-7lj72" event={"ID":"d262c787-030d-4330-81b0-7611adefc9ba","Type":"ContainerStarted","Data":"5077b768e6c80d101447484bb407ab296644519e67310b0d1cf2a9c1a7c9d87a"}
Sep 18 14:02:16.801108 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:16.801056    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df2eb8-7lj72" podStartSLOduration=1.80104033 podStartE2EDuration="1.80104033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:02:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:02:16.798564796 +0000 UTC m=+1085.851224139" watchObservedRunningTime="2026-09-18 14:02:16.80104033 +0000 UTC m=+1085.853699676"
Sep 18 14:02:17.542555 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:17.542523    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:17.789380 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:17.789348    2577 generic.go:356] "Generic (PLEG): container finished" podID="d262c787-030d-4330-81b0-7611adefc9ba" containerID="8b3581ba1551b1e9ec9f89df626d2eb839eefd84e404993f1ceede0584d4ba4b" exitCode=0
Sep 18 14:02:17.789538 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:17.789438    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df2eb8-7lj72" event={"ID":"d262c787-030d-4330-81b0-7611adefc9ba","Type":"ContainerDied","Data":"8b3581ba1551b1e9ec9f89df626d2eb839eefd84e404993f1ceede0584d4ba4b"}
Sep 18 14:02:18.352499 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:18.352464    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:18.923061 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:18.923033    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df2eb8-7lj72"
Sep 18 14:02:19.020129 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:19.020095    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d262c787-030d-4330-81b0-7611adefc9ba-maas-api-service-ca\") pod \"d262c787-030d-4330-81b0-7611adefc9ba\" (UID: \"d262c787-030d-4330-81b0-7611adefc9ba\") "
Sep 18 14:02:19.020461 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:19.020438    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d262c787-030d-4330-81b0-7611adefc9ba-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d262c787-030d-4330-81b0-7611adefc9ba" (UID: "d262c787-030d-4330-81b0-7611adefc9ba"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:02:19.088873 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:19.088845    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:19.121062 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:19.121035    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d262c787-030d-4330-81b0-7611adefc9ba-maas-api-service-ca\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:02:19.192057 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:19.192025    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:19.292397 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:19.292368    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:19.646424 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:19.646397    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:19.747855 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:19.747813    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:19.802364 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:19.802333    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df2eb8-7lj72"
Sep 18 14:02:19.802364 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:19.802341    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df2eb8-7lj72" event={"ID":"d262c787-030d-4330-81b0-7611adefc9ba","Type":"ContainerDied","Data":"5077b768e6c80d101447484bb407ab296644519e67310b0d1cf2a9c1a7c9d87a"}
Sep 18 14:02:19.802620 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:19.802383    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5077b768e6c80d101447484bb407ab296644519e67310b0d1cf2a9c1a7c9d87a"
Sep 18 14:02:20.277097 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:20.277062    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:21.151479 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:21.151443    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:21.930713 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:21.930677    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:22.700882 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:22.700806    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:22.964040 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:22.963958    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:25.823594 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:25.823556    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:25.948975 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:25.948920    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:26.434797 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:26.434761    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-df2eb8-29829000-729fw"]
Sep 18 14:02:26.436797 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:26.436745    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:26.441272 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:26.441241    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-df2eb8-29829000-729fw"]
Sep 18 14:02:26.523113 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:26.523050    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:26.624432 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:26.624391    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:26.705320 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:26.705206    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df2eb8-7lj72"]
Sep 18 14:02:26.711060 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:26.711029    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df2eb8-7lj72"]
Sep 18 14:02:26.739672 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:26.739638    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:27.042623 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:27.042521    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:27.108166 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:27.108134    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1abe0e88-3fae-4d49-82d0-ad9d4fc91be3" path="/var/lib/kubelet/pods/1abe0e88-3fae-4d49-82d0-ad9d4fc91be3/volumes"
Sep 18 14:02:27.108457 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:27.108443    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d262c787-030d-4330-81b0-7611adefc9ba" path="/var/lib/kubelet/pods/d262c787-030d-4330-81b0-7611adefc9ba/volumes"
Sep 18 14:02:27.143902 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:27.143813    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:28.115974 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:28.115944    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:28.213338 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:28.213301    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 14:02:28.971325 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:28.971283    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:29.458138 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:29.458106    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:29.818664 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:29.818578    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:30.176161 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:30.176125    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7624-f7ab6d"]
Sep 18 14:02:30.176501 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:30.176488    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d262c787-030d-4330-81b0-7611adefc9ba" containerName="revoke-keys"
Sep 18 14:02:30.176543 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:30.176502    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d262c787-030d-4330-81b0-7611adefc9ba" containerName="revoke-keys"
Sep 18 14:02:30.176579 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:30.176567    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d262c787-030d-4330-81b0-7611adefc9ba" containerName="revoke-keys"
Sep 18 14:02:30.181030 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:30.181008    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7624-f7ab6d"
Sep 18 14:02:30.183525 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:30.183499    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-knszx\""
Sep 18 14:02:30.185661 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:30.185632    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7624-f7ab6d"]
Sep 18 14:02:30.323238 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:30.323207    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v44h4\" (UniqueName: \"kubernetes.io/projected/f8f6375a-9f4b-467b-829a-2ba56ca94d2b-kube-api-access-v44h4\") pod \"test-curl-7624-f7ab6d\" (UID: \"f8f6375a-9f4b-467b-829a-2ba56ca94d2b\") " pod="opendatahub/test-curl-7624-f7ab6d"
Sep 18 14:02:30.424183 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:30.424141    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v44h4\" (UniqueName: \"kubernetes.io/projected/f8f6375a-9f4b-467b-829a-2ba56ca94d2b-kube-api-access-v44h4\") pod \"test-curl-7624-f7ab6d\" (UID: \"f8f6375a-9f4b-467b-829a-2ba56ca94d2b\") " pod="opendatahub/test-curl-7624-f7ab6d"
Sep 18 14:02:30.436300 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:30.436242    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v44h4\" (UniqueName: \"kubernetes.io/projected/f8f6375a-9f4b-467b-829a-2ba56ca94d2b-kube-api-access-v44h4\") pod \"test-curl-7624-f7ab6d\" (UID: \"f8f6375a-9f4b-467b-829a-2ba56ca94d2b\") " pod="opendatahub/test-curl-7624-f7ab6d"
Sep 18 14:02:30.491261 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:30.491230    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7624-f7ab6d"
Sep 18 14:02:30.545707 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:30.545679    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:30.768579 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:30.768552    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7624-f7ab6d"]
Sep 18 14:02:30.771195 ip-10-0-141-222 kubenswrapper[2577]: W0918 14:02:30.771154    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf8f6375a_9f4b_467b_829a_2ba56ca94d2b.slice/crio-e74bb0abbf9ad6a34d3a626172686c3ed4210307f0f88e70c617bcb0dbf6e621 WatchSource:0}: Error finding container e74bb0abbf9ad6a34d3a626172686c3ed4210307f0f88e70c617bcb0dbf6e621: Status 404 returned error can't find the container with id e74bb0abbf9ad6a34d3a626172686c3ed4210307f0f88e70c617bcb0dbf6e621
Sep 18 14:02:30.876604 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:30.876570    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7624-f7ab6d" event={"ID":"f8f6375a-9f4b-467b-829a-2ba56ca94d2b","Type":"ContainerStarted","Data":"c0812ba4d9801d40e91805808e7a66c3a6b1a35040ffa3c46058f13a3b4a86cf"}
Sep 18 14:02:30.876604 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:30.876610    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7624-f7ab6d" event={"ID":"f8f6375a-9f4b-467b-829a-2ba56ca94d2b","Type":"ContainerStarted","Data":"e74bb0abbf9ad6a34d3a626172686c3ed4210307f0f88e70c617bcb0dbf6e621"}
Sep 18 14:02:30.903481 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:30.903421    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7624-f7ab6d" podStartSLOduration=0.903405988 podStartE2EDuration="903.405988ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:02:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:02:30.900604818 +0000 UTC m=+1099.953264146" watchObservedRunningTime="2026-09-18 14:02:30.903405988 +0000 UTC m=+1099.956065315"
Sep 18 14:02:31.340351 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:31.340325    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:31.478884 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:31.478849    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7624-f7ab6d"]
Sep 18 14:02:31.483112 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:31.483074    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7624-f7ab6d"]
Sep 18 14:02:31.600417 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:31.600323    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7624-b45b7d"]
Sep 18 14:02:31.602282 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:31.602242    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8f6375a-9f4b-467b-829a-2ba56ca94d2b" containerName="test-curl-7624-f7ab6d"
Sep 18 14:02:31.602282 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:31.602280    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8f6375a-9f4b-467b-829a-2ba56ca94d2b" containerName="test-curl-7624-f7ab6d"
Sep 18 14:02:31.602468 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:31.602400    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8f6375a-9f4b-467b-829a-2ba56ca94d2b" containerName="test-curl-7624-f7ab6d"
Sep 18 14:02:31.605879 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:31.605853    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7624-b45b7d"
Sep 18 14:02:31.609135 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:31.609114    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7624-b45b7d"]
Sep 18 14:02:31.736788 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:31.736749    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgcsw\" (UniqueName: \"kubernetes.io/projected/23b67288-9407-42f6-8b28-c5d0158b8975-kube-api-access-dgcsw\") pod \"test-curl-7624-b45b7d\" (UID: \"23b67288-9407-42f6-8b28-c5d0158b8975\") " pod="opendatahub/test-curl-7624-b45b7d"
Sep 18 14:02:31.838006 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:31.837973    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgcsw\" (UniqueName: \"kubernetes.io/projected/23b67288-9407-42f6-8b28-c5d0158b8975-kube-api-access-dgcsw\") pod \"test-curl-7624-b45b7d\" (UID: \"23b67288-9407-42f6-8b28-c5d0158b8975\") " pod="opendatahub/test-curl-7624-b45b7d"
Sep 18 14:02:31.852341 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:31.852272    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgcsw\" (UniqueName: \"kubernetes.io/projected/23b67288-9407-42f6-8b28-c5d0158b8975-kube-api-access-dgcsw\") pod \"test-curl-7624-b45b7d\" (UID: \"23b67288-9407-42f6-8b28-c5d0158b8975\") " pod="opendatahub/test-curl-7624-b45b7d"
Sep 18 14:02:31.886150 ip-10-0-141-222 kubenswrapper[2577]: E0918 14:02:31.886117    2577 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7624-f7ab6d" container="test-curl-7624-f7ab6d"
Sep 18 14:02:31.916579 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:31.916535    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7624-b45b7d"
Sep 18 14:02:32.122734 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:32.122643    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:32.190196 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:32.190155    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7624-b45b7d"]
Sep 18 14:02:32.204997 ip-10-0-141-222 kubenswrapper[2577]: W0918 14:02:32.204890    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod23b67288_9407_42f6_8b28_c5d0158b8975.slice/crio-5d691d5d897912d56440d82ec0a4346c46f97a32d86adcc99d9863743fc98009 WatchSource:0}: Error finding container 5d691d5d897912d56440d82ec0a4346c46f97a32d86adcc99d9863743fc98009: Status 404 returned error can't find the container with id 5d691d5d897912d56440d82ec0a4346c46f97a32d86adcc99d9863743fc98009
Sep 18 14:02:32.896469 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:32.896412    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7624-f7ab6d" podUID="f8f6375a-9f4b-467b-829a-2ba56ca94d2b" containerName="test-curl-7624-f7ab6d" containerID="cri-o://c0812ba4d9801d40e91805808e7a66c3a6b1a35040ffa3c46058f13a3b4a86cf" gracePeriod=2
Sep 18 14:02:32.897344 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:32.897303    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7624-b45b7d" event={"ID":"23b67288-9407-42f6-8b28-c5d0158b8975","Type":"ContainerStarted","Data":"44f42aa2c1e999e39c4ea2f4d08e34642d4618b7d959eda9257e927dd9609938"}
Sep 18 14:02:32.897457 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:32.897354    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7624-b45b7d" event={"ID":"23b67288-9407-42f6-8b28-c5d0158b8975","Type":"ContainerStarted","Data":"5d691d5d897912d56440d82ec0a4346c46f97a32d86adcc99d9863743fc98009"}
Sep 18 14:02:32.900119 ip-10-0-141-222 kubenswrapper[2577]: E0918 14:02:32.900087    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7624-f7ab6d\" is forbidden: User \"system:node:ip-10-0-141-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-222.ec2.internal' and this object" podUID="f8f6375a-9f4b-467b-829a-2ba56ca94d2b" pod="opendatahub/test-curl-7624-f7ab6d"
Sep 18 14:02:32.925667 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:32.925616    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7624-b45b7d" podStartSLOduration=1.925601977 podStartE2EDuration="1.925601977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:02:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:02:32.923718538 +0000 UTC m=+1101.976377865" watchObservedRunningTime="2026-09-18 14:02:32.925601977 +0000 UTC m=+1101.978261304"
Sep 18 14:02:32.925888 ip-10-0-141-222 kubenswrapper[2577]: E0918 14:02:32.925859    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7624-f7ab6d\" is forbidden: User \"system:node:ip-10-0-141-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-222.ec2.internal' and this object" podUID="f8f6375a-9f4b-467b-829a-2ba56ca94d2b" pod="opendatahub/test-curl-7624-f7ab6d"
Sep 18 14:02:33.101053 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:33.100994    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:33.106193 ip-10-0-141-222 kubenswrapper[2577]: E0918 14:02:33.106139    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7624-f7ab6d\" is forbidden: User \"system:node:ip-10-0-141-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-222.ec2.internal' and this object" podUID="f8f6375a-9f4b-467b-829a-2ba56ca94d2b" pod="opendatahub/test-curl-7624-f7ab6d"
Sep 18 14:02:33.541421 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:33.541381    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7624-b45b7d"]
Sep 18 14:02:33.546181 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:33.546154    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7624-b45b7d"]
Sep 18 14:02:33.666939 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:33.666903    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7624-9d3adc"]
Sep 18 14:02:33.667293 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:33.667281    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23b67288-9407-42f6-8b28-c5d0158b8975" containerName="test-curl-7624-b45b7d"
Sep 18 14:02:33.667340 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:33.667295    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="23b67288-9407-42f6-8b28-c5d0158b8975" containerName="test-curl-7624-b45b7d"
Sep 18 14:02:33.667384 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:33.667368    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="23b67288-9407-42f6-8b28-c5d0158b8975" containerName="test-curl-7624-b45b7d"
Sep 18 14:02:33.670427 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:33.670410    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7624-9d3adc"
Sep 18 14:02:33.687845 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:33.679603    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7624-9d3adc"]
Sep 18 14:02:33.755717 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:33.755685    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dtbh\" (UniqueName: \"kubernetes.io/projected/369c25fd-3331-4897-9fce-bb239b8a4be2-kube-api-access-6dtbh\") pod \"test-curl-7624-9d3adc\" (UID: \"369c25fd-3331-4897-9fce-bb239b8a4be2\") " pod="opendatahub/test-curl-7624-9d3adc"
Sep 18 14:02:33.857099 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:33.857061    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dtbh\" (UniqueName: \"kubernetes.io/projected/369c25fd-3331-4897-9fce-bb239b8a4be2-kube-api-access-6dtbh\") pod \"test-curl-7624-9d3adc\" (UID: \"369c25fd-3331-4897-9fce-bb239b8a4be2\") " pod="opendatahub/test-curl-7624-9d3adc"
Sep 18 14:02:33.864918 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:33.864888    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dtbh\" (UniqueName: \"kubernetes.io/projected/369c25fd-3331-4897-9fce-bb239b8a4be2-kube-api-access-6dtbh\") pod \"test-curl-7624-9d3adc\" (UID: \"369c25fd-3331-4897-9fce-bb239b8a4be2\") " pod="opendatahub/test-curl-7624-9d3adc"
Sep 18 14:02:33.905737 ip-10-0-141-222 kubenswrapper[2577]: E0918 14:02:33.905699    2577 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7624-b45b7d" container="test-curl-7624-b45b7d"
Sep 18 14:02:33.986054 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:33.985953    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7624-9d3adc"
Sep 18 14:02:34.176809 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:34.176749    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:34.281627 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:34.281536    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:34.286660 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:34.286606    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7624-9d3adc"]
Sep 18 14:02:34.427220 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:34.427186    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:34.803596 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:34.803553    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-b63f26/test-model-b63f26-kserve-8589457995-qslk6"]
Sep 18 14:02:34.806329 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:34.806291    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:34.903279 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:34.903231    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:34.914261 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:34.914232    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7624-9d3adc" event={"ID":"369c25fd-3331-4897-9fce-bb239b8a4be2","Type":"ContainerStarted","Data":"1c8a62dd977ac373e56106984a96341f8a1b21ba23e3053a6d8e956a8d2d0312"}
Sep 18 14:02:34.914576 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:34.914288    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7624-9d3adc" event={"ID":"369c25fd-3331-4897-9fce-bb239b8a4be2","Type":"ContainerStarted","Data":"71550f8c6fc2dc82c3fa1fc78fd58c221a0df27b03839e4cd94bea1c5141d721"}
Sep 18 14:02:34.914576 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:34.914333    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7624-b45b7d" podUID="23b67288-9407-42f6-8b28-c5d0158b8975" containerName="test-curl-7624-b45b7d" containerID="cri-o://44f42aa2c1e999e39c4ea2f4d08e34642d4618b7d959eda9257e927dd9609938" gracePeriod=2
Sep 18 14:02:34.929732 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:34.929689    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7624-9d3adc" podStartSLOduration=1.929674986 podStartE2EDuration="1.929674986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:02:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:02:34.928074984 +0000 UTC m=+1103.980734325" watchObservedRunningTime="2026-09-18 14:02:34.929674986 +0000 UTC m=+1103.982334313"
Sep 18 14:02:34.929898 ip-10-0-141-222 kubenswrapper[2577]: E0918 14:02:34.929872    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7624-b45b7d\" is forbidden: User \"system:node:ip-10-0-141-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-222.ec2.internal' and this object" podUID="23b67288-9407-42f6-8b28-c5d0158b8975" pod="opendatahub/test-curl-7624-b45b7d"
Sep 18 14:02:34.931504 ip-10-0-141-222 kubenswrapper[2577]: E0918 14:02:34.931475    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7624-b45b7d\" is forbidden: User \"system:node:ip-10-0-141-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-222.ec2.internal' and this object" podUID="23b67288-9407-42f6-8b28-c5d0158b8975" pod="opendatahub/test-curl-7624-b45b7d"
Sep 18 14:02:35.124380 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:35.124351    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7624-f7ab6d"
Sep 18 14:02:35.272030 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:35.271998    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v44h4\" (UniqueName: \"kubernetes.io/projected/f8f6375a-9f4b-467b-829a-2ba56ca94d2b-kube-api-access-v44h4\") pod \"f8f6375a-9f4b-467b-829a-2ba56ca94d2b\" (UID: \"f8f6375a-9f4b-467b-829a-2ba56ca94d2b\") "
Sep 18 14:02:35.274137 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:35.274114    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8f6375a-9f4b-467b-829a-2ba56ca94d2b-kube-api-access-v44h4" (OuterVolumeSpecName: "kube-api-access-v44h4") pod "f8f6375a-9f4b-467b-829a-2ba56ca94d2b" (UID: "f8f6375a-9f4b-467b-829a-2ba56ca94d2b"). InnerVolumeSpecName "kube-api-access-v44h4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:02:35.373183 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:35.373150    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v44h4\" (UniqueName: \"kubernetes.io/projected/f8f6375a-9f4b-467b-829a-2ba56ca94d2b-kube-api-access-v44h4\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:02:35.546962 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:35.546914    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:35.565469 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:35.565436    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7624-9d3adc"]
Sep 18 14:02:35.569536 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:35.569514    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7624-9d3adc"]
Sep 18 14:02:35.921496 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:35.921461    2577 generic.go:356] "Generic (PLEG): container finished" podID="f8f6375a-9f4b-467b-829a-2ba56ca94d2b" containerID="c0812ba4d9801d40e91805808e7a66c3a6b1a35040ffa3c46058f13a3b4a86cf" exitCode=137
Sep 18 14:02:35.921952 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:35.921510    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7624-f7ab6d"
Sep 18 14:02:35.921952 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:35.921568    2577 scope.go:117] "RemoveContainer" containerID="c0812ba4d9801d40e91805808e7a66c3a6b1a35040ffa3c46058f13a3b4a86cf"
Sep 18 14:02:35.921952 ip-10-0-141-222 kubenswrapper[2577]: E0918 14:02:35.921891    2577 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7624-9d3adc" container="test-curl-7624-9d3adc"
Sep 18 14:02:35.930911 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:35.930898    2577 scope.go:117] "RemoveContainer" containerID="c0812ba4d9801d40e91805808e7a66c3a6b1a35040ffa3c46058f13a3b4a86cf"
Sep 18 14:02:35.931210 ip-10-0-141-222 kubenswrapper[2577]: E0918 14:02:35.931190    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0812ba4d9801d40e91805808e7a66c3a6b1a35040ffa3c46058f13a3b4a86cf\": container with ID starting with c0812ba4d9801d40e91805808e7a66c3a6b1a35040ffa3c46058f13a3b4a86cf not found: ID does not exist" containerID="c0812ba4d9801d40e91805808e7a66c3a6b1a35040ffa3c46058f13a3b4a86cf"
Sep 18 14:02:35.931282 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:35.931224    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0812ba4d9801d40e91805808e7a66c3a6b1a35040ffa3c46058f13a3b4a86cf"} err="failed to get container status \"c0812ba4d9801d40e91805808e7a66c3a6b1a35040ffa3c46058f13a3b4a86cf\": rpc error: code = NotFound desc = could not find container \"c0812ba4d9801d40e91805808e7a66c3a6b1a35040ffa3c46058f13a3b4a86cf\": container with ID starting with c0812ba4d9801d40e91805808e7a66c3a6b1a35040ffa3c46058f13a3b4a86cf not found: ID does not exist"
Sep 18 14:02:35.931417 ip-10-0-141-222 kubenswrapper[2577]: E0918 14:02:35.931400    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7624-f7ab6d\" is forbidden: User \"system:node:ip-10-0-141-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-222.ec2.internal' and this object" podUID="f8f6375a-9f4b-467b-829a-2ba56ca94d2b" pod="opendatahub/test-curl-7624-f7ab6d"
Sep 18 14:02:36.307368 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:36.307333    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:36.929340 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:36.929307    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7624-9d3adc" podUID="369c25fd-3331-4897-9fce-bb239b8a4be2" containerName="test-curl-7624-9d3adc" containerID="cri-o://1c8a62dd977ac373e56106984a96341f8a1b21ba23e3053a6d8e956a8d2d0312" gracePeriod=2
Sep 18 14:02:36.931898 ip-10-0-141-222 kubenswrapper[2577]: E0918 14:02:36.931727    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7624-9d3adc\" is forbidden: User \"system:node:ip-10-0-141-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-222.ec2.internal' and this object" podUID="369c25fd-3331-4897-9fce-bb239b8a4be2" pod="opendatahub/test-curl-7624-9d3adc"
Sep 18 14:02:36.933494 ip-10-0-141-222 kubenswrapper[2577]: E0918 14:02:36.933467    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7624-f7ab6d\" is forbidden: User \"system:node:ip-10-0-141-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-222.ec2.internal' and this object" podUID="f8f6375a-9f4b-467b-829a-2ba56ca94d2b" pod="opendatahub/test-curl-7624-f7ab6d"
Sep 18 14:02:37.070867 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:37.070842    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7624-b45b7d"
Sep 18 14:02:37.072891 ip-10-0-141-222 kubenswrapper[2577]: E0918 14:02:37.072862    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7624-9d3adc\" is forbidden: User \"system:node:ip-10-0-141-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-222.ec2.internal' and this object" podUID="369c25fd-3331-4897-9fce-bb239b8a4be2" pod="opendatahub/test-curl-7624-9d3adc"
Sep 18 14:02:37.074288 ip-10-0-141-222 kubenswrapper[2577]: E0918 14:02:37.074256    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7624-f7ab6d\" is forbidden: User \"system:node:ip-10-0-141-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-222.ec2.internal' and this object" podUID="f8f6375a-9f4b-467b-829a-2ba56ca94d2b" pod="opendatahub/test-curl-7624-f7ab6d"
Sep 18 14:02:37.108599 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:37.108571    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8f6375a-9f4b-467b-829a-2ba56ca94d2b" path="/var/lib/kubelet/pods/f8f6375a-9f4b-467b-829a-2ba56ca94d2b/volumes"
Sep 18 14:02:37.150986 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:37.150943    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:37.190066 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:37.189965    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dgcsw\" (UniqueName: \"kubernetes.io/projected/23b67288-9407-42f6-8b28-c5d0158b8975-kube-api-access-dgcsw\") pod \"23b67288-9407-42f6-8b28-c5d0158b8975\" (UID: \"23b67288-9407-42f6-8b28-c5d0158b8975\") "
Sep 18 14:02:37.192570 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:37.192539    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23b67288-9407-42f6-8b28-c5d0158b8975-kube-api-access-dgcsw" (OuterVolumeSpecName: "kube-api-access-dgcsw") pod "23b67288-9407-42f6-8b28-c5d0158b8975" (UID: "23b67288-9407-42f6-8b28-c5d0158b8975"). InnerVolumeSpecName "kube-api-access-dgcsw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:02:37.291790 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:37.291753    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgcsw\" (UniqueName: \"kubernetes.io/projected/23b67288-9407-42f6-8b28-c5d0158b8975-kube-api-access-dgcsw\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:02:37.828775 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:37.828724    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:37.937680 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:37.937646    2577 generic.go:356] "Generic (PLEG): container finished" podID="23b67288-9407-42f6-8b28-c5d0158b8975" containerID="44f42aa2c1e999e39c4ea2f4d08e34642d4618b7d959eda9257e927dd9609938" exitCode=137
Sep 18 14:02:37.938106 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:37.937686    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7624-b45b7d"
Sep 18 14:02:37.938106 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:37.937734    2577 scope.go:117] "RemoveContainer" containerID="44f42aa2c1e999e39c4ea2f4d08e34642d4618b7d959eda9257e927dd9609938"
Sep 18 14:02:37.947569 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:37.947552    2577 scope.go:117] "RemoveContainer" containerID="44f42aa2c1e999e39c4ea2f4d08e34642d4618b7d959eda9257e927dd9609938"
Sep 18 14:02:37.947875 ip-10-0-141-222 kubenswrapper[2577]: E0918 14:02:37.947854    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44f42aa2c1e999e39c4ea2f4d08e34642d4618b7d959eda9257e927dd9609938\": container with ID starting with 44f42aa2c1e999e39c4ea2f4d08e34642d4618b7d959eda9257e927dd9609938 not found: ID does not exist" containerID="44f42aa2c1e999e39c4ea2f4d08e34642d4618b7d959eda9257e927dd9609938"
Sep 18 14:02:37.947954 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:37.947888    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44f42aa2c1e999e39c4ea2f4d08e34642d4618b7d959eda9257e927dd9609938"} err="failed to get container status \"44f42aa2c1e999e39c4ea2f4d08e34642d4618b7d959eda9257e927dd9609938\": rpc error: code = NotFound desc = could not find container \"44f42aa2c1e999e39c4ea2f4d08e34642d4618b7d959eda9257e927dd9609938\": container with ID starting with 44f42aa2c1e999e39c4ea2f4d08e34642d4618b7d959eda9257e927dd9609938 not found: ID does not exist"
Sep 18 14:02:38.574723 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:38.574682    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:38.966376 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:38.966348    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:39.108872 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:39.106815    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7624-9d3adc"
Sep 18 14:02:39.114658 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:39.114629    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23b67288-9407-42f6-8b28-c5d0158b8975" path="/var/lib/kubelet/pods/23b67288-9407-42f6-8b28-c5d0158b8975/volumes"
Sep 18 14:02:39.212489 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:39.212411    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6dtbh\" (UniqueName: \"kubernetes.io/projected/369c25fd-3331-4897-9fce-bb239b8a4be2-kube-api-access-6dtbh\") pod \"369c25fd-3331-4897-9fce-bb239b8a4be2\" (UID: \"369c25fd-3331-4897-9fce-bb239b8a4be2\") "
Sep 18 14:02:39.214541 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:39.214515    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/369c25fd-3331-4897-9fce-bb239b8a4be2-kube-api-access-6dtbh" (OuterVolumeSpecName: "kube-api-access-6dtbh") pod "369c25fd-3331-4897-9fce-bb239b8a4be2" (UID: "369c25fd-3331-4897-9fce-bb239b8a4be2"). InnerVolumeSpecName "kube-api-access-6dtbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:02:39.313810 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:39.313778    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dtbh\" (UniqueName: \"kubernetes.io/projected/369c25fd-3331-4897-9fce-bb239b8a4be2-kube-api-access-6dtbh\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:02:39.368556 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:39.368524    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:39.463356 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:39.463284    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:39.940349 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:39.940317    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:39.951980 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:39.951946    2577 generic.go:356] "Generic (PLEG): container finished" podID="369c25fd-3331-4897-9fce-bb239b8a4be2" containerID="1c8a62dd977ac373e56106984a96341f8a1b21ba23e3053a6d8e956a8d2d0312" exitCode=137
Sep 18 14:02:39.952116 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:39.951986    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7624-9d3adc"
Sep 18 14:02:39.952116 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:39.952047    2577 scope.go:117] "RemoveContainer" containerID="1c8a62dd977ac373e56106984a96341f8a1b21ba23e3053a6d8e956a8d2d0312"
Sep 18 14:02:39.953897 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:39.953877    2577 generic.go:356] "Generic (PLEG): container finished" podID="1b4e8091-639f-434e-8bf3-4411ef5b0b7e" containerID="d2238084965e842fce7e00c218dd42f2e36814d052f5374cf387c7890c11474e" exitCode=0
Sep 18 14:02:39.954015 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:39.953933    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6" event={"ID":"1b4e8091-639f-434e-8bf3-4411ef5b0b7e","Type":"ContainerDied","Data":"d2238084965e842fce7e00c218dd42f2e36814d052f5374cf387c7890c11474e"}
Sep 18 14:02:39.954329 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:39.954311    2577 scope.go:117] "RemoveContainer" containerID="d2238084965e842fce7e00c218dd42f2e36814d052f5374cf387c7890c11474e"
Sep 18 14:02:39.955465 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:39.955432    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:02:39.955565 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:39.955505    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:02:39.955565 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:39.955545    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:02:39.962588 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:39.962374    2577 scope.go:117] "RemoveContainer" containerID="1c8a62dd977ac373e56106984a96341f8a1b21ba23e3053a6d8e956a8d2d0312"
Sep 18 14:02:39.962692 ip-10-0-141-222 kubenswrapper[2577]: E0918 14:02:39.962673    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c8a62dd977ac373e56106984a96341f8a1b21ba23e3053a6d8e956a8d2d0312\": container with ID starting with 1c8a62dd977ac373e56106984a96341f8a1b21ba23e3053a6d8e956a8d2d0312 not found: ID does not exist" containerID="1c8a62dd977ac373e56106984a96341f8a1b21ba23e3053a6d8e956a8d2d0312"
Sep 18 14:02:39.962736 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:39.962700    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c8a62dd977ac373e56106984a96341f8a1b21ba23e3053a6d8e956a8d2d0312"} err="failed to get container status \"1c8a62dd977ac373e56106984a96341f8a1b21ba23e3053a6d8e956a8d2d0312\": rpc error: code = NotFound desc = could not find container \"1c8a62dd977ac373e56106984a96341f8a1b21ba23e3053a6d8e956a8d2d0312\": container with ID starting with 1c8a62dd977ac373e56106984a96341f8a1b21ba23e3053a6d8e956a8d2d0312 not found: ID does not exist"
Sep 18 14:02:40.406316 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:40.406278    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:40.962539 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:40.962509    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6" event={"ID":"1b4e8091-639f-434e-8bf3-4411ef5b0b7e","Type":"ContainerStarted","Data":"df549ff85096cbcbbd24faded92a397abb3a94433a0e1db79e1dd98358dd4143"}
Sep 18 14:02:41.107958 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:41.107917    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="369c25fd-3331-4897-9fce-bb239b8a4be2" path="/var/lib/kubelet/pods/369c25fd-3331-4897-9fce-bb239b8a4be2/volumes"
Sep 18 14:02:41.221241 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:41.221151    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6"
Sep 18 14:02:41.222109 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:41.222081    2577 patch_prober.go:28] interesting pod/e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Sep 18 14:02:41.222228 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:41.222141    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6" podUID="1b4e8091-639f-434e-8bf3-4411ef5b0b7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Sep 18 14:02:42.079801 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:42.079759    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:42.222221 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:42.222182    2577 patch_prober.go:28] interesting pod/e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Sep 18 14:02:42.222405 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:42.222239    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6" podUID="1b4e8091-639f-434e-8bf3-4411ef5b0b7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Sep 18 14:02:42.665451 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:42.665411    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:42.759149 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:42.759116    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:43.126465 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:43.126431    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:43.221798 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:43.221761    2577 patch_prober.go:28] interesting pod/e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Sep 18 14:02:43.222013 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:43.221816    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6" podUID="1b4e8091-639f-434e-8bf3-4411ef5b0b7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Sep 18 14:02:43.764391 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:43.764348    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6"
Sep 18 14:02:43.920108 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:43.920071    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:44.221553 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:44.221519    2577 patch_prober.go:28] interesting pod/e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Sep 18 14:02:44.221957 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:44.221579    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6" podUID="1b4e8091-639f-434e-8bf3-4411ef5b0b7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Sep 18 14:02:44.616895 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:44.616853    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:45.222187 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:45.222151    2577 patch_prober.go:28] interesting pod/e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Sep 18 14:02:45.222581 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:45.222227    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6" podUID="1b4e8091-639f-434e-8bf3-4411ef5b0b7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Sep 18 14:02:45.629228 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:45.629191    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:46.221318 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:46.221277    2577 patch_prober.go:28] interesting pod/e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Sep 18 14:02:46.221516 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:46.221343    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6" podUID="1b4e8091-639f-434e-8bf3-4411ef5b0b7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Sep 18 14:02:46.244990 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:46.244960    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:46.999613 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:46.999585    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:47.115869 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:47.115837    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:47.224104 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:47.224071    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-ac71f8/noten-model-ac71f8-kserve-7d487b5787-8sf75"]
Sep 18 14:02:47.224462 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:47.224449    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="369c25fd-3331-4897-9fce-bb239b8a4be2" containerName="test-curl-7624-9d3adc"
Sep 18 14:02:47.224462 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:47.224464    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="369c25fd-3331-4897-9fce-bb239b8a4be2" containerName="test-curl-7624-9d3adc"
Sep 18 14:02:47.224564 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:47.224534    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="369c25fd-3331-4897-9fce-bb239b8a4be2" containerName="test-curl-7624-9d3adc"
Sep 18 14:02:47.227277 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:47.227254    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6"
Sep 18 14:02:47.227377 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:47.227326    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-ac71f8/noten-model-ac71f8-kserve-7d487b5787-8sf75"
Sep 18 14:02:47.228478 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:47.228457    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6"
Sep 18 14:02:47.231777 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:47.231761    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-ac71f8\"/\"kube-root-ca.crt\""
Sep 18 14:02:47.232484 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:47.232468    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-ac71f8\"/\"default-dockercfg-9pjvb\""
Sep 18 14:02:47.232550 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:47.232490    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-ac71f8\"/\"openshift-service-ca.crt\""
Sep 18 14:02:47.232550 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:47.232511    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-ac71f8\"/\"noten-model-ac71f8-kserve-self-signed-certs\""
Sep 18 14:02:47.246881 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:47.246857    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-ac71f8/noten-model-ac71f8-kserve-7d487b5787-8sf75"]
Sep 18 14:02:47.338482 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:47.338455    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:47.387763 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:47.387731    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1744a255-284b-49fe-aa41-eda1668129bc-model-cache\") pod \"noten-model-ac71f8-kserve-7d487b5787-8sf75\" (UID: \"1744a255-284b-49fe-aa41-eda1668129bc\") " pod="ai-tenant-e2e-noten-ac71f8/noten-model-ac71f8-kserve-7d487b5787-8sf75"
Sep 18 14:02:47.387763 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:47.387767    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1744a255-284b-49fe-aa41-eda1668129bc-tmp-dir\") pod \"noten-model-ac71f8-kserve-7d487b5787-8sf75\" (UID: \"1744a255-284b-49fe-aa41-eda1668129bc\") " pod="ai-tenant-e2e-noten-ac71f8/noten-model-ac71f8-kserve-7d487b5787-8sf75"
Sep 18 14:02:47.388036 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:47.387942    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1744a255-284b-49fe-aa41-eda1668129bc-tls-certs\") pod \"noten-model-ac71f8-kserve-7d487b5787-8sf75\" (UID: \"1744a255-284b-49fe-aa41-eda1668129bc\") " pod="ai-tenant-e2e-noten-ac71f8/noten-model-ac71f8-kserve-7d487b5787-8sf75"
Sep 18 14:02:47.388098 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:47.388068    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1744a255-284b-49fe-aa41-eda1668129bc-home\") pod \"noten-model-ac71f8-kserve-7d487b5787-8sf75\" (UID: \"1744a255-284b-49fe-aa41-eda1668129bc\") " pod="ai-tenant-e2e-noten-ac71f8/noten-model-ac71f8-kserve-7d487b5787-8sf75"
Sep 18 14:02:47.388154 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:47.388118    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1744a255-284b-49fe-aa41-eda1668129bc-dshm\") pod \"noten-model-ac71f8-kserve-7d487b5787-8sf75\" (UID: \"1744a255-284b-49fe-aa41-eda1668129bc\") " pod="ai-tenant-e2e-noten-ac71f8/noten-model-ac71f8-kserve-7d487b5787-8sf75"
Sep 18 14:02:47.388215 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:47.388152    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjhjp\" (UniqueName: \"kubernetes.io/projected/1744a255-284b-49fe-aa41-eda1668129bc-kube-api-access-zjhjp\") pod \"noten-model-ac71f8-kserve-7d487b5787-8sf75\" (UID: \"1744a255-284b-49fe-aa41-eda1668129bc\") " pod="ai-tenant-e2e-noten-ac71f8/noten-model-ac71f8-kserve-7d487b5787-8sf75"
Sep 18 14:02:47.451309 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:47.451253    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:47.489622 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:47.489586    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1744a255-284b-49fe-aa41-eda1668129bc-model-cache\") pod \"noten-model-ac71f8-kserve-7d487b5787-8sf75\" (UID: \"1744a255-284b-49fe-aa41-eda1668129bc\") " pod="ai-tenant-e2e-noten-ac71f8/noten-model-ac71f8-kserve-7d487b5787-8sf75"
Sep 18 14:02:47.489622 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:47.489631    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1744a255-284b-49fe-aa41-eda1668129bc-tmp-dir\") pod \"noten-model-ac71f8-kserve-7d487b5787-8sf75\" (UID: \"1744a255-284b-49fe-aa41-eda1668129bc\") " pod="ai-tenant-e2e-noten-ac71f8/noten-model-ac71f8-kserve-7d487b5787-8sf75"
Sep 18 14:02:47.489914 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:47.489672    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1744a255-284b-49fe-aa41-eda1668129bc-tls-certs\") pod \"noten-model-ac71f8-kserve-7d487b5787-8sf75\" (UID: \"1744a255-284b-49fe-aa41-eda1668129bc\") " pod="ai-tenant-e2e-noten-ac71f8/noten-model-ac71f8-kserve-7d487b5787-8sf75"
Sep 18 14:02:47.489914 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:47.489732    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1744a255-284b-49fe-aa41-eda1668129bc-home\") pod \"noten-model-ac71f8-kserve-7d487b5787-8sf75\" (UID: \"1744a255-284b-49fe-aa41-eda1668129bc\") " pod="ai-tenant-e2e-noten-ac71f8/noten-model-ac71f8-kserve-7d487b5787-8sf75"
Sep 18 14:02:47.489914 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:47.489765    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1744a255-284b-49fe-aa41-eda1668129bc-dshm\") pod \"noten-model-ac71f8-kserve-7d487b5787-8sf75\" (UID: \"1744a255-284b-49fe-aa41-eda1668129bc\") " pod="ai-tenant-e2e-noten-ac71f8/noten-model-ac71f8-kserve-7d487b5787-8sf75"
Sep 18 14:02:47.489914 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:47.489788    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjhjp\" (UniqueName: \"kubernetes.io/projected/1744a255-284b-49fe-aa41-eda1668129bc-kube-api-access-zjhjp\") pod \"noten-model-ac71f8-kserve-7d487b5787-8sf75\" (UID: \"1744a255-284b-49fe-aa41-eda1668129bc\") " pod="ai-tenant-e2e-noten-ac71f8/noten-model-ac71f8-kserve-7d487b5787-8sf75"
Sep 18 14:02:47.490142 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:47.490098    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1744a255-284b-49fe-aa41-eda1668129bc-model-cache\") pod \"noten-model-ac71f8-kserve-7d487b5787-8sf75\" (UID: \"1744a255-284b-49fe-aa41-eda1668129bc\") " pod="ai-tenant-e2e-noten-ac71f8/noten-model-ac71f8-kserve-7d487b5787-8sf75"
Sep 18 14:02:47.490198 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:47.490138    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1744a255-284b-49fe-aa41-eda1668129bc-tmp-dir\") pod \"noten-model-ac71f8-kserve-7d487b5787-8sf75\" (UID: \"1744a255-284b-49fe-aa41-eda1668129bc\") " pod="ai-tenant-e2e-noten-ac71f8/noten-model-ac71f8-kserve-7d487b5787-8sf75"
Sep 18 14:02:47.490198 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:47.490183    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1744a255-284b-49fe-aa41-eda1668129bc-home\") pod \"noten-model-ac71f8-kserve-7d487b5787-8sf75\" (UID: \"1744a255-284b-49fe-aa41-eda1668129bc\") " pod="ai-tenant-e2e-noten-ac71f8/noten-model-ac71f8-kserve-7d487b5787-8sf75"
Sep 18 14:02:47.492000 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:47.491979    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1744a255-284b-49fe-aa41-eda1668129bc-dshm\") pod \"noten-model-ac71f8-kserve-7d487b5787-8sf75\" (UID: \"1744a255-284b-49fe-aa41-eda1668129bc\") " pod="ai-tenant-e2e-noten-ac71f8/noten-model-ac71f8-kserve-7d487b5787-8sf75"
Sep 18 14:02:47.492321 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:47.492303    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1744a255-284b-49fe-aa41-eda1668129bc-tls-certs\") pod \"noten-model-ac71f8-kserve-7d487b5787-8sf75\" (UID: \"1744a255-284b-49fe-aa41-eda1668129bc\") " pod="ai-tenant-e2e-noten-ac71f8/noten-model-ac71f8-kserve-7d487b5787-8sf75"
Sep 18 14:02:47.499706 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:47.499684    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjhjp\" (UniqueName: \"kubernetes.io/projected/1744a255-284b-49fe-aa41-eda1668129bc-kube-api-access-zjhjp\") pod \"noten-model-ac71f8-kserve-7d487b5787-8sf75\" (UID: \"1744a255-284b-49fe-aa41-eda1668129bc\") " pod="ai-tenant-e2e-noten-ac71f8/noten-model-ac71f8-kserve-7d487b5787-8sf75"
Sep 18 14:02:47.536805 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:47.536781    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-ac71f8/noten-model-ac71f8-kserve-7d487b5787-8sf75"
Sep 18 14:02:47.694785 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:47.694756    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:47.797934 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:47.797901    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:47.813796 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:47.813763    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-ac71f8/noten-model-ac71f8-kserve-7d487b5787-8sf75"]
Sep 18 14:02:47.816982 ip-10-0-141-222 kubenswrapper[2577]: W0918 14:02:47.816949    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1744a255_284b_49fe_aa41_eda1668129bc.slice/crio-fde1880fe3dba7cab78aec85aef731311f8489dad8191910ad321d0206185449 WatchSource:0}: Error finding container fde1880fe3dba7cab78aec85aef731311f8489dad8191910ad321d0206185449: Status 404 returned error can't find the container with id fde1880fe3dba7cab78aec85aef731311f8489dad8191910ad321d0206185449
Sep 18 14:02:48.026883 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:48.026838    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-ac71f8/noten-model-ac71f8-kserve-7d487b5787-8sf75" event={"ID":"1744a255-284b-49fe-aa41-eda1668129bc","Type":"ContainerStarted","Data":"ff25c60334de60d8597741542d1c49bcbc0d028e7638df0449f0ff4200897956"}
Sep 18 14:02:48.027054 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:48.026892    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-ac71f8/noten-model-ac71f8-kserve-7d487b5787-8sf75" event={"ID":"1744a255-284b-49fe-aa41-eda1668129bc","Type":"ContainerStarted","Data":"fde1880fe3dba7cab78aec85aef731311f8489dad8191910ad321d0206185449"}
Sep 18 14:02:48.028036 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:48.028010    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:48.045216 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:48.045169    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-ac71f8/noten-model-ac71f8-kserve-7d487b5787-8sf75" podStartSLOduration=1.045155566 podStartE2EDuration="1.045155566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:02:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:02:48.04461067 +0000 UTC m=+1117.097269997" watchObservedRunningTime="2026-09-18 14:02:48.045155566 +0000 UTC m=+1117.097814893"
Sep 18 14:02:48.070394 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:48.070359    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b63f26-fd596bb8f-fhz8b"]
Sep 18 14:02:48.070700 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:48.070672    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-b63f26-fd596bb8f-fhz8b" podUID="7e06b525-41a0-43ac-a7c9-337a24138e89" containerName="payload-processing" containerID="cri-o://0438d633f236bb85b26ab317c38ef6cf26fda60a0591dfe39f80e4500b2e26c5" gracePeriod=30
Sep 18 14:02:48.081138 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:48.081107    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b63f26-65bd777cc4-z8cfp"]
Sep 18 14:02:48.081325 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:48.081300    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b63f26-65bd777cc4-z8cfp" podUID="2d080cb6-7e9f-4bf3-82ca-acb9b838f943" containerName="payload-pre-processing" containerID="cri-o://3ceff005192bd6a3cb9065c9edf2d55ec3b0263ed95ff68e85bdacedfc330941" gracePeriod=30
Sep 18 14:02:48.108921 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:48.108818    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:48.208465 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:48.208427    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:48.307546 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:48.307513    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:48.334541 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:48.334515    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b63f26-fd596bb8f-fhz8b"
Sep 18 14:02:48.345121 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:48.345100    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b63f26-65bd777cc4-z8cfp"
Sep 18 14:02:48.383567 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:48.383501    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:48.503894 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:48.503868    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e06b525-41a0-43ac-a7c9-337a24138e89-plugins-config-volume\") pod \"7e06b525-41a0-43ac-a7c9-337a24138e89\" (UID: \"7e06b525-41a0-43ac-a7c9-337a24138e89\") "
Sep 18 14:02:48.504034 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:48.503924    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d080cb6-7e9f-4bf3-82ca-acb9b838f943-plugins-config-volume\") pod \"2d080cb6-7e9f-4bf3-82ca-acb9b838f943\" (UID: \"2d080cb6-7e9f-4bf3-82ca-acb9b838f943\") "
Sep 18 14:02:48.504034 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:48.503973    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kph7s\" (UniqueName: \"kubernetes.io/projected/7e06b525-41a0-43ac-a7c9-337a24138e89-kube-api-access-kph7s\") pod \"7e06b525-41a0-43ac-a7c9-337a24138e89\" (UID: \"7e06b525-41a0-43ac-a7c9-337a24138e89\") "
Sep 18 14:02:48.504153 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:48.504062    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tgkj5\" (UniqueName: \"kubernetes.io/projected/2d080cb6-7e9f-4bf3-82ca-acb9b838f943-kube-api-access-tgkj5\") pod \"2d080cb6-7e9f-4bf3-82ca-acb9b838f943\" (UID: \"2d080cb6-7e9f-4bf3-82ca-acb9b838f943\") "
Sep 18 14:02:48.504338 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:48.504313    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d080cb6-7e9f-4bf3-82ca-acb9b838f943-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2d080cb6-7e9f-4bf3-82ca-acb9b838f943" (UID: "2d080cb6-7e9f-4bf3-82ca-acb9b838f943"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:02:48.504412 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:48.504311    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e06b525-41a0-43ac-a7c9-337a24138e89-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7e06b525-41a0-43ac-a7c9-337a24138e89" (UID: "7e06b525-41a0-43ac-a7c9-337a24138e89"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:02:48.506296 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:48.506259    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d080cb6-7e9f-4bf3-82ca-acb9b838f943-kube-api-access-tgkj5" (OuterVolumeSpecName: "kube-api-access-tgkj5") pod "2d080cb6-7e9f-4bf3-82ca-acb9b838f943" (UID: "2d080cb6-7e9f-4bf3-82ca-acb9b838f943"). InnerVolumeSpecName "kube-api-access-tgkj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:02:48.506417 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:48.506328    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e06b525-41a0-43ac-a7c9-337a24138e89-kube-api-access-kph7s" (OuterVolumeSpecName: "kube-api-access-kph7s") pod "7e06b525-41a0-43ac-a7c9-337a24138e89" (UID: "7e06b525-41a0-43ac-a7c9-337a24138e89"). InnerVolumeSpecName "kube-api-access-kph7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:02:48.605532 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:48.605496    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d080cb6-7e9f-4bf3-82ca-acb9b838f943-plugins-config-volume\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:02:48.605532 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:48.605525    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kph7s\" (UniqueName: \"kubernetes.io/projected/7e06b525-41a0-43ac-a7c9-337a24138e89-kube-api-access-kph7s\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:02:48.605532 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:48.605537    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgkj5\" (UniqueName: \"kubernetes.io/projected/2d080cb6-7e9f-4bf3-82ca-acb9b838f943-kube-api-access-tgkj5\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:02:48.605764 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:48.605547    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e06b525-41a0-43ac-a7c9-337a24138e89-plugins-config-volume\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:02:48.675727 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:48.675626    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:49.034115 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:49.034031    2577 generic.go:356] "Generic (PLEG): container finished" podID="7e06b525-41a0-43ac-a7c9-337a24138e89" containerID="0438d633f236bb85b26ab317c38ef6cf26fda60a0591dfe39f80e4500b2e26c5" exitCode=0
Sep 18 14:02:49.034115 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:49.034085    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b63f26-fd596bb8f-fhz8b"
Sep 18 14:02:49.034339 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:49.034126    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b63f26-fd596bb8f-fhz8b" event={"ID":"7e06b525-41a0-43ac-a7c9-337a24138e89","Type":"ContainerDied","Data":"0438d633f236bb85b26ab317c38ef6cf26fda60a0591dfe39f80e4500b2e26c5"}
Sep 18 14:02:49.034339 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:49.034185    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b63f26-fd596bb8f-fhz8b" event={"ID":"7e06b525-41a0-43ac-a7c9-337a24138e89","Type":"ContainerDied","Data":"c12a3c8a702cd3f9b2a6f52ad4662de19200dee0fb8b0ed734f15970354e56cc"}
Sep 18 14:02:49.034339 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:49.034217    2577 scope.go:117] "RemoveContainer" containerID="0438d633f236bb85b26ab317c38ef6cf26fda60a0591dfe39f80e4500b2e26c5"
Sep 18 14:02:49.035741 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:49.035602    2577 generic.go:356] "Generic (PLEG): container finished" podID="2d080cb6-7e9f-4bf3-82ca-acb9b838f943" containerID="3ceff005192bd6a3cb9065c9edf2d55ec3b0263ed95ff68e85bdacedfc330941" exitCode=0
Sep 18 14:02:49.035741 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:49.035645    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b63f26-65bd777cc4-z8cfp"
Sep 18 14:02:49.035741 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:49.035651    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b63f26-65bd777cc4-z8cfp" event={"ID":"2d080cb6-7e9f-4bf3-82ca-acb9b838f943","Type":"ContainerDied","Data":"3ceff005192bd6a3cb9065c9edf2d55ec3b0263ed95ff68e85bdacedfc330941"}
Sep 18 14:02:49.035741 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:49.035702    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b63f26-65bd777cc4-z8cfp" event={"ID":"2d080cb6-7e9f-4bf3-82ca-acb9b838f943","Type":"ContainerDied","Data":"7455839ade90a8942e1307be7d3de7a76cc57741ceb01d22efece6589d4c0a8f"}
Sep 18 14:02:49.045351 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:49.045335    2577 scope.go:117] "RemoveContainer" containerID="0438d633f236bb85b26ab317c38ef6cf26fda60a0591dfe39f80e4500b2e26c5"
Sep 18 14:02:49.045686 ip-10-0-141-222 kubenswrapper[2577]: E0918 14:02:49.045654    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0438d633f236bb85b26ab317c38ef6cf26fda60a0591dfe39f80e4500b2e26c5\": container with ID starting with 0438d633f236bb85b26ab317c38ef6cf26fda60a0591dfe39f80e4500b2e26c5 not found: ID does not exist" containerID="0438d633f236bb85b26ab317c38ef6cf26fda60a0591dfe39f80e4500b2e26c5"
Sep 18 14:02:49.045784 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:49.045689    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0438d633f236bb85b26ab317c38ef6cf26fda60a0591dfe39f80e4500b2e26c5"} err="failed to get container status \"0438d633f236bb85b26ab317c38ef6cf26fda60a0591dfe39f80e4500b2e26c5\": rpc error: code = NotFound desc = could not find container \"0438d633f236bb85b26ab317c38ef6cf26fda60a0591dfe39f80e4500b2e26c5\": container with ID starting with 0438d633f236bb85b26ab317c38ef6cf26fda60a0591dfe39f80e4500b2e26c5 not found: ID does not exist"
Sep 18 14:02:49.045784 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:49.045711    2577 scope.go:117] "RemoveContainer" containerID="3ceff005192bd6a3cb9065c9edf2d55ec3b0263ed95ff68e85bdacedfc330941"
Sep 18 14:02:49.054306 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:49.054259    2577 scope.go:117] "RemoveContainer" containerID="3ceff005192bd6a3cb9065c9edf2d55ec3b0263ed95ff68e85bdacedfc330941"
Sep 18 14:02:49.054580 ip-10-0-141-222 kubenswrapper[2577]: E0918 14:02:49.054560    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ceff005192bd6a3cb9065c9edf2d55ec3b0263ed95ff68e85bdacedfc330941\": container with ID starting with 3ceff005192bd6a3cb9065c9edf2d55ec3b0263ed95ff68e85bdacedfc330941 not found: ID does not exist" containerID="3ceff005192bd6a3cb9065c9edf2d55ec3b0263ed95ff68e85bdacedfc330941"
Sep 18 14:02:49.054633 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:49.054587    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ceff005192bd6a3cb9065c9edf2d55ec3b0263ed95ff68e85bdacedfc330941"} err="failed to get container status \"3ceff005192bd6a3cb9065c9edf2d55ec3b0263ed95ff68e85bdacedfc330941\": rpc error: code = NotFound desc = could not find container \"3ceff005192bd6a3cb9065c9edf2d55ec3b0263ed95ff68e85bdacedfc330941\": container with ID starting with 3ceff005192bd6a3cb9065c9edf2d55ec3b0263ed95ff68e85bdacedfc330941 not found: ID does not exist"
Sep 18 14:02:49.060409 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:49.060386    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:49.060791 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:49.060770    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b63f26-65bd777cc4-z8cfp"]
Sep 18 14:02:49.067695 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:49.067670    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b63f26-65bd777cc4-z8cfp"]
Sep 18 14:02:49.078688 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:49.078666    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b63f26-fd596bb8f-fhz8b"]
Sep 18 14:02:49.082948 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:49.082929    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b63f26-fd596bb8f-fhz8b"]
Sep 18 14:02:49.107750 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:49.107725    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d080cb6-7e9f-4bf3-82ca-acb9b838f943" path="/var/lib/kubelet/pods/2d080cb6-7e9f-4bf3-82ca-acb9b838f943/volumes"
Sep 18 14:02:49.108131 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:49.108118    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e06b525-41a0-43ac-a7c9-337a24138e89" path="/var/lib/kubelet/pods/7e06b525-41a0-43ac-a7c9-337a24138e89/volumes"
Sep 18 14:02:49.825973 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:49.825933    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-b63f26/test-model-b63f26-kserve-8589457995-qslk6" podUID="675617d1-fe1b-4ec5-9ee6-03ce4e0859bf" containerName="main" containerID="cri-o://eab18aa61409a2b1a849df370bbeadc6e0d37fd3f9e94878109da31ca218c6f1" gracePeriod=45
Sep 18 14:02:50.026385 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:50.026357    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:50.045944 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:50.045915    2577 generic.go:356] "Generic (PLEG): container finished" podID="675617d1-fe1b-4ec5-9ee6-03ce4e0859bf" containerID="eab18aa61409a2b1a849df370bbeadc6e0d37fd3f9e94878109da31ca218c6f1" exitCode=0
Sep 18 14:02:50.046140 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:50.045998    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b63f26/test-model-b63f26-kserve-8589457995-qslk6" event={"ID":"675617d1-fe1b-4ec5-9ee6-03ce4e0859bf","Type":"ContainerDied","Data":"eab18aa61409a2b1a849df370bbeadc6e0d37fd3f9e94878109da31ca218c6f1"}
Sep 18 14:02:50.079272 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:50.079247    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b63f26/test-model-b63f26-kserve-8589457995-qslk6"
Sep 18 14:02:50.220782 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:50.220752    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/675617d1-fe1b-4ec5-9ee6-03ce4e0859bf-tls-certs\") pod \"675617d1-fe1b-4ec5-9ee6-03ce4e0859bf\" (UID: \"675617d1-fe1b-4ec5-9ee6-03ce4e0859bf\") "
Sep 18 14:02:50.220972 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:50.220810    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/675617d1-fe1b-4ec5-9ee6-03ce4e0859bf-tmp-dir\") pod \"675617d1-fe1b-4ec5-9ee6-03ce4e0859bf\" (UID: \"675617d1-fe1b-4ec5-9ee6-03ce4e0859bf\") "
Sep 18 14:02:50.220972 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:50.220861    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/675617d1-fe1b-4ec5-9ee6-03ce4e0859bf-model-cache\") pod \"675617d1-fe1b-4ec5-9ee6-03ce4e0859bf\" (UID: \"675617d1-fe1b-4ec5-9ee6-03ce4e0859bf\") "
Sep 18 14:02:50.220972 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:50.220910    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zs2gv\" (UniqueName: \"kubernetes.io/projected/675617d1-fe1b-4ec5-9ee6-03ce4e0859bf-kube-api-access-zs2gv\") pod \"675617d1-fe1b-4ec5-9ee6-03ce4e0859bf\" (UID: \"675617d1-fe1b-4ec5-9ee6-03ce4e0859bf\") "
Sep 18 14:02:50.220972 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:50.220947    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/675617d1-fe1b-4ec5-9ee6-03ce4e0859bf-home\") pod \"675617d1-fe1b-4ec5-9ee6-03ce4e0859bf\" (UID: \"675617d1-fe1b-4ec5-9ee6-03ce4e0859bf\") "
Sep 18 14:02:50.221195 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:50.220976    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/675617d1-fe1b-4ec5-9ee6-03ce4e0859bf-dshm\") pod \"675617d1-fe1b-4ec5-9ee6-03ce4e0859bf\" (UID: \"675617d1-fe1b-4ec5-9ee6-03ce4e0859bf\") "
Sep 18 14:02:50.221195 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:50.221132    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/675617d1-fe1b-4ec5-9ee6-03ce4e0859bf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "675617d1-fe1b-4ec5-9ee6-03ce4e0859bf" (UID: "675617d1-fe1b-4ec5-9ee6-03ce4e0859bf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:02:50.221195 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:50.221169    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/675617d1-fe1b-4ec5-9ee6-03ce4e0859bf-model-cache" (OuterVolumeSpecName: "model-cache") pod "675617d1-fe1b-4ec5-9ee6-03ce4e0859bf" (UID: "675617d1-fe1b-4ec5-9ee6-03ce4e0859bf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:02:50.221355 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:50.221229    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/675617d1-fe1b-4ec5-9ee6-03ce4e0859bf-home" (OuterVolumeSpecName: "home") pod "675617d1-fe1b-4ec5-9ee6-03ce4e0859bf" (UID: "675617d1-fe1b-4ec5-9ee6-03ce4e0859bf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:02:50.221406 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:50.221365    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/675617d1-fe1b-4ec5-9ee6-03ce4e0859bf-home\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:02:50.221406 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:50.221383    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/675617d1-fe1b-4ec5-9ee6-03ce4e0859bf-tmp-dir\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:02:50.221406 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:50.221393    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/675617d1-fe1b-4ec5-9ee6-03ce4e0859bf-model-cache\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:02:50.223125 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:50.223104    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/675617d1-fe1b-4ec5-9ee6-03ce4e0859bf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "675617d1-fe1b-4ec5-9ee6-03ce4e0859bf" (UID: "675617d1-fe1b-4ec5-9ee6-03ce4e0859bf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:02:50.223213 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:50.223104    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/675617d1-fe1b-4ec5-9ee6-03ce4e0859bf-dshm" (OuterVolumeSpecName: "dshm") pod "675617d1-fe1b-4ec5-9ee6-03ce4e0859bf" (UID: "675617d1-fe1b-4ec5-9ee6-03ce4e0859bf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:02:50.223262 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:50.223211    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/675617d1-fe1b-4ec5-9ee6-03ce4e0859bf-kube-api-access-zs2gv" (OuterVolumeSpecName: "kube-api-access-zs2gv") pod "675617d1-fe1b-4ec5-9ee6-03ce4e0859bf" (UID: "675617d1-fe1b-4ec5-9ee6-03ce4e0859bf"). InnerVolumeSpecName "kube-api-access-zs2gv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:02:50.322776 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:50.322699    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/675617d1-fe1b-4ec5-9ee6-03ce4e0859bf-tls-certs\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:02:50.322776 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:50.322724    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zs2gv\" (UniqueName: \"kubernetes.io/projected/675617d1-fe1b-4ec5-9ee6-03ce4e0859bf-kube-api-access-zs2gv\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:02:50.322776 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:50.322735    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/675617d1-fe1b-4ec5-9ee6-03ce4e0859bf-dshm\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:02:50.684785 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:50.684747    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:51.054502 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:51.054416    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b63f26/test-model-b63f26-kserve-8589457995-qslk6"
Sep 18 14:02:51.054502 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:51.054437    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b63f26/test-model-b63f26-kserve-8589457995-qslk6" event={"ID":"675617d1-fe1b-4ec5-9ee6-03ce4e0859bf","Type":"ContainerDied","Data":"a0538a68d214741bf2cacbe1e9b7109a6043d6acdd3e14acf5a12e33911101e1"}
Sep 18 14:02:51.054502 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:51.054495    2577 scope.go:117] "RemoveContainer" containerID="eab18aa61409a2b1a849df370bbeadc6e0d37fd3f9e94878109da31ca218c6f1"
Sep 18 14:02:51.075111 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:51.075087    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-b63f26/test-model-b63f26-kserve-8589457995-qslk6"]
Sep 18 14:02:51.081460 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:51.081439    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-b63f26/test-model-b63f26-kserve-8589457995-qslk6"]
Sep 18 14:02:51.107093 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:51.107069    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="675617d1-fe1b-4ec5-9ee6-03ce4e0859bf" path="/var/lib/kubelet/pods/675617d1-fe1b-4ec5-9ee6-03ce4e0859bf/volumes"
Sep 18 14:02:51.360503 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:51.360472    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:52.132634 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:52.132599    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:52.790878 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:52.790838    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:55.270736 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:55.270701    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:55.346660 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:55.346628    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:55.513315 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:55.513283    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:56.060131 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:56.060100    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:56.339315 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:56.339289    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h"]
Sep 18 14:02:56.339671 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:56.339658    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="675617d1-fe1b-4ec5-9ee6-03ce4e0859bf" containerName="main"
Sep 18 14:02:56.339711 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:56.339673    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="675617d1-fe1b-4ec5-9ee6-03ce4e0859bf" containerName="main"
Sep 18 14:02:56.339711 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:56.339690    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e06b525-41a0-43ac-a7c9-337a24138e89" containerName="payload-processing"
Sep 18 14:02:56.339711 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:56.339695    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e06b525-41a0-43ac-a7c9-337a24138e89" containerName="payload-processing"
Sep 18 14:02:56.339711 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:56.339705    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d080cb6-7e9f-4bf3-82ca-acb9b838f943" containerName="payload-pre-processing"
Sep 18 14:02:56.339711 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:56.339711    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d080cb6-7e9f-4bf3-82ca-acb9b838f943" containerName="payload-pre-processing"
Sep 18 14:02:56.339908 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:56.339772    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e06b525-41a0-43ac-a7c9-337a24138e89" containerName="payload-processing"
Sep 18 14:02:56.339908 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:56.339784    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d080cb6-7e9f-4bf3-82ca-acb9b838f943" containerName="payload-pre-processing"
Sep 18 14:02:56.339908 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:56.339792    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="675617d1-fe1b-4ec5-9ee6-03ce4e0859bf" containerName="main"
Sep 18 14:02:56.342723 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:56.342706    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h"
Sep 18 14:02:56.345932 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:56.345907    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-783872-openshift-default-dockercfg-7h5fp\""
Sep 18 14:02:56.369543 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:56.369517    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h"]
Sep 18 14:02:56.372957 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:56.372928    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e5f9a01-999f-41d8-bb46-422417d62a33-workload-certs\") pod \"e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h\" (UID: \"4e5f9a01-999f-41d8-bb46-422417d62a33\") " pod="openshift-ingress/e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h"
Sep 18 14:02:56.373097 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:56.372988    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e5f9a01-999f-41d8-bb46-422417d62a33-istio-envoy\") pod \"e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h\" (UID: \"4e5f9a01-999f-41d8-bb46-422417d62a33\") " pod="openshift-ingress/e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h"
Sep 18 14:02:56.373097 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:56.373021    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e5f9a01-999f-41d8-bb46-422417d62a33-istio-podinfo\") pod \"e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h\" (UID: \"4e5f9a01-999f-41d8-bb46-422417d62a33\") " pod="openshift-ingress/e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h"
Sep 18 14:02:56.373197 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:56.373146    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e5f9a01-999f-41d8-bb46-422417d62a33-istio-data\") pod \"e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h\" (UID: \"4e5f9a01-999f-41d8-bb46-422417d62a33\") " pod="openshift-ingress/e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h"
Sep 18 14:02:56.373197 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:56.373183    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e5f9a01-999f-41d8-bb46-422417d62a33-istio-token\") pod \"e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h\" (UID: \"4e5f9a01-999f-41d8-bb46-422417d62a33\") " pod="openshift-ingress/e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h"
Sep 18 14:02:56.373302 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:56.373264    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e5f9a01-999f-41d8-bb46-422417d62a33-istiod-ca-cert\") pod \"e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h\" (UID: \"4e5f9a01-999f-41d8-bb46-422417d62a33\") " pod="openshift-ingress/e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h"
Sep 18 14:02:56.373383 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:56.373296    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e5f9a01-999f-41d8-bb46-422417d62a33-credential-socket\") pod \"e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h\" (UID: \"4e5f9a01-999f-41d8-bb46-422417d62a33\") " pod="openshift-ingress/e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h"
Sep 18 14:02:56.373383 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:56.373332    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2szv\" (UniqueName: \"kubernetes.io/projected/4e5f9a01-999f-41d8-bb46-422417d62a33-kube-api-access-t2szv\") pod \"e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h\" (UID: \"4e5f9a01-999f-41d8-bb46-422417d62a33\") " pod="openshift-ingress/e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h"
Sep 18 14:02:56.373383 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:56.373370    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e5f9a01-999f-41d8-bb46-422417d62a33-workload-socket\") pod \"e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h\" (UID: \"4e5f9a01-999f-41d8-bb46-422417d62a33\") " pod="openshift-ingress/e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h"
Sep 18 14:02:56.373551 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:56.373429    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:56.474174 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:56.474139    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2szv\" (UniqueName: \"kubernetes.io/projected/4e5f9a01-999f-41d8-bb46-422417d62a33-kube-api-access-t2szv\") pod \"e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h\" (UID: \"4e5f9a01-999f-41d8-bb46-422417d62a33\") " pod="openshift-ingress/e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h"
Sep 18 14:02:56.474373 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:56.474184    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e5f9a01-999f-41d8-bb46-422417d62a33-workload-socket\") pod \"e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h\" (UID: \"4e5f9a01-999f-41d8-bb46-422417d62a33\") " pod="openshift-ingress/e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h"
Sep 18 14:02:56.474373 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:56.474327    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e5f9a01-999f-41d8-bb46-422417d62a33-workload-certs\") pod \"e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h\" (UID: \"4e5f9a01-999f-41d8-bb46-422417d62a33\") " pod="openshift-ingress/e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h"
Sep 18 14:02:56.474508 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:56.474395    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e5f9a01-999f-41d8-bb46-422417d62a33-istio-envoy\") pod \"e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h\" (UID: \"4e5f9a01-999f-41d8-bb46-422417d62a33\") " pod="openshift-ingress/e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h"
Sep 18 14:02:56.474508 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:56.474428    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e5f9a01-999f-41d8-bb46-422417d62a33-istio-podinfo\") pod \"e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h\" (UID: \"4e5f9a01-999f-41d8-bb46-422417d62a33\") " pod="openshift-ingress/e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h"
Sep 18 14:02:56.474624 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:56.474506    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e5f9a01-999f-41d8-bb46-422417d62a33-istio-data\") pod \"e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h\" (UID: \"4e5f9a01-999f-41d8-bb46-422417d62a33\") " pod="openshift-ingress/e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h"
Sep 18 14:02:56.474624 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:56.474526    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e5f9a01-999f-41d8-bb46-422417d62a33-workload-socket\") pod \"e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h\" (UID: \"4e5f9a01-999f-41d8-bb46-422417d62a33\") " pod="openshift-ingress/e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h"
Sep 18 14:02:56.474624 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:56.474536    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e5f9a01-999f-41d8-bb46-422417d62a33-istio-token\") pod \"e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h\" (UID: \"4e5f9a01-999f-41d8-bb46-422417d62a33\") " pod="openshift-ingress/e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h"
Sep 18 14:02:56.474624 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:56.474609    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e5f9a01-999f-41d8-bb46-422417d62a33-istiod-ca-cert\") pod \"e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h\" (UID: \"4e5f9a01-999f-41d8-bb46-422417d62a33\") " pod="openshift-ingress/e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h"
Sep 18 14:02:56.474817 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:56.474637    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e5f9a01-999f-41d8-bb46-422417d62a33-credential-socket\") pod \"e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h\" (UID: \"4e5f9a01-999f-41d8-bb46-422417d62a33\") " pod="openshift-ingress/e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h"
Sep 18 14:02:56.474817 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:56.474660    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e5f9a01-999f-41d8-bb46-422417d62a33-workload-certs\") pod \"e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h\" (UID: \"4e5f9a01-999f-41d8-bb46-422417d62a33\") " pod="openshift-ingress/e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h"
Sep 18 14:02:56.474817 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:56.474801    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e5f9a01-999f-41d8-bb46-422417d62a33-istio-data\") pod \"e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h\" (UID: \"4e5f9a01-999f-41d8-bb46-422417d62a33\") " pod="openshift-ingress/e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h"
Sep 18 14:02:56.474967 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:56.474937    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e5f9a01-999f-41d8-bb46-422417d62a33-credential-socket\") pod \"e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h\" (UID: \"4e5f9a01-999f-41d8-bb46-422417d62a33\") " pod="openshift-ingress/e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h"
Sep 18 14:02:56.475284 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:56.475262    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e5f9a01-999f-41d8-bb46-422417d62a33-istiod-ca-cert\") pod \"e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h\" (UID: \"4e5f9a01-999f-41d8-bb46-422417d62a33\") " pod="openshift-ingress/e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h"
Sep 18 14:02:56.476769 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:56.476749    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e5f9a01-999f-41d8-bb46-422417d62a33-istio-envoy\") pod \"e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h\" (UID: \"4e5f9a01-999f-41d8-bb46-422417d62a33\") " pod="openshift-ingress/e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h"
Sep 18 14:02:56.477043 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:56.477022    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e5f9a01-999f-41d8-bb46-422417d62a33-istio-podinfo\") pod \"e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h\" (UID: \"4e5f9a01-999f-41d8-bb46-422417d62a33\") " pod="openshift-ingress/e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h"
Sep 18 14:02:56.484365 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:56.484345    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e5f9a01-999f-41d8-bb46-422417d62a33-istio-token\") pod \"e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h\" (UID: \"4e5f9a01-999f-41d8-bb46-422417d62a33\") " pod="openshift-ingress/e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h"
Sep 18 14:02:56.485527 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:56.485509    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2szv\" (UniqueName: \"kubernetes.io/projected/4e5f9a01-999f-41d8-bb46-422417d62a33-kube-api-access-t2szv\") pod \"e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h\" (UID: \"4e5f9a01-999f-41d8-bb46-422417d62a33\") " pod="openshift-ingress/e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h"
Sep 18 14:02:56.514078 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:56.514055    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:56.653475 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:56.653385    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h"
Sep 18 14:02:56.942226 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:56.942185    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h"]
Sep 18 14:02:56.942428 ip-10-0-141-222 kubenswrapper[2577]: W0918 14:02:56.942399    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e5f9a01_999f_41d8_bb46_422417d62a33.slice/crio-2ce62b385207fe3ff453ae0484c90ba0d122b9ba9f4763bfe56da9b540a86b61 WatchSource:0}: Error finding container 2ce62b385207fe3ff453ae0484c90ba0d122b9ba9f4763bfe56da9b540a86b61: Status 404 returned error can't find the container with id 2ce62b385207fe3ff453ae0484c90ba0d122b9ba9f4763bfe56da9b540a86b61
Sep 18 14:02:56.945114 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:56.945079    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:02:56.945191 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:56.945148    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:02:56.945191 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:56.945178    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:02:57.092220 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:57.092182    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h" event={"ID":"4e5f9a01-999f-41d8-bb46-422417d62a33","Type":"ContainerStarted","Data":"2cf8b9527178b27969acd289ec8b7b37e12513119787a02e0cb68db5b8f3952c"}
Sep 18 14:02:57.092446 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:57.092247    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h" event={"ID":"4e5f9a01-999f-41d8-bb46-422417d62a33","Type":"ContainerStarted","Data":"2ce62b385207fe3ff453ae0484c90ba0d122b9ba9f4763bfe56da9b540a86b61"}
Sep 18 14:02:57.116412 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:57.116352    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h" podStartSLOduration=1.116338015 podStartE2EDuration="1.116338015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:02:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:02:57.115973383 +0000 UTC m=+1126.168632714" watchObservedRunningTime="2026-09-18 14:02:57.116338015 +0000 UTC m=+1126.168997347"
Sep 18 14:02:57.536945 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:57.536909    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-ac71f8/noten-model-ac71f8-kserve-7d487b5787-8sf75"
Sep 18 14:02:57.537411 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:57.536959    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-ac71f8/noten-model-ac71f8-kserve-7d487b5787-8sf75"
Sep 18 14:02:57.549705 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:57.549682    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-ac71f8/noten-model-ac71f8-kserve-7d487b5787-8sf75"
Sep 18 14:02:57.653786 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:57.653748    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h"
Sep 18 14:02:57.658890 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:57.658858    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h"
Sep 18 14:02:58.098135 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:58.098099    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h"
Sep 18 14:02:58.099106 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:58.099085    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h"
Sep 18 14:02:58.109230 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:58.109208    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-ac71f8/noten-model-ac71f8-kserve-7d487b5787-8sf75"
Sep 18 14:02:58.300109 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:58.300067    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:59.145975 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:59.145932    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:59.263772 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:59.263721    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:59.960948 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:02:59.960914    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:00.941732 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:00.941699    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:01.647173 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:01.647140    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:02.533557 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:02.533487    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:02.632757 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:02.632724    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:03.343205 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:03.343145    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:03.477013 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:03.476977    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:03.577983 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:03.577951    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:03.724086 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:03.723993    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 14:03:03.976276 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:03.976195    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:04.082196 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:04.082157    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:04.089968 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:04.089939    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-783872-668dc565db-cktw6"]
Sep 18 14:03:04.092427 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:04.092413    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-783872-668dc565db-cktw6"
Sep 18 14:03:04.094777 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:04.094760    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-783872\""
Sep 18 14:03:04.105494 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:04.105461    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-783872-668dc565db-cktw6"]
Sep 18 14:03:04.145722 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:04.145687    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9kqs\" (UniqueName: \"kubernetes.io/projected/987fc83f-9e16-4765-894d-66787ce477ef-kube-api-access-q9kqs\") pod \"payload-pre-processing-e2e-shared-a-w3-783872-668dc565db-cktw6\" (UID: \"987fc83f-9e16-4765-894d-66787ce477ef\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-783872-668dc565db-cktw6"
Sep 18 14:03:04.145902 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:04.145730    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/987fc83f-9e16-4765-894d-66787ce477ef-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-783872-668dc565db-cktw6\" (UID: \"987fc83f-9e16-4765-894d-66787ce477ef\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-783872-668dc565db-cktw6"
Sep 18 14:03:04.247190 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:04.247091    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9kqs\" (UniqueName: \"kubernetes.io/projected/987fc83f-9e16-4765-894d-66787ce477ef-kube-api-access-q9kqs\") pod \"payload-pre-processing-e2e-shared-a-w3-783872-668dc565db-cktw6\" (UID: \"987fc83f-9e16-4765-894d-66787ce477ef\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-783872-668dc565db-cktw6"
Sep 18 14:03:04.247190 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:04.247160    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/987fc83f-9e16-4765-894d-66787ce477ef-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-783872-668dc565db-cktw6\" (UID: \"987fc83f-9e16-4765-894d-66787ce477ef\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-783872-668dc565db-cktw6"
Sep 18 14:03:04.247931 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:04.247902    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/987fc83f-9e16-4765-894d-66787ce477ef-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-783872-668dc565db-cktw6\" (UID: \"987fc83f-9e16-4765-894d-66787ce477ef\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-783872-668dc565db-cktw6"
Sep 18 14:03:04.259617 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:04.259587    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9kqs\" (UniqueName: \"kubernetes.io/projected/987fc83f-9e16-4765-894d-66787ce477ef-kube-api-access-q9kqs\") pod \"payload-pre-processing-e2e-shared-a-w3-783872-668dc565db-cktw6\" (UID: \"987fc83f-9e16-4765-894d-66787ce477ef\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-783872-668dc565db-cktw6"
Sep 18 14:03:04.404958 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:04.404924    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-783872-668dc565db-cktw6"
Sep 18 14:03:04.542811 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:04.542713    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:04.678918 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:04.678893    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-783872-668dc565db-cktw6"]
Sep 18 14:03:04.680763 ip-10-0-141-222 kubenswrapper[2577]: W0918 14:03:04.680731    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod987fc83f_9e16_4765_894d_66787ce477ef.slice/crio-c07e40130824b51ba3478f5402268de849326e82433556f0a4758a4aa43dc8c1 WatchSource:0}: Error finding container c07e40130824b51ba3478f5402268de849326e82433556f0a4758a4aa43dc8c1: Status 404 returned error can't find the container with id c07e40130824b51ba3478f5402268de849326e82433556f0a4758a4aa43dc8c1
Sep 18 14:03:05.141464 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:05.141429    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-783872-668dc565db-cktw6" event={"ID":"987fc83f-9e16-4765-894d-66787ce477ef","Type":"ContainerStarted","Data":"1e795a3f996f73e4e2a72773d631e23c43b19ac4db757534d3cca4e34f9bfdb0"}
Sep 18 14:03:05.141637 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:05.141485    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-783872-668dc565db-cktw6" event={"ID":"987fc83f-9e16-4765-894d-66787ce477ef","Type":"ContainerStarted","Data":"c07e40130824b51ba3478f5402268de849326e82433556f0a4758a4aa43dc8c1"}
Sep 18 14:03:05.141637 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:05.141590    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-783872-668dc565db-cktw6"
Sep 18 14:03:05.160561 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:05.160504    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-783872-668dc565db-cktw6" podStartSLOduration=1.160483663 podStartE2EDuration="1.160483663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:03:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:03:05.158085626 +0000 UTC m=+1134.210744953" watchObservedRunningTime="2026-09-18 14:03:05.160483663 +0000 UTC m=+1134.213142991"
Sep 18 14:03:05.331343 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:05.331307    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:06.031602 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:06.031565    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:06.832309 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:06.832272    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:07.543699 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:07.543652    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:08.538349 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:08.538310    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:09.245757 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:09.245720    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:16.149432 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:16.149400    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-783872-668dc565db-cktw6"
Sep 18 14:03:19.751102 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:19.751073    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:19.982424 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:19.982376    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:20.086500 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:20.086398    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:24.495700 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:24.495656    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:24.883689 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:24.883644    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-377ca2/e2e-conflict-b-d79684ca-kserve-647977bc69-k766t"]
Sep 18 14:03:24.887843 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:24.887798    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-377ca2/e2e-conflict-b-d79684ca-kserve-647977bc69-k766t"
Sep 18 14:03:24.890025 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:24.889998    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-377ca2\"/\"e2e-conflict-b-d79684ca-kserve-self-signed-certs\""
Sep 18 14:03:24.894234 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:24.894209    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-377ca2/e2e-conflict-b-d79684ca-kserve-647977bc69-k766t"]
Sep 18 14:03:25.051563 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:25.051529    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b41c0fbf-f3d1-483d-b66e-7b462a330097-tls-certs\") pod \"e2e-conflict-b-d79684ca-kserve-647977bc69-k766t\" (UID: \"b41c0fbf-f3d1-483d-b66e-7b462a330097\") " pod="e2e-models-e2e-worker-w4-377ca2/e2e-conflict-b-d79684ca-kserve-647977bc69-k766t"
Sep 18 14:03:25.051725 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:25.051571    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9stj\" (UniqueName: \"kubernetes.io/projected/b41c0fbf-f3d1-483d-b66e-7b462a330097-kube-api-access-l9stj\") pod \"e2e-conflict-b-d79684ca-kserve-647977bc69-k766t\" (UID: \"b41c0fbf-f3d1-483d-b66e-7b462a330097\") " pod="e2e-models-e2e-worker-w4-377ca2/e2e-conflict-b-d79684ca-kserve-647977bc69-k766t"
Sep 18 14:03:25.051725 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:25.051593    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b41c0fbf-f3d1-483d-b66e-7b462a330097-home\") pod \"e2e-conflict-b-d79684ca-kserve-647977bc69-k766t\" (UID: \"b41c0fbf-f3d1-483d-b66e-7b462a330097\") " pod="e2e-models-e2e-worker-w4-377ca2/e2e-conflict-b-d79684ca-kserve-647977bc69-k766t"
Sep 18 14:03:25.051815 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:25.051722    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b41c0fbf-f3d1-483d-b66e-7b462a330097-model-cache\") pod \"e2e-conflict-b-d79684ca-kserve-647977bc69-k766t\" (UID: \"b41c0fbf-f3d1-483d-b66e-7b462a330097\") " pod="e2e-models-e2e-worker-w4-377ca2/e2e-conflict-b-d79684ca-kserve-647977bc69-k766t"
Sep 18 14:03:25.051880 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:25.051814    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b41c0fbf-f3d1-483d-b66e-7b462a330097-dshm\") pod \"e2e-conflict-b-d79684ca-kserve-647977bc69-k766t\" (UID: \"b41c0fbf-f3d1-483d-b66e-7b462a330097\") " pod="e2e-models-e2e-worker-w4-377ca2/e2e-conflict-b-d79684ca-kserve-647977bc69-k766t"
Sep 18 14:03:25.051880 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:25.051852    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b41c0fbf-f3d1-483d-b66e-7b462a330097-tmp-dir\") pod \"e2e-conflict-b-d79684ca-kserve-647977bc69-k766t\" (UID: \"b41c0fbf-f3d1-483d-b66e-7b462a330097\") " pod="e2e-models-e2e-worker-w4-377ca2/e2e-conflict-b-d79684ca-kserve-647977bc69-k766t"
Sep 18 14:03:25.068700 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:25.068674    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-ac71f8/noten-model-ac71f8-kserve-7d487b5787-8sf75"]
Sep 18 14:03:25.125589 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:25.125515    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:25.152538 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:25.152462    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b41c0fbf-f3d1-483d-b66e-7b462a330097-dshm\") pod \"e2e-conflict-b-d79684ca-kserve-647977bc69-k766t\" (UID: \"b41c0fbf-f3d1-483d-b66e-7b462a330097\") " pod="e2e-models-e2e-worker-w4-377ca2/e2e-conflict-b-d79684ca-kserve-647977bc69-k766t"
Sep 18 14:03:25.152538 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:25.152494    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b41c0fbf-f3d1-483d-b66e-7b462a330097-tmp-dir\") pod \"e2e-conflict-b-d79684ca-kserve-647977bc69-k766t\" (UID: \"b41c0fbf-f3d1-483d-b66e-7b462a330097\") " pod="e2e-models-e2e-worker-w4-377ca2/e2e-conflict-b-d79684ca-kserve-647977bc69-k766t"
Sep 18 14:03:25.152724 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:25.152638    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b41c0fbf-f3d1-483d-b66e-7b462a330097-tls-certs\") pod \"e2e-conflict-b-d79684ca-kserve-647977bc69-k766t\" (UID: \"b41c0fbf-f3d1-483d-b66e-7b462a330097\") " pod="e2e-models-e2e-worker-w4-377ca2/e2e-conflict-b-d79684ca-kserve-647977bc69-k766t"
Sep 18 14:03:25.152724 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:25.152670    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9stj\" (UniqueName: \"kubernetes.io/projected/b41c0fbf-f3d1-483d-b66e-7b462a330097-kube-api-access-l9stj\") pod \"e2e-conflict-b-d79684ca-kserve-647977bc69-k766t\" (UID: \"b41c0fbf-f3d1-483d-b66e-7b462a330097\") " pod="e2e-models-e2e-worker-w4-377ca2/e2e-conflict-b-d79684ca-kserve-647977bc69-k766t"
Sep 18 14:03:25.152724 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:25.152699    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b41c0fbf-f3d1-483d-b66e-7b462a330097-home\") pod \"e2e-conflict-b-d79684ca-kserve-647977bc69-k766t\" (UID: \"b41c0fbf-f3d1-483d-b66e-7b462a330097\") " pod="e2e-models-e2e-worker-w4-377ca2/e2e-conflict-b-d79684ca-kserve-647977bc69-k766t"
Sep 18 14:03:25.152901 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:25.152749    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b41c0fbf-f3d1-483d-b66e-7b462a330097-model-cache\") pod \"e2e-conflict-b-d79684ca-kserve-647977bc69-k766t\" (UID: \"b41c0fbf-f3d1-483d-b66e-7b462a330097\") " pod="e2e-models-e2e-worker-w4-377ca2/e2e-conflict-b-d79684ca-kserve-647977bc69-k766t"
Sep 18 14:03:25.152901 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:25.152868    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b41c0fbf-f3d1-483d-b66e-7b462a330097-tmp-dir\") pod \"e2e-conflict-b-d79684ca-kserve-647977bc69-k766t\" (UID: \"b41c0fbf-f3d1-483d-b66e-7b462a330097\") " pod="e2e-models-e2e-worker-w4-377ca2/e2e-conflict-b-d79684ca-kserve-647977bc69-k766t"
Sep 18 14:03:25.153114 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:25.153072    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b41c0fbf-f3d1-483d-b66e-7b462a330097-model-cache\") pod \"e2e-conflict-b-d79684ca-kserve-647977bc69-k766t\" (UID: \"b41c0fbf-f3d1-483d-b66e-7b462a330097\") " pod="e2e-models-e2e-worker-w4-377ca2/e2e-conflict-b-d79684ca-kserve-647977bc69-k766t"
Sep 18 14:03:25.153236 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:25.153126    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b41c0fbf-f3d1-483d-b66e-7b462a330097-home\") pod \"e2e-conflict-b-d79684ca-kserve-647977bc69-k766t\" (UID: \"b41c0fbf-f3d1-483d-b66e-7b462a330097\") " pod="e2e-models-e2e-worker-w4-377ca2/e2e-conflict-b-d79684ca-kserve-647977bc69-k766t"
Sep 18 14:03:25.154807 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:25.154783    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b41c0fbf-f3d1-483d-b66e-7b462a330097-dshm\") pod \"e2e-conflict-b-d79684ca-kserve-647977bc69-k766t\" (UID: \"b41c0fbf-f3d1-483d-b66e-7b462a330097\") " pod="e2e-models-e2e-worker-w4-377ca2/e2e-conflict-b-d79684ca-kserve-647977bc69-k766t"
Sep 18 14:03:25.155132 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:25.155112    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b41c0fbf-f3d1-483d-b66e-7b462a330097-tls-certs\") pod \"e2e-conflict-b-d79684ca-kserve-647977bc69-k766t\" (UID: \"b41c0fbf-f3d1-483d-b66e-7b462a330097\") " pod="e2e-models-e2e-worker-w4-377ca2/e2e-conflict-b-d79684ca-kserve-647977bc69-k766t"
Sep 18 14:03:25.160298 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:25.160276    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9stj\" (UniqueName: \"kubernetes.io/projected/b41c0fbf-f3d1-483d-b66e-7b462a330097-kube-api-access-l9stj\") pod \"e2e-conflict-b-d79684ca-kserve-647977bc69-k766t\" (UID: \"b41c0fbf-f3d1-483d-b66e-7b462a330097\") " pod="e2e-models-e2e-worker-w4-377ca2/e2e-conflict-b-d79684ca-kserve-647977bc69-k766t"
Sep 18 14:03:25.199975 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:25.199945    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-377ca2/e2e-conflict-b-d79684ca-kserve-647977bc69-k766t"
Sep 18 14:03:25.478073 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:25.478038    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-377ca2/e2e-conflict-b-d79684ca-kserve-647977bc69-k766t"]
Sep 18 14:03:25.479445 ip-10-0-141-222 kubenswrapper[2577]: W0918 14:03:25.479418    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb41c0fbf_f3d1_483d_b66e_7b462a330097.slice/crio-534b8a16004a7348e25632f7ba03d9b82a1c8a96492cab420495f6b4641e10c6 WatchSource:0}: Error finding container 534b8a16004a7348e25632f7ba03d9b82a1c8a96492cab420495f6b4641e10c6: Status 404 returned error can't find the container with id 534b8a16004a7348e25632f7ba03d9b82a1c8a96492cab420495f6b4641e10c6
Sep 18 14:03:25.490655 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:25.490624    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:25.940634 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:25.940598    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:26.275596 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:26.275497    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-377ca2/e2e-conflict-b-d79684ca-kserve-647977bc69-k766t" event={"ID":"b41c0fbf-f3d1-483d-b66e-7b462a330097","Type":"ContainerStarted","Data":"0ff487597f6efd4a2fccf83d8e63d73d3cdfc5eecc17ca654902fe32f8afdbd0"}
Sep 18 14:03:26.275596 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:26.275545    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-377ca2/e2e-conflict-b-d79684ca-kserve-647977bc69-k766t" event={"ID":"b41c0fbf-f3d1-483d-b66e-7b462a330097","Type":"ContainerStarted","Data":"534b8a16004a7348e25632f7ba03d9b82a1c8a96492cab420495f6b4641e10c6"}
Sep 18 14:03:26.291930 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:26.291878    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-377ca2/e2e-conflict-b-d79684ca-kserve-647977bc69-k766t" podStartSLOduration=2.291861559 podStartE2EDuration="2.291861559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:03:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:03:26.290793292 +0000 UTC m=+1155.343452622" watchObservedRunningTime="2026-09-18 14:03:26.291861559 +0000 UTC m=+1155.344520886"
Sep 18 14:03:26.723748 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:26.723710    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:27.363745 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:27.363708    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:28.033467 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:28.033439    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:28.105086 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:28.105047    2577 patch_prober.go:28] interesting pod/e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": read tcp 10.132.0.2:35852->10.132.0.74:15021: read: connection reset by peer" start-of-body=
Sep 18 14:03:28.105221 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:28.105124    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h" podUID="4e5f9a01-999f-41d8-bb46-422417d62a33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": read tcp 10.132.0.2:35852->10.132.0.74:15021: read: connection reset by peer"
Sep 18 14:03:28.291768 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:28.291668    2577 generic.go:356] "Generic (PLEG): container finished" podID="4e5f9a01-999f-41d8-bb46-422417d62a33" containerID="2cf8b9527178b27969acd289ec8b7b37e12513119787a02e0cb68db5b8f3952c" exitCode=0
Sep 18 14:03:28.291768 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:28.291747    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h" event={"ID":"4e5f9a01-999f-41d8-bb46-422417d62a33","Type":"ContainerDied","Data":"2cf8b9527178b27969acd289ec8b7b37e12513119787a02e0cb68db5b8f3952c"}
Sep 18 14:03:28.292198 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:28.292183    2577 scope.go:117] "RemoveContainer" containerID="2cf8b9527178b27969acd289ec8b7b37e12513119787a02e0cb68db5b8f3952c"
Sep 18 14:03:28.293427 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:28.293399    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:03:28.293491 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:28.293466    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:03:28.293528 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:28.293496    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:03:28.845026 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:28.844990    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:29.078225 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:29.078181    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:29.306548 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:29.306446    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h" event={"ID":"4e5f9a01-999f-41d8-bb46-422417d62a33","Type":"ContainerStarted","Data":"11b0dbb74aa212753063903a81f87cbd7157f12e0904e986997a92645ce8d07b"}
Sep 18 14:03:29.487547 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:29.487512    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:29.654178 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:29.654142    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h"
Sep 18 14:03:29.654436 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:29.654413    2577 patch_prober.go:28] interesting pod/e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Sep 18 14:03:29.654484 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:29.654465    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h" podUID="4e5f9a01-999f-41d8-bb46-422417d62a33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Sep 18 14:03:29.926854 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:29.926738    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:30.062250 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:30.062212    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:30.300904 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:30.300418    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:30.653867 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:30.653809    2577 patch_prober.go:28] interesting pod/e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Sep 18 14:03:30.654019 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:30.653887    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h" podUID="4e5f9a01-999f-41d8-bb46-422417d62a33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Sep 18 14:03:30.690718 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:30.690684    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:31.653970 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:31.653935    2577 patch_prober.go:28] interesting pod/e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Sep 18 14:03:31.654389 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:31.653993    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h" podUID="4e5f9a01-999f-41d8-bb46-422417d62a33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Sep 18 14:03:32.433424 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:32.433391    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:32.654298 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:32.654247    2577 patch_prober.go:28] interesting pod/e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Sep 18 14:03:32.654680 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:32.654314    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h" podUID="4e5f9a01-999f-41d8-bb46-422417d62a33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Sep 18 14:03:33.137818 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:33.137784    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:33.225247 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:33.225206    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:33.654193 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:33.654155    2577 patch_prober.go:28] interesting pod/e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Sep 18 14:03:33.654382 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:33.654228    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h" podUID="4e5f9a01-999f-41d8-bb46-422417d62a33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Sep 18 14:03:34.383205 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:34.383160    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:34.658260 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:34.658182    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h"
Sep 18 14:03:34.658632 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:34.658460    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h"
Sep 18 14:03:34.659653 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:34.659632    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h"
Sep 18 14:03:35.200571 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:35.200540    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-377ca2/e2e-conflict-b-d79684ca-kserve-647977bc69-k766t"
Sep 18 14:03:35.200792 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:35.200586    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-377ca2/e2e-conflict-b-d79684ca-kserve-647977bc69-k766t"
Sep 18 14:03:35.213413 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:35.213385    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-377ca2/e2e-conflict-b-d79684ca-kserve-647977bc69-k766t"
Sep 18 14:03:35.290710 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:35.290678    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:35.353342 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:35.353315    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-377ca2/e2e-conflict-b-d79684ca-kserve-647977bc69-k766t"
Sep 18 14:03:36.175756 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:36.175726    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:36.857124 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:36.857084    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:36.898668 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:36.898634    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 14:03:37.732900 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:37.732867    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:38.507387 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:38.507354    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1a4b6d-764599fb9c-xxckl"]
Sep 18 14:03:38.510085 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:38.510066    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1a4b6d-764599fb9c-xxckl"
Sep 18 14:03:38.512877 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:38.512854    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-1a4b6d-serving-cert\""
Sep 18 14:03:38.513007 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:38.512980    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 18 14:03:38.513074 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:38.512867    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vx9xd\""
Sep 18 14:03:38.513136 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:38.512919    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 18 14:03:38.513136 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:38.512867    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 14:03:38.520867 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:38.520844    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1a4b6d-764599fb9c-xxckl"]
Sep 18 14:03:38.544070 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:38.544044    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:38.581346 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:38.581319    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39003008-ec43-427c-8642-784fcb7ea73b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-1a4b6d-764599fb9c-xxckl\" (UID: \"39003008-ec43-427c-8642-784fcb7ea73b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1a4b6d-764599fb9c-xxckl"
Sep 18 14:03:38.581346 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:38.581349    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39003008-ec43-427c-8642-784fcb7ea73b-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-1a4b6d-764599fb9c-xxckl\" (UID: \"39003008-ec43-427c-8642-784fcb7ea73b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1a4b6d-764599fb9c-xxckl"
Sep 18 14:03:38.581522 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:38.581383    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39003008-ec43-427c-8642-784fcb7ea73b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-1a4b6d-764599fb9c-xxckl\" (UID: \"39003008-ec43-427c-8642-784fcb7ea73b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1a4b6d-764599fb9c-xxckl"
Sep 18 14:03:38.581522 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:38.581475    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wljnf\" (UniqueName: \"kubernetes.io/projected/39003008-ec43-427c-8642-784fcb7ea73b-kube-api-access-wljnf\") pod \"maas-api-e2e-shared-b-w3-1a4b6d-764599fb9c-xxckl\" (UID: \"39003008-ec43-427c-8642-784fcb7ea73b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1a4b6d-764599fb9c-xxckl"
Sep 18 14:03:38.639523 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:38.639490    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:38.682324 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:38.682279    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39003008-ec43-427c-8642-784fcb7ea73b-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-1a4b6d-764599fb9c-xxckl\" (UID: \"39003008-ec43-427c-8642-784fcb7ea73b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1a4b6d-764599fb9c-xxckl"
Sep 18 14:03:38.682508 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:38.682359    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39003008-ec43-427c-8642-784fcb7ea73b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-1a4b6d-764599fb9c-xxckl\" (UID: \"39003008-ec43-427c-8642-784fcb7ea73b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1a4b6d-764599fb9c-xxckl"
Sep 18 14:03:38.682508 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:38.682449    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wljnf\" (UniqueName: \"kubernetes.io/projected/39003008-ec43-427c-8642-784fcb7ea73b-kube-api-access-wljnf\") pod \"maas-api-e2e-shared-b-w3-1a4b6d-764599fb9c-xxckl\" (UID: \"39003008-ec43-427c-8642-784fcb7ea73b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1a4b6d-764599fb9c-xxckl"
Sep 18 14:03:38.682635 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:38.682518    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39003008-ec43-427c-8642-784fcb7ea73b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-1a4b6d-764599fb9c-xxckl\" (UID: \"39003008-ec43-427c-8642-784fcb7ea73b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1a4b6d-764599fb9c-xxckl"
Sep 18 14:03:38.683156 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:38.683135    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39003008-ec43-427c-8642-784fcb7ea73b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-1a4b6d-764599fb9c-xxckl\" (UID: \"39003008-ec43-427c-8642-784fcb7ea73b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1a4b6d-764599fb9c-xxckl"
Sep 18 14:03:38.684817 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:38.684797    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39003008-ec43-427c-8642-784fcb7ea73b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-1a4b6d-764599fb9c-xxckl\" (UID: \"39003008-ec43-427c-8642-784fcb7ea73b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1a4b6d-764599fb9c-xxckl"
Sep 18 14:03:38.684917 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:38.684812    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39003008-ec43-427c-8642-784fcb7ea73b-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-1a4b6d-764599fb9c-xxckl\" (UID: \"39003008-ec43-427c-8642-784fcb7ea73b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1a4b6d-764599fb9c-xxckl"
Sep 18 14:03:38.690689 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:38.690671    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wljnf\" (UniqueName: \"kubernetes.io/projected/39003008-ec43-427c-8642-784fcb7ea73b-kube-api-access-wljnf\") pod \"maas-api-e2e-shared-b-w3-1a4b6d-764599fb9c-xxckl\" (UID: \"39003008-ec43-427c-8642-784fcb7ea73b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1a4b6d-764599fb9c-xxckl"
Sep 18 14:03:38.822227 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:38.822128    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1a4b6d-764599fb9c-xxckl"
Sep 18 14:03:39.042687 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:39.042657    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:39.098533 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:39.098505    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1a4b6d-764599fb9c-xxckl"]
Sep 18 14:03:39.100023 ip-10-0-141-222 kubenswrapper[2577]: W0918 14:03:39.099994    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39003008_ec43_427c_8642_784fcb7ea73b.slice/crio-0a748c68e3707e6b9e7eaa9cdc40f6f9b0f9f0f9a78842a4b286bd19678cbabf WatchSource:0}: Error finding container 0a748c68e3707e6b9e7eaa9cdc40f6f9b0f9f0f9a78842a4b286bd19678cbabf: Status 404 returned error can't find the container with id 0a748c68e3707e6b9e7eaa9cdc40f6f9b0f9f0f9a78842a4b286bd19678cbabf
Sep 18 14:03:39.101643 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:39.101622    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 14:03:39.141735 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:39.141705    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:39.366880 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:39.366778    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1a4b6d-764599fb9c-xxckl" event={"ID":"39003008-ec43-427c-8642-784fcb7ea73b","Type":"ContainerStarted","Data":"0a748c68e3707e6b9e7eaa9cdc40f6f9b0f9f0f9a78842a4b286bd19678cbabf"}
Sep 18 14:03:40.087626 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:40.087580    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-ac71f8/noten-model-ac71f8-kserve-7d487b5787-8sf75" podUID="1744a255-284b-49fe-aa41-eda1668129bc" containerName="main" containerID="cri-o://ff25c60334de60d8597741542d1c49bcbc0d028e7638df0449f0ff4200897956" gracePeriod=45
Sep 18 14:03:40.349263 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:40.349198    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-ac71f8/noten-model-ac71f8-kserve-7d487b5787-8sf75"
Sep 18 14:03:40.375810 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:40.375777    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1a4b6d-764599fb9c-xxckl" event={"ID":"39003008-ec43-427c-8642-784fcb7ea73b","Type":"ContainerStarted","Data":"45776957fa82e091194308f91f1a49b369def9a710682ee711dceda59fba54c9"}
Sep 18 14:03:40.376096 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:40.376037    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1a4b6d-764599fb9c-xxckl"
Sep 18 14:03:40.377548 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:40.377522    2577 generic.go:356] "Generic (PLEG): container finished" podID="1744a255-284b-49fe-aa41-eda1668129bc" containerID="ff25c60334de60d8597741542d1c49bcbc0d028e7638df0449f0ff4200897956" exitCode=0
Sep 18 14:03:40.377548 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:40.377552    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-ac71f8/noten-model-ac71f8-kserve-7d487b5787-8sf75"
Sep 18 14:03:40.377742 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:40.377592    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-ac71f8/noten-model-ac71f8-kserve-7d487b5787-8sf75" event={"ID":"1744a255-284b-49fe-aa41-eda1668129bc","Type":"ContainerDied","Data":"ff25c60334de60d8597741542d1c49bcbc0d028e7638df0449f0ff4200897956"}
Sep 18 14:03:40.377742 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:40.377627    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-ac71f8/noten-model-ac71f8-kserve-7d487b5787-8sf75" event={"ID":"1744a255-284b-49fe-aa41-eda1668129bc","Type":"ContainerDied","Data":"fde1880fe3dba7cab78aec85aef731311f8489dad8191910ad321d0206185449"}
Sep 18 14:03:40.377742 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:40.377646    2577 scope.go:117] "RemoveContainer" containerID="ff25c60334de60d8597741542d1c49bcbc0d028e7638df0449f0ff4200897956"
Sep 18 14:03:40.391616 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:40.390067    2577 scope.go:117] "RemoveContainer" containerID="ff25c60334de60d8597741542d1c49bcbc0d028e7638df0449f0ff4200897956"
Sep 18 14:03:40.391802 ip-10-0-141-222 kubenswrapper[2577]: E0918 14:03:40.391755    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff25c60334de60d8597741542d1c49bcbc0d028e7638df0449f0ff4200897956\": container with ID starting with ff25c60334de60d8597741542d1c49bcbc0d028e7638df0449f0ff4200897956 not found: ID does not exist" containerID="ff25c60334de60d8597741542d1c49bcbc0d028e7638df0449f0ff4200897956"
Sep 18 14:03:40.391936 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:40.391795    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff25c60334de60d8597741542d1c49bcbc0d028e7638df0449f0ff4200897956"} err="failed to get container status \"ff25c60334de60d8597741542d1c49bcbc0d028e7638df0449f0ff4200897956\": rpc error: code = NotFound desc = could not find container \"ff25c60334de60d8597741542d1c49bcbc0d028e7638df0449f0ff4200897956\": container with ID starting with ff25c60334de60d8597741542d1c49bcbc0d028e7638df0449f0ff4200897956 not found: ID does not exist"
Sep 18 14:03:40.395009 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:40.394968    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1a4b6d-764599fb9c-xxckl" podStartSLOduration=1.973311824 podStartE2EDuration="2.394953812s" totalImagesPullingTime="421.641988ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:03:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:03:40.392286013 +0000 UTC m=+1169.444945341" watchObservedRunningTime="2026-09-18 14:03:40.394953812 +0000 UTC m=+1169.447613143"
Sep 18 14:03:40.400366 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:40.400341    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1744a255-284b-49fe-aa41-eda1668129bc-tls-certs\") pod \"1744a255-284b-49fe-aa41-eda1668129bc\" (UID: \"1744a255-284b-49fe-aa41-eda1668129bc\") "
Sep 18 14:03:40.400468 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:40.400406    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1744a255-284b-49fe-aa41-eda1668129bc-tmp-dir\") pod \"1744a255-284b-49fe-aa41-eda1668129bc\" (UID: \"1744a255-284b-49fe-aa41-eda1668129bc\") "
Sep 18 14:03:40.400539 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:40.400468    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1744a255-284b-49fe-aa41-eda1668129bc-dshm\") pod \"1744a255-284b-49fe-aa41-eda1668129bc\" (UID: \"1744a255-284b-49fe-aa41-eda1668129bc\") "
Sep 18 14:03:40.400539 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:40.400508    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1744a255-284b-49fe-aa41-eda1668129bc-model-cache\") pod \"1744a255-284b-49fe-aa41-eda1668129bc\" (UID: \"1744a255-284b-49fe-aa41-eda1668129bc\") "
Sep 18 14:03:40.400646 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:40.400629    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zjhjp\" (UniqueName: \"kubernetes.io/projected/1744a255-284b-49fe-aa41-eda1668129bc-kube-api-access-zjhjp\") pod \"1744a255-284b-49fe-aa41-eda1668129bc\" (UID: \"1744a255-284b-49fe-aa41-eda1668129bc\") "
Sep 18 14:03:40.400726 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:40.400701    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1744a255-284b-49fe-aa41-eda1668129bc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1744a255-284b-49fe-aa41-eda1668129bc" (UID: "1744a255-284b-49fe-aa41-eda1668129bc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:03:40.400859 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:40.400814    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1744a255-284b-49fe-aa41-eda1668129bc-model-cache" (OuterVolumeSpecName: "model-cache") pod "1744a255-284b-49fe-aa41-eda1668129bc" (UID: "1744a255-284b-49fe-aa41-eda1668129bc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:03:40.401043 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:40.401026    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1744a255-284b-49fe-aa41-eda1668129bc-home\") pod \"1744a255-284b-49fe-aa41-eda1668129bc\" (UID: \"1744a255-284b-49fe-aa41-eda1668129bc\") "
Sep 18 14:03:40.401370 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:40.401347    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1744a255-284b-49fe-aa41-eda1668129bc-home" (OuterVolumeSpecName: "home") pod "1744a255-284b-49fe-aa41-eda1668129bc" (UID: "1744a255-284b-49fe-aa41-eda1668129bc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:03:40.401503 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:40.401466    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1744a255-284b-49fe-aa41-eda1668129bc-home\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:03:40.401503 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:40.401485    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1744a255-284b-49fe-aa41-eda1668129bc-tmp-dir\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:03:40.401503 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:40.401499    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1744a255-284b-49fe-aa41-eda1668129bc-model-cache\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:03:40.402887 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:40.402858    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1744a255-284b-49fe-aa41-eda1668129bc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1744a255-284b-49fe-aa41-eda1668129bc" (UID: "1744a255-284b-49fe-aa41-eda1668129bc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:03:40.402887 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:40.402874    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1744a255-284b-49fe-aa41-eda1668129bc-dshm" (OuterVolumeSpecName: "dshm") pod "1744a255-284b-49fe-aa41-eda1668129bc" (UID: "1744a255-284b-49fe-aa41-eda1668129bc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:03:40.403135 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:40.403113    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1744a255-284b-49fe-aa41-eda1668129bc-kube-api-access-zjhjp" (OuterVolumeSpecName: "kube-api-access-zjhjp") pod "1744a255-284b-49fe-aa41-eda1668129bc" (UID: "1744a255-284b-49fe-aa41-eda1668129bc"). InnerVolumeSpecName "kube-api-access-zjhjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:03:40.502336 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:40.502297    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjhjp\" (UniqueName: \"kubernetes.io/projected/1744a255-284b-49fe-aa41-eda1668129bc-kube-api-access-zjhjp\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:03:40.502336 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:40.502327    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1744a255-284b-49fe-aa41-eda1668129bc-tls-certs\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:03:40.502336 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:40.502338    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1744a255-284b-49fe-aa41-eda1668129bc-dshm\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:03:40.706277 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:40.706246    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-ac71f8/noten-model-ac71f8-kserve-7d487b5787-8sf75"]
Sep 18 14:03:40.719094 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:40.713449    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-ac71f8/noten-model-ac71f8-kserve-7d487b5787-8sf75"]
Sep 18 14:03:41.107860 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:41.107801    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1744a255-284b-49fe-aa41-eda1668129bc" path="/var/lib/kubelet/pods/1744a255-284b-49fe-aa41-eda1668129bc/volumes"
Sep 18 14:03:46.390565 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:03:46.390539    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1a4b6d-764599fb9c-xxckl"
Sep 18 14:04:00.573097 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:00.573059    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:04:01.224256 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:01.224222    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:04:02.133990 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:02.133957    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:04:02.674242 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:02.674214    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:04:02.832024 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:02.831993    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:04:03.155056 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:03.155022    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:04:03.419145 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:03.419061    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:04:03.685011 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:03.684925    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:04:04.538527 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:04.538491    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:04:05.192793 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:05.192760    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:04:06.030919 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:06.030885    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:04:06.502600 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:06.502567    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 14:04:06.836172 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:06.836087    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:04:13.167954 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:13.167922    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-8tnxl_69d3178a-5769-4b5f-a243-30a1bedc5bd6/console-operator/1.log"
Sep 18 14:04:13.170023 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:13.169997    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-8tnxl_69d3178a-5769-4b5f-a243-30a1bedc5bd6/console-operator/1.log"
Sep 18 14:04:16.023191 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:16.023151    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-ed4c53/distinct-2-w0-ed4c53-kserve-7ccfc9d644-hjn2r"]
Sep 18 14:04:16.024181 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:16.024140    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1744a255-284b-49fe-aa41-eda1668129bc" containerName="main"
Sep 18 14:04:16.024181 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:16.024182    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1744a255-284b-49fe-aa41-eda1668129bc" containerName="main"
Sep 18 14:04:16.024548 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:16.024530    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1744a255-284b-49fe-aa41-eda1668129bc" containerName="main"
Sep 18 14:04:16.029294 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:16.029268    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ed4c53/distinct-2-w0-ed4c53-kserve-7ccfc9d644-hjn2r"
Sep 18 14:04:16.032831 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:16.032800    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ed4c53\"/\"distinct-2-w0-ed4c53-kserve-self-signed-certs\""
Sep 18 14:04:16.042634 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:16.042610    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ed4c53/distinct-2-w0-ed4c53-kserve-7ccfc9d644-hjn2r"]
Sep 18 14:04:16.118077 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:16.118044    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:04:16.146382 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:16.146353    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d030a33-c49f-4697-9968-47f837e16bea-model-cache\") pod \"distinct-2-w0-ed4c53-kserve-7ccfc9d644-hjn2r\" (UID: \"1d030a33-c49f-4697-9968-47f837e16bea\") " pod="e2e-models-e2e-worker-w0-ed4c53/distinct-2-w0-ed4c53-kserve-7ccfc9d644-hjn2r"
Sep 18 14:04:16.146535 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:16.146406    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7q4t\" (UniqueName: \"kubernetes.io/projected/1d030a33-c49f-4697-9968-47f837e16bea-kube-api-access-p7q4t\") pod \"distinct-2-w0-ed4c53-kserve-7ccfc9d644-hjn2r\" (UID: \"1d030a33-c49f-4697-9968-47f837e16bea\") " pod="e2e-models-e2e-worker-w0-ed4c53/distinct-2-w0-ed4c53-kserve-7ccfc9d644-hjn2r"
Sep 18 14:04:16.146535 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:16.146480    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d030a33-c49f-4697-9968-47f837e16bea-tls-certs\") pod \"distinct-2-w0-ed4c53-kserve-7ccfc9d644-hjn2r\" (UID: \"1d030a33-c49f-4697-9968-47f837e16bea\") " pod="e2e-models-e2e-worker-w0-ed4c53/distinct-2-w0-ed4c53-kserve-7ccfc9d644-hjn2r"
Sep 18 14:04:16.146535 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:16.146514    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d030a33-c49f-4697-9968-47f837e16bea-dshm\") pod \"distinct-2-w0-ed4c53-kserve-7ccfc9d644-hjn2r\" (UID: \"1d030a33-c49f-4697-9968-47f837e16bea\") " pod="e2e-models-e2e-worker-w0-ed4c53/distinct-2-w0-ed4c53-kserve-7ccfc9d644-hjn2r"
Sep 18 14:04:16.146643 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:16.146538    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d030a33-c49f-4697-9968-47f837e16bea-home\") pod \"distinct-2-w0-ed4c53-kserve-7ccfc9d644-hjn2r\" (UID: \"1d030a33-c49f-4697-9968-47f837e16bea\") " pod="e2e-models-e2e-worker-w0-ed4c53/distinct-2-w0-ed4c53-kserve-7ccfc9d644-hjn2r"
Sep 18 14:04:16.146643 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:16.146576    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d030a33-c49f-4697-9968-47f837e16bea-tmp-dir\") pod \"distinct-2-w0-ed4c53-kserve-7ccfc9d644-hjn2r\" (UID: \"1d030a33-c49f-4697-9968-47f837e16bea\") " pod="e2e-models-e2e-worker-w0-ed4c53/distinct-2-w0-ed4c53-kserve-7ccfc9d644-hjn2r"
Sep 18 14:04:16.248095 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:16.248044    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7q4t\" (UniqueName: \"kubernetes.io/projected/1d030a33-c49f-4697-9968-47f837e16bea-kube-api-access-p7q4t\") pod \"distinct-2-w0-ed4c53-kserve-7ccfc9d644-hjn2r\" (UID: \"1d030a33-c49f-4697-9968-47f837e16bea\") " pod="e2e-models-e2e-worker-w0-ed4c53/distinct-2-w0-ed4c53-kserve-7ccfc9d644-hjn2r"
Sep 18 14:04:16.248281 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:16.248114    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d030a33-c49f-4697-9968-47f837e16bea-tls-certs\") pod \"distinct-2-w0-ed4c53-kserve-7ccfc9d644-hjn2r\" (UID: \"1d030a33-c49f-4697-9968-47f837e16bea\") " pod="e2e-models-e2e-worker-w0-ed4c53/distinct-2-w0-ed4c53-kserve-7ccfc9d644-hjn2r"
Sep 18 14:04:16.248281 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:16.248164    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d030a33-c49f-4697-9968-47f837e16bea-dshm\") pod \"distinct-2-w0-ed4c53-kserve-7ccfc9d644-hjn2r\" (UID: \"1d030a33-c49f-4697-9968-47f837e16bea\") " pod="e2e-models-e2e-worker-w0-ed4c53/distinct-2-w0-ed4c53-kserve-7ccfc9d644-hjn2r"
Sep 18 14:04:16.248281 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:16.248191    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d030a33-c49f-4697-9968-47f837e16bea-home\") pod \"distinct-2-w0-ed4c53-kserve-7ccfc9d644-hjn2r\" (UID: \"1d030a33-c49f-4697-9968-47f837e16bea\") " pod="e2e-models-e2e-worker-w0-ed4c53/distinct-2-w0-ed4c53-kserve-7ccfc9d644-hjn2r"
Sep 18 14:04:16.248432 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:16.248312    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d030a33-c49f-4697-9968-47f837e16bea-tmp-dir\") pod \"distinct-2-w0-ed4c53-kserve-7ccfc9d644-hjn2r\" (UID: \"1d030a33-c49f-4697-9968-47f837e16bea\") " pod="e2e-models-e2e-worker-w0-ed4c53/distinct-2-w0-ed4c53-kserve-7ccfc9d644-hjn2r"
Sep 18 14:04:16.248432 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:16.248421    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d030a33-c49f-4697-9968-47f837e16bea-model-cache\") pod \"distinct-2-w0-ed4c53-kserve-7ccfc9d644-hjn2r\" (UID: \"1d030a33-c49f-4697-9968-47f837e16bea\") " pod="e2e-models-e2e-worker-w0-ed4c53/distinct-2-w0-ed4c53-kserve-7ccfc9d644-hjn2r"
Sep 18 14:04:16.248611 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:16.248591    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d030a33-c49f-4697-9968-47f837e16bea-home\") pod \"distinct-2-w0-ed4c53-kserve-7ccfc9d644-hjn2r\" (UID: \"1d030a33-c49f-4697-9968-47f837e16bea\") " pod="e2e-models-e2e-worker-w0-ed4c53/distinct-2-w0-ed4c53-kserve-7ccfc9d644-hjn2r"
Sep 18 14:04:16.248709 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:16.248686    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d030a33-c49f-4697-9968-47f837e16bea-tmp-dir\") pod \"distinct-2-w0-ed4c53-kserve-7ccfc9d644-hjn2r\" (UID: \"1d030a33-c49f-4697-9968-47f837e16bea\") " pod="e2e-models-e2e-worker-w0-ed4c53/distinct-2-w0-ed4c53-kserve-7ccfc9d644-hjn2r"
Sep 18 14:04:16.248767 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:16.248723    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d030a33-c49f-4697-9968-47f837e16bea-model-cache\") pod \"distinct-2-w0-ed4c53-kserve-7ccfc9d644-hjn2r\" (UID: \"1d030a33-c49f-4697-9968-47f837e16bea\") " pod="e2e-models-e2e-worker-w0-ed4c53/distinct-2-w0-ed4c53-kserve-7ccfc9d644-hjn2r"
Sep 18 14:04:16.250626 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:16.250602    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d030a33-c49f-4697-9968-47f837e16bea-dshm\") pod \"distinct-2-w0-ed4c53-kserve-7ccfc9d644-hjn2r\" (UID: \"1d030a33-c49f-4697-9968-47f837e16bea\") " pod="e2e-models-e2e-worker-w0-ed4c53/distinct-2-w0-ed4c53-kserve-7ccfc9d644-hjn2r"
Sep 18 14:04:16.250914 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:16.250898    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d030a33-c49f-4697-9968-47f837e16bea-tls-certs\") pod \"distinct-2-w0-ed4c53-kserve-7ccfc9d644-hjn2r\" (UID: \"1d030a33-c49f-4697-9968-47f837e16bea\") " pod="e2e-models-e2e-worker-w0-ed4c53/distinct-2-w0-ed4c53-kserve-7ccfc9d644-hjn2r"
Sep 18 14:04:16.257817 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:16.257794    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7q4t\" (UniqueName: \"kubernetes.io/projected/1d030a33-c49f-4697-9968-47f837e16bea-kube-api-access-p7q4t\") pod \"distinct-2-w0-ed4c53-kserve-7ccfc9d644-hjn2r\" (UID: \"1d030a33-c49f-4697-9968-47f837e16bea\") " pod="e2e-models-e2e-worker-w0-ed4c53/distinct-2-w0-ed4c53-kserve-7ccfc9d644-hjn2r"
Sep 18 14:04:16.339404 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:16.339376    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ed4c53/distinct-2-w0-ed4c53-kserve-7ccfc9d644-hjn2r"
Sep 18 14:04:16.622385 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:16.622294    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ed4c53/distinct-2-w0-ed4c53-kserve-7ccfc9d644-hjn2r"]
Sep 18 14:04:16.625313 ip-10-0-141-222 kubenswrapper[2577]: W0918 14:04:16.625286    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1d030a33_c49f_4697_9968_47f837e16bea.slice/crio-a44f727e6912f5a06bd97c8a07911755020088a7bb5f287bb640eb378aed9dcc WatchSource:0}: Error finding container a44f727e6912f5a06bd97c8a07911755020088a7bb5f287bb640eb378aed9dcc: Status 404 returned error can't find the container with id a44f727e6912f5a06bd97c8a07911755020088a7bb5f287bb640eb378aed9dcc
Sep 18 14:04:17.595081 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:17.595041    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ed4c53/distinct-2-w0-ed4c53-kserve-7ccfc9d644-hjn2r" event={"ID":"1d030a33-c49f-4697-9968-47f837e16bea","Type":"ContainerStarted","Data":"9ce7e11c667a658dc3f38ef824a538bebb9e5b27094fc5a65f5d6780b615346f"}
Sep 18 14:04:17.595081 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:17.595087    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ed4c53/distinct-2-w0-ed4c53-kserve-7ccfc9d644-hjn2r" event={"ID":"1d030a33-c49f-4697-9968-47f837e16bea","Type":"ContainerStarted","Data":"a44f727e6912f5a06bd97c8a07911755020088a7bb5f287bb640eb378aed9dcc"}
Sep 18 14:04:17.614518 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:17.614472    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-ed4c53/distinct-2-w0-ed4c53-kserve-7ccfc9d644-hjn2r" podStartSLOduration=1.614457572 podStartE2EDuration="1.614457572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:04:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:04:17.612415208 +0000 UTC m=+1206.665074535" watchObservedRunningTime="2026-09-18 14:04:17.614457572 +0000 UTC m=+1206.667116931"
Sep 18 14:04:26.339602 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:26.339561    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-ed4c53/distinct-2-w0-ed4c53-kserve-7ccfc9d644-hjn2r"
Sep 18 14:04:26.340037 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:26.339621    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ed4c53/distinct-2-w0-ed4c53-kserve-7ccfc9d644-hjn2r"
Sep 18 14:04:26.352383 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:26.352358    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-ed4c53/distinct-2-w0-ed4c53-kserve-7ccfc9d644-hjn2r"
Sep 18 14:04:26.659940 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:26.659854    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ed4c53/distinct-2-w0-ed4c53-kserve-7ccfc9d644-hjn2r"
Sep 18 14:04:27.265230 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:27.265199    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-bfd52075/e2e-lifecycle-model-bfd52075-kserve-7db78f579f-rsw47"]
Sep 18 14:04:27.267685 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:27.267669    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-bfd52075/e2e-lifecycle-model-bfd52075-kserve-7db78f579f-rsw47"
Sep 18 14:04:27.274760 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:27.274728    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-bfd52075\"/\"openshift-service-ca.crt\""
Sep 18 14:04:27.274955 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:27.274929    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-bfd52075\"/\"e2e-lifecycle-model-bfd52075-kserve-self-signed-certs\""
Sep 18 14:04:27.275143 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:27.274955    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-bfd52075\"/\"kube-root-ca.crt\""
Sep 18 14:04:27.275270 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:27.274794    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-bfd52075\"/\"default-dockercfg-brbbf\""
Sep 18 14:04:27.286745 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:27.286722    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-bfd52075/e2e-lifecycle-model-bfd52075-kserve-7db78f579f-rsw47"]
Sep 18 14:04:27.313134 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:27.313107    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:04:27.355176 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:27.355146    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng2bx\" (UniqueName: \"kubernetes.io/projected/e6342501-29e4-4f74-936a-94f796d7889f-kube-api-access-ng2bx\") pod \"e2e-lifecycle-model-bfd52075-kserve-7db78f579f-rsw47\" (UID: \"e6342501-29e4-4f74-936a-94f796d7889f\") " pod="ai-tenant-e2e-mt-bfd52075/e2e-lifecycle-model-bfd52075-kserve-7db78f579f-rsw47"
Sep 18 14:04:27.355556 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:27.355216    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6342501-29e4-4f74-936a-94f796d7889f-tmp-dir\") pod \"e2e-lifecycle-model-bfd52075-kserve-7db78f579f-rsw47\" (UID: \"e6342501-29e4-4f74-936a-94f796d7889f\") " pod="ai-tenant-e2e-mt-bfd52075/e2e-lifecycle-model-bfd52075-kserve-7db78f579f-rsw47"
Sep 18 14:04:27.355556 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:27.355265    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6342501-29e4-4f74-936a-94f796d7889f-tls-certs\") pod \"e2e-lifecycle-model-bfd52075-kserve-7db78f579f-rsw47\" (UID: \"e6342501-29e4-4f74-936a-94f796d7889f\") " pod="ai-tenant-e2e-mt-bfd52075/e2e-lifecycle-model-bfd52075-kserve-7db78f579f-rsw47"
Sep 18 14:04:27.355556 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:27.355338    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6342501-29e4-4f74-936a-94f796d7889f-model-cache\") pod \"e2e-lifecycle-model-bfd52075-kserve-7db78f579f-rsw47\" (UID: \"e6342501-29e4-4f74-936a-94f796d7889f\") " pod="ai-tenant-e2e-mt-bfd52075/e2e-lifecycle-model-bfd52075-kserve-7db78f579f-rsw47"
Sep 18 14:04:27.355556 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:27.355389    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6342501-29e4-4f74-936a-94f796d7889f-home\") pod \"e2e-lifecycle-model-bfd52075-kserve-7db78f579f-rsw47\" (UID: \"e6342501-29e4-4f74-936a-94f796d7889f\") " pod="ai-tenant-e2e-mt-bfd52075/e2e-lifecycle-model-bfd52075-kserve-7db78f579f-rsw47"
Sep 18 14:04:27.355556 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:27.355440    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6342501-29e4-4f74-936a-94f796d7889f-dshm\") pod \"e2e-lifecycle-model-bfd52075-kserve-7db78f579f-rsw47\" (UID: \"e6342501-29e4-4f74-936a-94f796d7889f\") " pod="ai-tenant-e2e-mt-bfd52075/e2e-lifecycle-model-bfd52075-kserve-7db78f579f-rsw47"
Sep 18 14:04:27.456449 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:27.456412    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6342501-29e4-4f74-936a-94f796d7889f-model-cache\") pod \"e2e-lifecycle-model-bfd52075-kserve-7db78f579f-rsw47\" (UID: \"e6342501-29e4-4f74-936a-94f796d7889f\") " pod="ai-tenant-e2e-mt-bfd52075/e2e-lifecycle-model-bfd52075-kserve-7db78f579f-rsw47"
Sep 18 14:04:27.456649 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:27.456473    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6342501-29e4-4f74-936a-94f796d7889f-home\") pod \"e2e-lifecycle-model-bfd52075-kserve-7db78f579f-rsw47\" (UID: \"e6342501-29e4-4f74-936a-94f796d7889f\") " pod="ai-tenant-e2e-mt-bfd52075/e2e-lifecycle-model-bfd52075-kserve-7db78f579f-rsw47"
Sep 18 14:04:27.456649 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:27.456500    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6342501-29e4-4f74-936a-94f796d7889f-dshm\") pod \"e2e-lifecycle-model-bfd52075-kserve-7db78f579f-rsw47\" (UID: \"e6342501-29e4-4f74-936a-94f796d7889f\") " pod="ai-tenant-e2e-mt-bfd52075/e2e-lifecycle-model-bfd52075-kserve-7db78f579f-rsw47"
Sep 18 14:04:27.456649 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:27.456521    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng2bx\" (UniqueName: \"kubernetes.io/projected/e6342501-29e4-4f74-936a-94f796d7889f-kube-api-access-ng2bx\") pod \"e2e-lifecycle-model-bfd52075-kserve-7db78f579f-rsw47\" (UID: \"e6342501-29e4-4f74-936a-94f796d7889f\") " pod="ai-tenant-e2e-mt-bfd52075/e2e-lifecycle-model-bfd52075-kserve-7db78f579f-rsw47"
Sep 18 14:04:27.456649 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:27.456551    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6342501-29e4-4f74-936a-94f796d7889f-tmp-dir\") pod \"e2e-lifecycle-model-bfd52075-kserve-7db78f579f-rsw47\" (UID: \"e6342501-29e4-4f74-936a-94f796d7889f\") " pod="ai-tenant-e2e-mt-bfd52075/e2e-lifecycle-model-bfd52075-kserve-7db78f579f-rsw47"
Sep 18 14:04:27.456649 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:27.456609    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6342501-29e4-4f74-936a-94f796d7889f-tls-certs\") pod \"e2e-lifecycle-model-bfd52075-kserve-7db78f579f-rsw47\" (UID: \"e6342501-29e4-4f74-936a-94f796d7889f\") " pod="ai-tenant-e2e-mt-bfd52075/e2e-lifecycle-model-bfd52075-kserve-7db78f579f-rsw47"
Sep 18 14:04:27.457041 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:27.457015    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6342501-29e4-4f74-936a-94f796d7889f-model-cache\") pod \"e2e-lifecycle-model-bfd52075-kserve-7db78f579f-rsw47\" (UID: \"e6342501-29e4-4f74-936a-94f796d7889f\") " pod="ai-tenant-e2e-mt-bfd52075/e2e-lifecycle-model-bfd52075-kserve-7db78f579f-rsw47"
Sep 18 14:04:27.457135 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:27.457057    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6342501-29e4-4f74-936a-94f796d7889f-home\") pod \"e2e-lifecycle-model-bfd52075-kserve-7db78f579f-rsw47\" (UID: \"e6342501-29e4-4f74-936a-94f796d7889f\") " pod="ai-tenant-e2e-mt-bfd52075/e2e-lifecycle-model-bfd52075-kserve-7db78f579f-rsw47"
Sep 18 14:04:27.457193 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:27.457081    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6342501-29e4-4f74-936a-94f796d7889f-tmp-dir\") pod \"e2e-lifecycle-model-bfd52075-kserve-7db78f579f-rsw47\" (UID: \"e6342501-29e4-4f74-936a-94f796d7889f\") " pod="ai-tenant-e2e-mt-bfd52075/e2e-lifecycle-model-bfd52075-kserve-7db78f579f-rsw47"
Sep 18 14:04:27.458977 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:27.458953    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6342501-29e4-4f74-936a-94f796d7889f-dshm\") pod \"e2e-lifecycle-model-bfd52075-kserve-7db78f579f-rsw47\" (UID: \"e6342501-29e4-4f74-936a-94f796d7889f\") " pod="ai-tenant-e2e-mt-bfd52075/e2e-lifecycle-model-bfd52075-kserve-7db78f579f-rsw47"
Sep 18 14:04:27.459347 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:27.459325    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6342501-29e4-4f74-936a-94f796d7889f-tls-certs\") pod \"e2e-lifecycle-model-bfd52075-kserve-7db78f579f-rsw47\" (UID: \"e6342501-29e4-4f74-936a-94f796d7889f\") " pod="ai-tenant-e2e-mt-bfd52075/e2e-lifecycle-model-bfd52075-kserve-7db78f579f-rsw47"
Sep 18 14:04:27.465031 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:27.465006    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng2bx\" (UniqueName: \"kubernetes.io/projected/e6342501-29e4-4f74-936a-94f796d7889f-kube-api-access-ng2bx\") pod \"e2e-lifecycle-model-bfd52075-kserve-7db78f579f-rsw47\" (UID: \"e6342501-29e4-4f74-936a-94f796d7889f\") " pod="ai-tenant-e2e-mt-bfd52075/e2e-lifecycle-model-bfd52075-kserve-7db78f579f-rsw47"
Sep 18 14:04:27.582403 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:27.582304    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-bfd52075/e2e-lifecycle-model-bfd52075-kserve-7db78f579f-rsw47"
Sep 18 14:04:27.857734 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:27.857703    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-bfd52075/e2e-lifecycle-model-bfd52075-kserve-7db78f579f-rsw47"]
Sep 18 14:04:27.858903 ip-10-0-141-222 kubenswrapper[2577]: W0918 14:04:27.858869    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode6342501_29e4_4f74_936a_94f796d7889f.slice/crio-32ac2b6a3cda7c055ca76f5c932319f371baf96c57efade284462b35b2f991f9 WatchSource:0}: Error finding container 32ac2b6a3cda7c055ca76f5c932319f371baf96c57efade284462b35b2f991f9: Status 404 returned error can't find the container with id 32ac2b6a3cda7c055ca76f5c932319f371baf96c57efade284462b35b2f991f9
Sep 18 14:04:28.364591 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:28.364558    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:04:28.438601 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:28.438567    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:04:28.664667 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:28.664580    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-bfd52075/e2e-lifecycle-model-bfd52075-kserve-7db78f579f-rsw47" event={"ID":"e6342501-29e4-4f74-936a-94f796d7889f","Type":"ContainerStarted","Data":"70f28e19d7360265ac42c004aae3decf859232a2d54033b88aa5eca71c0541c4"}
Sep 18 14:04:28.664667 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:28.664627    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-bfd52075/e2e-lifecycle-model-bfd52075-kserve-7db78f579f-rsw47" event={"ID":"e6342501-29e4-4f74-936a-94f796d7889f","Type":"ContainerStarted","Data":"32ac2b6a3cda7c055ca76f5c932319f371baf96c57efade284462b35b2f991f9"}
Sep 18 14:04:28.685387 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:28.685340    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-bfd52075/e2e-lifecycle-model-bfd52075-kserve-7db78f579f-rsw47" podStartSLOduration=1.6853247630000001 podStartE2EDuration="1.685324763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:04:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:04:28.684504293 +0000 UTC m=+1217.737163621" watchObservedRunningTime="2026-09-18 14:04:28.685324763 +0000 UTC m=+1217.737984085"
Sep 18 14:04:32.499897 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:32.499854    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 14:04:37.583301 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:37.583262    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-bfd52075/e2e-lifecycle-model-bfd52075-kserve-7db78f579f-rsw47"
Sep 18 14:04:37.583712 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:37.583314    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-bfd52075/e2e-lifecycle-model-bfd52075-kserve-7db78f579f-rsw47"
Sep 18 14:04:37.595964 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:37.595937    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-bfd52075/e2e-lifecycle-model-bfd52075-kserve-7db78f579f-rsw47"
Sep 18 14:04:37.738109 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:37.738078    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-bfd52075/e2e-lifecycle-model-bfd52075-kserve-7db78f579f-rsw47"
Sep 18 14:04:47.998893 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:47.998858    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:04:58.309472 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:04:58.309430    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 14:05:05.987818 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:05.987765    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:05:06.970789 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:06.970754    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:05:13.279565 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:13.279531    2577 scope.go:117] "RemoveContainer" containerID="6c4315fc1c2b0cb8f6fd8e80af0206e589f729002e5857938145177719e520e6"
Sep 18 14:05:16.735767 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:16.735733    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-377ca2/e2e-conflict-b-d79684ca-kserve-647977bc69-k766t"]
Sep 18 14:05:16.751594 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:16.751564    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:05:18.485745 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:18.485708    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:05:18.586654 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:18.586622    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:05:27.861583 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:27.861541    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:05:31.755020 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:31.754983    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-377ca2/e2e-conflict-b-d79684ca-kserve-647977bc69-k766t" podUID="b41c0fbf-f3d1-483d-b66e-7b462a330097" containerName="main" containerID="cri-o://0ff487597f6efd4a2fccf83d8e63d73d3cdfc5eecc17ca654902fe32f8afdbd0" gracePeriod=45
Sep 18 14:05:32.008038 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:32.007977    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-377ca2/e2e-conflict-b-d79684ca-kserve-647977bc69-k766t"
Sep 18 14:05:32.043561 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:32.043525    2577 generic.go:356] "Generic (PLEG): container finished" podID="b41c0fbf-f3d1-483d-b66e-7b462a330097" containerID="0ff487597f6efd4a2fccf83d8e63d73d3cdfc5eecc17ca654902fe32f8afdbd0" exitCode=0
Sep 18 14:05:32.043736 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:32.043565    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-377ca2/e2e-conflict-b-d79684ca-kserve-647977bc69-k766t"
Sep 18 14:05:32.043736 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:32.043608    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-377ca2/e2e-conflict-b-d79684ca-kserve-647977bc69-k766t" event={"ID":"b41c0fbf-f3d1-483d-b66e-7b462a330097","Type":"ContainerDied","Data":"0ff487597f6efd4a2fccf83d8e63d73d3cdfc5eecc17ca654902fe32f8afdbd0"}
Sep 18 14:05:32.043736 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:32.043650    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-377ca2/e2e-conflict-b-d79684ca-kserve-647977bc69-k766t" event={"ID":"b41c0fbf-f3d1-483d-b66e-7b462a330097","Type":"ContainerDied","Data":"534b8a16004a7348e25632f7ba03d9b82a1c8a96492cab420495f6b4641e10c6"}
Sep 18 14:05:32.043736 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:32.043669    2577 scope.go:117] "RemoveContainer" containerID="0ff487597f6efd4a2fccf83d8e63d73d3cdfc5eecc17ca654902fe32f8afdbd0"
Sep 18 14:05:32.053056 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:32.053038    2577 scope.go:117] "RemoveContainer" containerID="0ff487597f6efd4a2fccf83d8e63d73d3cdfc5eecc17ca654902fe32f8afdbd0"
Sep 18 14:05:32.053349 ip-10-0-141-222 kubenswrapper[2577]: E0918 14:05:32.053322    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ff487597f6efd4a2fccf83d8e63d73d3cdfc5eecc17ca654902fe32f8afdbd0\": container with ID starting with 0ff487597f6efd4a2fccf83d8e63d73d3cdfc5eecc17ca654902fe32f8afdbd0 not found: ID does not exist" containerID="0ff487597f6efd4a2fccf83d8e63d73d3cdfc5eecc17ca654902fe32f8afdbd0"
Sep 18 14:05:32.053454 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:32.053357    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ff487597f6efd4a2fccf83d8e63d73d3cdfc5eecc17ca654902fe32f8afdbd0"} err="failed to get container status \"0ff487597f6efd4a2fccf83d8e63d73d3cdfc5eecc17ca654902fe32f8afdbd0\": rpc error: code = NotFound desc = could not find container \"0ff487597f6efd4a2fccf83d8e63d73d3cdfc5eecc17ca654902fe32f8afdbd0\": container with ID starting with 0ff487597f6efd4a2fccf83d8e63d73d3cdfc5eecc17ca654902fe32f8afdbd0 not found: ID does not exist"
Sep 18 14:05:32.084729 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:32.084695    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b41c0fbf-f3d1-483d-b66e-7b462a330097-tmp-dir\") pod \"b41c0fbf-f3d1-483d-b66e-7b462a330097\" (UID: \"b41c0fbf-f3d1-483d-b66e-7b462a330097\") "
Sep 18 14:05:32.084962 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:32.084751    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b41c0fbf-f3d1-483d-b66e-7b462a330097-home\") pod \"b41c0fbf-f3d1-483d-b66e-7b462a330097\" (UID: \"b41c0fbf-f3d1-483d-b66e-7b462a330097\") "
Sep 18 14:05:32.084962 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:32.084778    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b41c0fbf-f3d1-483d-b66e-7b462a330097-model-cache\") pod \"b41c0fbf-f3d1-483d-b66e-7b462a330097\" (UID: \"b41c0fbf-f3d1-483d-b66e-7b462a330097\") "
Sep 18 14:05:32.084962 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:32.084862    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b41c0fbf-f3d1-483d-b66e-7b462a330097-dshm\") pod \"b41c0fbf-f3d1-483d-b66e-7b462a330097\" (UID: \"b41c0fbf-f3d1-483d-b66e-7b462a330097\") "
Sep 18 14:05:32.084962 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:32.084921    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l9stj\" (UniqueName: \"kubernetes.io/projected/b41c0fbf-f3d1-483d-b66e-7b462a330097-kube-api-access-l9stj\") pod \"b41c0fbf-f3d1-483d-b66e-7b462a330097\" (UID: \"b41c0fbf-f3d1-483d-b66e-7b462a330097\") "
Sep 18 14:05:32.085195 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:32.084982    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b41c0fbf-f3d1-483d-b66e-7b462a330097-tls-certs\") pod \"b41c0fbf-f3d1-483d-b66e-7b462a330097\" (UID: \"b41c0fbf-f3d1-483d-b66e-7b462a330097\") "
Sep 18 14:05:32.085195 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:32.085031    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b41c0fbf-f3d1-483d-b66e-7b462a330097-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b41c0fbf-f3d1-483d-b66e-7b462a330097" (UID: "b41c0fbf-f3d1-483d-b66e-7b462a330097"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:05:32.085195 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:32.085091    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b41c0fbf-f3d1-483d-b66e-7b462a330097-home" (OuterVolumeSpecName: "home") pod "b41c0fbf-f3d1-483d-b66e-7b462a330097" (UID: "b41c0fbf-f3d1-483d-b66e-7b462a330097"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:05:32.085195 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:32.085102    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b41c0fbf-f3d1-483d-b66e-7b462a330097-model-cache" (OuterVolumeSpecName: "model-cache") pod "b41c0fbf-f3d1-483d-b66e-7b462a330097" (UID: "b41c0fbf-f3d1-483d-b66e-7b462a330097"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:05:32.085416 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:32.085310    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b41c0fbf-f3d1-483d-b66e-7b462a330097-tmp-dir\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:05:32.085416 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:32.085323    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b41c0fbf-f3d1-483d-b66e-7b462a330097-home\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:05:32.085416 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:32.085331    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b41c0fbf-f3d1-483d-b66e-7b462a330097-model-cache\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:05:32.087246 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:32.087216    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b41c0fbf-f3d1-483d-b66e-7b462a330097-dshm" (OuterVolumeSpecName: "dshm") pod "b41c0fbf-f3d1-483d-b66e-7b462a330097" (UID: "b41c0fbf-f3d1-483d-b66e-7b462a330097"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:05:32.087246 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:32.087232    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b41c0fbf-f3d1-483d-b66e-7b462a330097-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b41c0fbf-f3d1-483d-b66e-7b462a330097" (UID: "b41c0fbf-f3d1-483d-b66e-7b462a330097"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:05:32.087399 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:32.087252    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b41c0fbf-f3d1-483d-b66e-7b462a330097-kube-api-access-l9stj" (OuterVolumeSpecName: "kube-api-access-l9stj") pod "b41c0fbf-f3d1-483d-b66e-7b462a330097" (UID: "b41c0fbf-f3d1-483d-b66e-7b462a330097"). InnerVolumeSpecName "kube-api-access-l9stj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:05:32.185950 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:32.185918    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b41c0fbf-f3d1-483d-b66e-7b462a330097-tls-certs\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:05:32.185950 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:32.185943    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b41c0fbf-f3d1-483d-b66e-7b462a330097-dshm\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:05:32.185950 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:32.185952    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9stj\" (UniqueName: \"kubernetes.io/projected/b41c0fbf-f3d1-483d-b66e-7b462a330097-kube-api-access-l9stj\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:05:32.246984 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:32.246948    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:05:32.366748 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:32.366716    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-377ca2/e2e-conflict-b-d79684ca-kserve-647977bc69-k766t"]
Sep 18 14:05:32.369986 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:32.369961    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-377ca2/e2e-conflict-b-d79684ca-kserve-647977bc69-k766t"]
Sep 18 14:05:33.114398 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:33.114363    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b41c0fbf-f3d1-483d-b66e-7b462a330097" path="/var/lib/kubelet/pods/b41c0fbf-f3d1-483d-b66e-7b462a330097/volumes"
Sep 18 14:05:39.109106 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:39.109075    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 14:05:40.228090 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:40.228054    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:05:45.684008 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:45.683966    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:05:46.435093 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:46.435055    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:05:47.189561 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:47.189528    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:05:47.856661 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:47.856629    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:05:48.546347 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:48.546310    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:05:49.206427 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:49.206391    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:05:49.949960 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:49.949929    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:05:50.619788 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:50.619755    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:05:51.423322 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:51.423285    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:05:57.697601 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:57.697567    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-377ca2/distinct-w4-377ca2-kserve-5f8574bffb-lhk8c"]
Sep 18 14:05:57.697988 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:57.697974    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b41c0fbf-f3d1-483d-b66e-7b462a330097" containerName="main"
Sep 18 14:05:57.698035 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:57.697989    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b41c0fbf-f3d1-483d-b66e-7b462a330097" containerName="main"
Sep 18 14:05:57.698074 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:57.698066    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b41c0fbf-f3d1-483d-b66e-7b462a330097" containerName="main"
Sep 18 14:05:57.700702 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:57.700682    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-377ca2/distinct-w4-377ca2-kserve-5f8574bffb-lhk8c"
Sep 18 14:05:57.703109 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:57.703086    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-377ca2\"/\"distinct-w4-377ca2-kserve-self-signed-certs\""
Sep 18 14:05:57.713219 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:57.713194    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-377ca2/distinct-w4-377ca2-kserve-5f8574bffb-lhk8c"]
Sep 18 14:05:57.731606 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:57.731577    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt6l2\" (UniqueName: \"kubernetes.io/projected/a58f638f-72bd-4d81-bdba-5b58e146ae90-kube-api-access-lt6l2\") pod \"distinct-w4-377ca2-kserve-5f8574bffb-lhk8c\" (UID: \"a58f638f-72bd-4d81-bdba-5b58e146ae90\") " pod="e2e-models-e2e-worker-w4-377ca2/distinct-w4-377ca2-kserve-5f8574bffb-lhk8c"
Sep 18 14:05:57.731758 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:57.731627    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a58f638f-72bd-4d81-bdba-5b58e146ae90-tmp-dir\") pod \"distinct-w4-377ca2-kserve-5f8574bffb-lhk8c\" (UID: \"a58f638f-72bd-4d81-bdba-5b58e146ae90\") " pod="e2e-models-e2e-worker-w4-377ca2/distinct-w4-377ca2-kserve-5f8574bffb-lhk8c"
Sep 18 14:05:57.731833 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:57.731751    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a58f638f-72bd-4d81-bdba-5b58e146ae90-home\") pod \"distinct-w4-377ca2-kserve-5f8574bffb-lhk8c\" (UID: \"a58f638f-72bd-4d81-bdba-5b58e146ae90\") " pod="e2e-models-e2e-worker-w4-377ca2/distinct-w4-377ca2-kserve-5f8574bffb-lhk8c"
Sep 18 14:05:57.731833 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:57.731787    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a58f638f-72bd-4d81-bdba-5b58e146ae90-tls-certs\") pod \"distinct-w4-377ca2-kserve-5f8574bffb-lhk8c\" (UID: \"a58f638f-72bd-4d81-bdba-5b58e146ae90\") " pod="e2e-models-e2e-worker-w4-377ca2/distinct-w4-377ca2-kserve-5f8574bffb-lhk8c"
Sep 18 14:05:57.731944 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:57.731869    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a58f638f-72bd-4d81-bdba-5b58e146ae90-model-cache\") pod \"distinct-w4-377ca2-kserve-5f8574bffb-lhk8c\" (UID: \"a58f638f-72bd-4d81-bdba-5b58e146ae90\") " pod="e2e-models-e2e-worker-w4-377ca2/distinct-w4-377ca2-kserve-5f8574bffb-lhk8c"
Sep 18 14:05:57.731944 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:57.731901    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a58f638f-72bd-4d81-bdba-5b58e146ae90-dshm\") pod \"distinct-w4-377ca2-kserve-5f8574bffb-lhk8c\" (UID: \"a58f638f-72bd-4d81-bdba-5b58e146ae90\") " pod="e2e-models-e2e-worker-w4-377ca2/distinct-w4-377ca2-kserve-5f8574bffb-lhk8c"
Sep 18 14:05:57.748195 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:57.748165    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:05:57.832381 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:57.832349    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a58f638f-72bd-4d81-bdba-5b58e146ae90-dshm\") pod \"distinct-w4-377ca2-kserve-5f8574bffb-lhk8c\" (UID: \"a58f638f-72bd-4d81-bdba-5b58e146ae90\") " pod="e2e-models-e2e-worker-w4-377ca2/distinct-w4-377ca2-kserve-5f8574bffb-lhk8c"
Sep 18 14:05:57.832551 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:57.832407    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt6l2\" (UniqueName: \"kubernetes.io/projected/a58f638f-72bd-4d81-bdba-5b58e146ae90-kube-api-access-lt6l2\") pod \"distinct-w4-377ca2-kserve-5f8574bffb-lhk8c\" (UID: \"a58f638f-72bd-4d81-bdba-5b58e146ae90\") " pod="e2e-models-e2e-worker-w4-377ca2/distinct-w4-377ca2-kserve-5f8574bffb-lhk8c"
Sep 18 14:05:57.832551 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:57.832438    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a58f638f-72bd-4d81-bdba-5b58e146ae90-tmp-dir\") pod \"distinct-w4-377ca2-kserve-5f8574bffb-lhk8c\" (UID: \"a58f638f-72bd-4d81-bdba-5b58e146ae90\") " pod="e2e-models-e2e-worker-w4-377ca2/distinct-w4-377ca2-kserve-5f8574bffb-lhk8c"
Sep 18 14:05:57.832551 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:57.832476    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a58f638f-72bd-4d81-bdba-5b58e146ae90-home\") pod \"distinct-w4-377ca2-kserve-5f8574bffb-lhk8c\" (UID: \"a58f638f-72bd-4d81-bdba-5b58e146ae90\") " pod="e2e-models-e2e-worker-w4-377ca2/distinct-w4-377ca2-kserve-5f8574bffb-lhk8c"
Sep 18 14:05:57.832551 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:57.832491    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a58f638f-72bd-4d81-bdba-5b58e146ae90-tls-certs\") pod \"distinct-w4-377ca2-kserve-5f8574bffb-lhk8c\" (UID: \"a58f638f-72bd-4d81-bdba-5b58e146ae90\") " pod="e2e-models-e2e-worker-w4-377ca2/distinct-w4-377ca2-kserve-5f8574bffb-lhk8c"
Sep 18 14:05:57.832551 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:57.832511    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a58f638f-72bd-4d81-bdba-5b58e146ae90-model-cache\") pod \"distinct-w4-377ca2-kserve-5f8574bffb-lhk8c\" (UID: \"a58f638f-72bd-4d81-bdba-5b58e146ae90\") " pod="e2e-models-e2e-worker-w4-377ca2/distinct-w4-377ca2-kserve-5f8574bffb-lhk8c"
Sep 18 14:05:57.832898 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:57.832874    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a58f638f-72bd-4d81-bdba-5b58e146ae90-tmp-dir\") pod \"distinct-w4-377ca2-kserve-5f8574bffb-lhk8c\" (UID: \"a58f638f-72bd-4d81-bdba-5b58e146ae90\") " pod="e2e-models-e2e-worker-w4-377ca2/distinct-w4-377ca2-kserve-5f8574bffb-lhk8c"
Sep 18 14:05:57.832973 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:57.832935    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a58f638f-72bd-4d81-bdba-5b58e146ae90-home\") pod \"distinct-w4-377ca2-kserve-5f8574bffb-lhk8c\" (UID: \"a58f638f-72bd-4d81-bdba-5b58e146ae90\") " pod="e2e-models-e2e-worker-w4-377ca2/distinct-w4-377ca2-kserve-5f8574bffb-lhk8c"
Sep 18 14:05:57.832973 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:57.832957    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a58f638f-72bd-4d81-bdba-5b58e146ae90-model-cache\") pod \"distinct-w4-377ca2-kserve-5f8574bffb-lhk8c\" (UID: \"a58f638f-72bd-4d81-bdba-5b58e146ae90\") " pod="e2e-models-e2e-worker-w4-377ca2/distinct-w4-377ca2-kserve-5f8574bffb-lhk8c"
Sep 18 14:05:57.834857 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:57.834803    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a58f638f-72bd-4d81-bdba-5b58e146ae90-dshm\") pod \"distinct-w4-377ca2-kserve-5f8574bffb-lhk8c\" (UID: \"a58f638f-72bd-4d81-bdba-5b58e146ae90\") " pod="e2e-models-e2e-worker-w4-377ca2/distinct-w4-377ca2-kserve-5f8574bffb-lhk8c"
Sep 18 14:05:57.835102 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:57.835082    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a58f638f-72bd-4d81-bdba-5b58e146ae90-tls-certs\") pod \"distinct-w4-377ca2-kserve-5f8574bffb-lhk8c\" (UID: \"a58f638f-72bd-4d81-bdba-5b58e146ae90\") " pod="e2e-models-e2e-worker-w4-377ca2/distinct-w4-377ca2-kserve-5f8574bffb-lhk8c"
Sep 18 14:05:57.840676 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:57.840654    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt6l2\" (UniqueName: \"kubernetes.io/projected/a58f638f-72bd-4d81-bdba-5b58e146ae90-kube-api-access-lt6l2\") pod \"distinct-w4-377ca2-kserve-5f8574bffb-lhk8c\" (UID: \"a58f638f-72bd-4d81-bdba-5b58e146ae90\") " pod="e2e-models-e2e-worker-w4-377ca2/distinct-w4-377ca2-kserve-5f8574bffb-lhk8c"
Sep 18 14:05:58.010182 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:58.010079    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-377ca2/distinct-w4-377ca2-kserve-5f8574bffb-lhk8c"
Sep 18 14:05:58.284960 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:58.284933    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-377ca2/distinct-w4-377ca2-kserve-5f8574bffb-lhk8c"]
Sep 18 14:05:58.286553 ip-10-0-141-222 kubenswrapper[2577]: W0918 14:05:58.286520    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda58f638f_72bd_4d81_bdba_5b58e146ae90.slice/crio-a5138d9ba5458552f482269e49c0c62baac0fd457989531a8fb7b786842d3fc0 WatchSource:0}: Error finding container a5138d9ba5458552f482269e49c0c62baac0fd457989531a8fb7b786842d3fc0: Status 404 returned error can't find the container with id a5138d9ba5458552f482269e49c0c62baac0fd457989531a8fb7b786842d3fc0
Sep 18 14:05:59.200536 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:59.200504    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-377ca2/distinct-w4-377ca2-kserve-5f8574bffb-lhk8c" event={"ID":"a58f638f-72bd-4d81-bdba-5b58e146ae90","Type":"ContainerStarted","Data":"e0d2362d7c83ad74f0d67972bea11cb73e8ccd3f723d9d9b43b0b290f236fd84"}
Sep 18 14:05:59.200958 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:59.200547    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-377ca2/distinct-w4-377ca2-kserve-5f8574bffb-lhk8c" event={"ID":"a58f638f-72bd-4d81-bdba-5b58e146ae90","Type":"ContainerStarted","Data":"a5138d9ba5458552f482269e49c0c62baac0fd457989531a8fb7b786842d3fc0"}
Sep 18 14:05:59.221868 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:05:59.221798    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-377ca2/distinct-w4-377ca2-kserve-5f8574bffb-lhk8c" podStartSLOduration=2.221782214 podStartE2EDuration="2.221782214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:05:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:05:59.219900543 +0000 UTC m=+1308.272559869" watchObservedRunningTime="2026-09-18 14:05:59.221782214 +0000 UTC m=+1308.274441591"
Sep 18 14:06:00.894922 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:06:00.894891    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:06:01.022851 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:06:01.022428    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:06:01.158029 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:06:01.157931    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:06:01.266337 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:06:01.266298    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:06:01.367047 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:06:01.367013    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:06:01.656145 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:06:01.656110    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:06:02.640335 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:06:02.640298    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:06:03.487944 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:06:03.487911    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:06:04.200647 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:06:04.200603    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:06:04.489323 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:06:04.489221    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:06:04.939493 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:06:04.939456    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:06:05.725100 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:06:05.725061    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:06:06.535725 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:06:06.535685    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:06:08.010748 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:06:08.010704    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-377ca2/distinct-w4-377ca2-kserve-5f8574bffb-lhk8c"
Sep 18 14:06:08.011190 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:06:08.010774    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-377ca2/distinct-w4-377ca2-kserve-5f8574bffb-lhk8c"
Sep 18 14:06:08.023604 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:06:08.023573    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-377ca2/distinct-w4-377ca2-kserve-5f8574bffb-lhk8c"
Sep 18 14:06:08.264537 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:06:08.264460    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-377ca2/distinct-w4-377ca2-kserve-5f8574bffb-lhk8c"
Sep 18 14:06:09.470406 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:06:09.470373    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-bfd52075/e2e-lifecycle-model-bfd52075-kserve-7db78f579f-rsw47"]
Sep 18 14:06:09.662096 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:06:09.662059    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:06:13.301447 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:06:13.301411    2577 scope.go:117] "RemoveContainer" containerID="4e04e952b9e9b27d6cccb0e1693e6c5f1b45ebe3cb0959ec95bdba6b37b18177"
Sep 18 14:06:16.914440 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:06:16.914396    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 14:06:24.490776 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:06:24.490717    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-bfd52075/e2e-lifecycle-model-bfd52075-kserve-7db78f579f-rsw47" podUID="e6342501-29e4-4f74-936a-94f796d7889f" containerName="main" containerID="cri-o://70f28e19d7360265ac42c004aae3decf859232a2d54033b88aa5eca71c0541c4" gracePeriod=45
Sep 18 14:06:24.745438 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:06:24.745378    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-bfd52075/e2e-lifecycle-model-bfd52075-kserve-7db78f579f-rsw47"
Sep 18 14:06:24.790717 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:06:24.790684    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6342501-29e4-4f74-936a-94f796d7889f-dshm\") pod \"e6342501-29e4-4f74-936a-94f796d7889f\" (UID: \"e6342501-29e4-4f74-936a-94f796d7889f\") "
Sep 18 14:06:24.790717 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:06:24.790717    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6342501-29e4-4f74-936a-94f796d7889f-home\") pod \"e6342501-29e4-4f74-936a-94f796d7889f\" (UID: \"e6342501-29e4-4f74-936a-94f796d7889f\") "
Sep 18 14:06:24.790980 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:06:24.790780    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6342501-29e4-4f74-936a-94f796d7889f-tls-certs\") pod \"e6342501-29e4-4f74-936a-94f796d7889f\" (UID: \"e6342501-29e4-4f74-936a-94f796d7889f\") "
Sep 18 14:06:24.790980 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:06:24.790808    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6342501-29e4-4f74-936a-94f796d7889f-model-cache\") pod \"e6342501-29e4-4f74-936a-94f796d7889f\" (UID: \"e6342501-29e4-4f74-936a-94f796d7889f\") "
Sep 18 14:06:24.790980 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:06:24.790860    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6342501-29e4-4f74-936a-94f796d7889f-tmp-dir\") pod \"e6342501-29e4-4f74-936a-94f796d7889f\" (UID: \"e6342501-29e4-4f74-936a-94f796d7889f\") "
Sep 18 14:06:24.791179 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:06:24.791149    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ng2bx\" (UniqueName: \"kubernetes.io/projected/e6342501-29e4-4f74-936a-94f796d7889f-kube-api-access-ng2bx\") pod \"e6342501-29e4-4f74-936a-94f796d7889f\" (UID: \"e6342501-29e4-4f74-936a-94f796d7889f\") "
Sep 18 14:06:24.791602 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:06:24.791313    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6342501-29e4-4f74-936a-94f796d7889f-home" (OuterVolumeSpecName: "home") pod "e6342501-29e4-4f74-936a-94f796d7889f" (UID: "e6342501-29e4-4f74-936a-94f796d7889f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:06:24.791602 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:06:24.791320    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6342501-29e4-4f74-936a-94f796d7889f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e6342501-29e4-4f74-936a-94f796d7889f" (UID: "e6342501-29e4-4f74-936a-94f796d7889f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:06:24.791602 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:06:24.791320    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6342501-29e4-4f74-936a-94f796d7889f-model-cache" (OuterVolumeSpecName: "model-cache") pod "e6342501-29e4-4f74-936a-94f796d7889f" (UID: "e6342501-29e4-4f74-936a-94f796d7889f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:06:24.792107 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:06:24.791780    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6342501-29e4-4f74-936a-94f796d7889f-model-cache\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:06:24.792107 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:06:24.791848    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6342501-29e4-4f74-936a-94f796d7889f-tmp-dir\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:06:24.792107 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:06:24.791864    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6342501-29e4-4f74-936a-94f796d7889f-home\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:06:24.793729 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:06:24.793678    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6342501-29e4-4f74-936a-94f796d7889f-dshm" (OuterVolumeSpecName: "dshm") pod "e6342501-29e4-4f74-936a-94f796d7889f" (UID: "e6342501-29e4-4f74-936a-94f796d7889f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:06:24.793729 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:06:24.793682    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6342501-29e4-4f74-936a-94f796d7889f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e6342501-29e4-4f74-936a-94f796d7889f" (UID: "e6342501-29e4-4f74-936a-94f796d7889f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:06:24.793729 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:06:24.793705    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6342501-29e4-4f74-936a-94f796d7889f-kube-api-access-ng2bx" (OuterVolumeSpecName: "kube-api-access-ng2bx") pod "e6342501-29e4-4f74-936a-94f796d7889f" (UID: "e6342501-29e4-4f74-936a-94f796d7889f"). InnerVolumeSpecName "kube-api-access-ng2bx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:06:24.893382 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:06:24.893350    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6342501-29e4-4f74-936a-94f796d7889f-tls-certs\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:06:24.893382 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:06:24.893378    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ng2bx\" (UniqueName: \"kubernetes.io/projected/e6342501-29e4-4f74-936a-94f796d7889f-kube-api-access-ng2bx\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:06:24.893382 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:06:24.893387    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6342501-29e4-4f74-936a-94f796d7889f-dshm\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:06:25.350171 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:06:25.350133    2577 generic.go:356] "Generic (PLEG): container finished" podID="e6342501-29e4-4f74-936a-94f796d7889f" containerID="70f28e19d7360265ac42c004aae3decf859232a2d54033b88aa5eca71c0541c4" exitCode=0
Sep 18 14:06:25.350356 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:06:25.350204    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-bfd52075/e2e-lifecycle-model-bfd52075-kserve-7db78f579f-rsw47"
Sep 18 14:06:25.350356 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:06:25.350214    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-bfd52075/e2e-lifecycle-model-bfd52075-kserve-7db78f579f-rsw47" event={"ID":"e6342501-29e4-4f74-936a-94f796d7889f","Type":"ContainerDied","Data":"70f28e19d7360265ac42c004aae3decf859232a2d54033b88aa5eca71c0541c4"}
Sep 18 14:06:25.350356 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:06:25.350256    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-bfd52075/e2e-lifecycle-model-bfd52075-kserve-7db78f579f-rsw47" event={"ID":"e6342501-29e4-4f74-936a-94f796d7889f","Type":"ContainerDied","Data":"32ac2b6a3cda7c055ca76f5c932319f371baf96c57efade284462b35b2f991f9"}
Sep 18 14:06:25.350356 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:06:25.350272    2577 scope.go:117] "RemoveContainer" containerID="70f28e19d7360265ac42c004aae3decf859232a2d54033b88aa5eca71c0541c4"
Sep 18 14:06:25.359719 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:06:25.359698    2577 scope.go:117] "RemoveContainer" containerID="70f28e19d7360265ac42c004aae3decf859232a2d54033b88aa5eca71c0541c4"
Sep 18 14:06:25.360028 ip-10-0-141-222 kubenswrapper[2577]: E0918 14:06:25.360004    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70f28e19d7360265ac42c004aae3decf859232a2d54033b88aa5eca71c0541c4\": container with ID starting with 70f28e19d7360265ac42c004aae3decf859232a2d54033b88aa5eca71c0541c4 not found: ID does not exist" containerID="70f28e19d7360265ac42c004aae3decf859232a2d54033b88aa5eca71c0541c4"
Sep 18 14:06:25.360116 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:06:25.360036    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70f28e19d7360265ac42c004aae3decf859232a2d54033b88aa5eca71c0541c4"} err="failed to get container status \"70f28e19d7360265ac42c004aae3decf859232a2d54033b88aa5eca71c0541c4\": rpc error: code = NotFound desc = could not find container \"70f28e19d7360265ac42c004aae3decf859232a2d54033b88aa5eca71c0541c4\": container with ID starting with 70f28e19d7360265ac42c004aae3decf859232a2d54033b88aa5eca71c0541c4 not found: ID does not exist"
Sep 18 14:06:25.366220 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:06:25.366170    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-bfd52075/e2e-lifecycle-model-bfd52075-kserve-7db78f579f-rsw47"]
Sep 18 14:06:25.371780 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:06:25.371761    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-bfd52075/e2e-lifecycle-model-bfd52075-kserve-7db78f579f-rsw47"]
Sep 18 14:06:27.109509 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:06:27.109476    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6342501-29e4-4f74-936a-94f796d7889f" path="/var/lib/kubelet/pods/e6342501-29e4-4f74-936a-94f796d7889f/volumes"
Sep 18 14:06:39.056894 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:06:39.056862    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:06:39.643785 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:06:39.643748    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:06:46.079264 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:06:46.079219    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:06:47.022457 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:06:47.022425    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:06:48.092319 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:06:48.092282    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:06:52.855212 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:06:52.855175    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:06:59.416230 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:06:59.416194    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 14:07:06.420832 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:07:06.420787    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:07:16.697924 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:07:16.697886    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:07:26.093758 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:07:26.093723    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:07:26.157812 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:07:26.157775    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:07:26.312161 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:07:26.312126    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 14:07:26.404789 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:07:26.404706    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:07:27.613509 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:07:27.613470    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:07:27.709703 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:07:27.709667    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:07:27.811185 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:07:27.811148    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:07:28.419942 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:07:28.419908    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:07:28.499397 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:07:28.499365    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:07:29.844166 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:07:29.844133    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:07:35.876559 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:07:35.876518    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:07:35.970423 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:07:35.970389    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:07:36.541360 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:07:36.541318    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:07:41.243302 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:07:41.243268    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:07:43.172227 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:07:43.172197    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:07:43.281246 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:07:43.281210    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:07:43.386205 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:07:43.386173    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:07:53.906039 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:07:53.906006    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:07:54.120925 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:07:54.120885    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:07:57.598090 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:07:57.598049    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 14:08:05.754252 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:05.754216    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:08:06.034303 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:06.034211    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:08:06.723681 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:06.723646    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:08:07.401391 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:07.401360    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:08:08.150385 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:08.150349    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:08:08.812357 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:08.812315    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:08:09.524034 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:09.523988    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:08:10.427537 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:10.427502    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:08:10.742022 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:10.741937    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ed4c53/distinct-2-w0-ed4c53-kserve-7ccfc9d644-hjn2r"]
Sep 18 14:08:10.764164 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:10.764134    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ed4c53/premium-w0-ed4c53-kserve-565c6b654d-mcfqz"]
Sep 18 14:08:10.776787 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:10.776758    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ed4c53/simulator-w0-ed4c53-kserve-989f79bd5-z4lwd"]
Sep 18 14:08:10.867512 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:10.867476    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:08:11.193475 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:11.193441    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:08:15.074843 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:15.074794    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:08:21.074190 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:21.074153    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:08:21.144116 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:21.144085    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:08:21.421525 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:21.421496    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:08:21.519404 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:21.519369    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:08:21.562045 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:21.562013    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ed4c53-59f4997598-7jfng"]
Sep 18 14:08:21.562246 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:21.562216    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ed4c53-59f4997598-7jfng" podUID="9d4b52c6-fb44-431d-adff-446bbd7860ea" containerName="payload-pre-processing" containerID="cri-o://0f9eec2be2f35acc1aa783d33c02ce1706763410d406cf88a87b7d32d54c4ffe" gracePeriod=30
Sep 18 14:08:21.590499 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:21.590471    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:08:21.691063 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:21.691030    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:08:21.778692 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:21.778657    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:08:21.826805 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:21.826771    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ed4c53-59f4997598-7jfng"
Sep 18 14:08:21.839858 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:21.839811    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:08:21.921272 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:21.921231    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8psgl\" (UniqueName: \"kubernetes.io/projected/9d4b52c6-fb44-431d-adff-446bbd7860ea-kube-api-access-8psgl\") pod \"9d4b52c6-fb44-431d-adff-446bbd7860ea\" (UID: \"9d4b52c6-fb44-431d-adff-446bbd7860ea\") "
Sep 18 14:08:21.921541 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:21.921335    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d4b52c6-fb44-431d-adff-446bbd7860ea-plugins-config-volume\") pod \"9d4b52c6-fb44-431d-adff-446bbd7860ea\" (UID: \"9d4b52c6-fb44-431d-adff-446bbd7860ea\") "
Sep 18 14:08:21.921782 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:21.921751    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d4b52c6-fb44-431d-adff-446bbd7860ea-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9d4b52c6-fb44-431d-adff-446bbd7860ea" (UID: "9d4b52c6-fb44-431d-adff-446bbd7860ea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:08:21.923480 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:21.923448    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d4b52c6-fb44-431d-adff-446bbd7860ea-kube-api-access-8psgl" (OuterVolumeSpecName: "kube-api-access-8psgl") pod "9d4b52c6-fb44-431d-adff-446bbd7860ea" (UID: "9d4b52c6-fb44-431d-adff-446bbd7860ea"). InnerVolumeSpecName "kube-api-access-8psgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:08:22.022660 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:22.022620    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8psgl\" (UniqueName: \"kubernetes.io/projected/9d4b52c6-fb44-431d-adff-446bbd7860ea-kube-api-access-8psgl\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:08:22.022660 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:22.022650    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d4b52c6-fb44-431d-adff-446bbd7860ea-plugins-config-volume\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:08:22.030513 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:22.030380    2577 generic.go:356] "Generic (PLEG): container finished" podID="9d4b52c6-fb44-431d-adff-446bbd7860ea" containerID="0f9eec2be2f35acc1aa783d33c02ce1706763410d406cf88a87b7d32d54c4ffe" exitCode=0
Sep 18 14:08:22.030513 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:22.030439    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ed4c53-59f4997598-7jfng" event={"ID":"9d4b52c6-fb44-431d-adff-446bbd7860ea","Type":"ContainerDied","Data":"0f9eec2be2f35acc1aa783d33c02ce1706763410d406cf88a87b7d32d54c4ffe"}
Sep 18 14:08:22.030513 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:22.030455    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ed4c53-59f4997598-7jfng"
Sep 18 14:08:22.030513 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:22.030500    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ed4c53-59f4997598-7jfng" event={"ID":"9d4b52c6-fb44-431d-adff-446bbd7860ea","Type":"ContainerDied","Data":"f9e62d780f6bffc4b1f6a4702952f0f648242a06c25a99c1dcca4dcb7d9c608d"}
Sep 18 14:08:22.030844 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:22.030532    2577 scope.go:117] "RemoveContainer" containerID="0f9eec2be2f35acc1aa783d33c02ce1706763410d406cf88a87b7d32d54c4ffe"
Sep 18 14:08:22.043408 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:22.043375    2577 scope.go:117] "RemoveContainer" containerID="0f9eec2be2f35acc1aa783d33c02ce1706763410d406cf88a87b7d32d54c4ffe"
Sep 18 14:08:22.043854 ip-10-0-141-222 kubenswrapper[2577]: E0918 14:08:22.043814    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f9eec2be2f35acc1aa783d33c02ce1706763410d406cf88a87b7d32d54c4ffe\": container with ID starting with 0f9eec2be2f35acc1aa783d33c02ce1706763410d406cf88a87b7d32d54c4ffe not found: ID does not exist" containerID="0f9eec2be2f35acc1aa783d33c02ce1706763410d406cf88a87b7d32d54c4ffe"
Sep 18 14:08:22.043953 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:22.043865    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f9eec2be2f35acc1aa783d33c02ce1706763410d406cf88a87b7d32d54c4ffe"} err="failed to get container status \"0f9eec2be2f35acc1aa783d33c02ce1706763410d406cf88a87b7d32d54c4ffe\": rpc error: code = NotFound desc = could not find container \"0f9eec2be2f35acc1aa783d33c02ce1706763410d406cf88a87b7d32d54c4ffe\": container with ID starting with 0f9eec2be2f35acc1aa783d33c02ce1706763410d406cf88a87b7d32d54c4ffe not found: ID does not exist"
Sep 18 14:08:22.062290 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:22.062241    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ed4c53-59f4997598-7jfng"]
Sep 18 14:08:22.066212 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:22.066177    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ed4c53-59f4997598-7jfng"]
Sep 18 14:08:23.109479 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:23.109444    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d4b52c6-fb44-431d-adff-446bbd7860ea" path="/var/lib/kubelet/pods/9d4b52c6-fb44-431d-adff-446bbd7860ea/volumes"
Sep 18 14:08:23.147635 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:23.147600    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:08:23.255272 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:23.255235    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:08:23.850349 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:23.850315    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:08:23.951644 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:23.951602    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:08:24.665540 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:24.665476    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:08:24.977076 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:24.976986    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:08:25.077455 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:25.077421    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:08:25.686469 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:25.686433    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:08:25.761853 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:25.761794    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-ed4c53/distinct-2-w0-ed4c53-kserve-7ccfc9d644-hjn2r" podUID="1d030a33-c49f-4697-9968-47f837e16bea" containerName="main" containerID="cri-o://9ce7e11c667a658dc3f38ef824a538bebb9e5b27094fc5a65f5d6780b615346f" gracePeriod=45
Sep 18 14:08:25.783563 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:25.783531    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-ed4c53/premium-w0-ed4c53-kserve-565c6b654d-mcfqz" podUID="66e829fe-01fb-48bc-a708-f1d35e050228" containerName="main" containerID="cri-o://9ea591b988b67c650993843d8474dac58dd6ddfafee71cf40666ba2594fb8f96" gracePeriod=45
Sep 18 14:08:25.795402 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:25.795367    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-ed4c53/simulator-w0-ed4c53-kserve-989f79bd5-z4lwd" podUID="3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e" containerName="main" containerID="cri-o://27d4b8142f62fac170c9503503fbb18a7469eaf10731424c23a6e5f4114031c8" gracePeriod=45
Sep 18 14:08:26.060780 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:26.060746    2577 generic.go:356] "Generic (PLEG): container finished" podID="3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e" containerID="27d4b8142f62fac170c9503503fbb18a7469eaf10731424c23a6e5f4114031c8" exitCode=0
Sep 18 14:08:26.060973 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:26.060794    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ed4c53/simulator-w0-ed4c53-kserve-989f79bd5-z4lwd" event={"ID":"3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e","Type":"ContainerDied","Data":"27d4b8142f62fac170c9503503fbb18a7469eaf10731424c23a6e5f4114031c8"}
Sep 18 14:08:26.062592 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:26.062561    2577 generic.go:356] "Generic (PLEG): container finished" podID="1d030a33-c49f-4697-9968-47f837e16bea" containerID="9ce7e11c667a658dc3f38ef824a538bebb9e5b27094fc5a65f5d6780b615346f" exitCode=0
Sep 18 14:08:26.062743 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:26.062636    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ed4c53/distinct-2-w0-ed4c53-kserve-7ccfc9d644-hjn2r" event={"ID":"1d030a33-c49f-4697-9968-47f837e16bea","Type":"ContainerDied","Data":"9ce7e11c667a658dc3f38ef824a538bebb9e5b27094fc5a65f5d6780b615346f"}
Sep 18 14:08:26.064397 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:26.064376    2577 generic.go:356] "Generic (PLEG): container finished" podID="66e829fe-01fb-48bc-a708-f1d35e050228" containerID="9ea591b988b67c650993843d8474dac58dd6ddfafee71cf40666ba2594fb8f96" exitCode=0
Sep 18 14:08:26.064518 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:26.064413    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ed4c53/premium-w0-ed4c53-kserve-565c6b654d-mcfqz" event={"ID":"66e829fe-01fb-48bc-a708-f1d35e050228","Type":"ContainerDied","Data":"9ea591b988b67c650993843d8474dac58dd6ddfafee71cf40666ba2594fb8f96"}
Sep 18 14:08:26.084043 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:26.084022    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ed4c53/distinct-2-w0-ed4c53-kserve-7ccfc9d644-hjn2r"
Sep 18 14:08:26.087879 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:26.087859    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ed4c53/premium-w0-ed4c53-kserve-565c6b654d-mcfqz"
Sep 18 14:08:26.092035 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:26.092018    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ed4c53/simulator-w0-ed4c53-kserve-989f79bd5-z4lwd"
Sep 18 14:08:26.160450 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:26.160423    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d030a33-c49f-4697-9968-47f837e16bea-tls-certs\") pod \"1d030a33-c49f-4697-9968-47f837e16bea\" (UID: \"1d030a33-c49f-4697-9968-47f837e16bea\") "
Sep 18 14:08:26.160637 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:26.160466    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e-tmp-dir\") pod \"3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e\" (UID: \"3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e\") "
Sep 18 14:08:26.160637 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:26.160489    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p7q4t\" (UniqueName: \"kubernetes.io/projected/1d030a33-c49f-4697-9968-47f837e16bea-kube-api-access-p7q4t\") pod \"1d030a33-c49f-4697-9968-47f837e16bea\" (UID: \"1d030a33-c49f-4697-9968-47f837e16bea\") "
Sep 18 14:08:26.160637 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:26.160510    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e-dshm\") pod \"3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e\" (UID: \"3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e\") "
Sep 18 14:08:26.160637 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:26.160531    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66e829fe-01fb-48bc-a708-f1d35e050228-tmp-dir\") pod \"66e829fe-01fb-48bc-a708-f1d35e050228\" (UID: \"66e829fe-01fb-48bc-a708-f1d35e050228\") "
Sep 18 14:08:26.160637 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:26.160564    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e-home\") pod \"3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e\" (UID: \"3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e\") "
Sep 18 14:08:26.160637 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:26.160588    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7jb7g\" (UniqueName: \"kubernetes.io/projected/66e829fe-01fb-48bc-a708-f1d35e050228-kube-api-access-7jb7g\") pod \"66e829fe-01fb-48bc-a708-f1d35e050228\" (UID: \"66e829fe-01fb-48bc-a708-f1d35e050228\") "
Sep 18 14:08:26.160637 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:26.160633    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d030a33-c49f-4697-9968-47f837e16bea-dshm\") pod \"1d030a33-c49f-4697-9968-47f837e16bea\" (UID: \"1d030a33-c49f-4697-9968-47f837e16bea\") "
Sep 18 14:08:26.161060 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:26.160676    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d030a33-c49f-4697-9968-47f837e16bea-model-cache\") pod \"1d030a33-c49f-4697-9968-47f837e16bea\" (UID: \"1d030a33-c49f-4697-9968-47f837e16bea\") "
Sep 18 14:08:26.161060 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:26.160704    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66e829fe-01fb-48bc-a708-f1d35e050228-home\") pod \"66e829fe-01fb-48bc-a708-f1d35e050228\" (UID: \"66e829fe-01fb-48bc-a708-f1d35e050228\") "
Sep 18 14:08:26.161060 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:26.160753    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e-tls-certs\") pod \"3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e\" (UID: \"3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e\") "
Sep 18 14:08:26.161060 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:26.160781    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e-model-cache\") pod \"3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e\" (UID: \"3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e\") "
Sep 18 14:08:26.161060 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:26.160797    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e" (UID: "3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:08:26.161060 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:26.160807    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4xh26\" (UniqueName: \"kubernetes.io/projected/3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e-kube-api-access-4xh26\") pod \"3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e\" (UID: \"3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e\") "
Sep 18 14:08:26.161060 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:26.160858    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d030a33-c49f-4697-9968-47f837e16bea-tmp-dir\") pod \"1d030a33-c49f-4697-9968-47f837e16bea\" (UID: \"1d030a33-c49f-4697-9968-47f837e16bea\") "
Sep 18 14:08:26.161060 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:26.160890    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66e829fe-01fb-48bc-a708-f1d35e050228-dshm\") pod \"66e829fe-01fb-48bc-a708-f1d35e050228\" (UID: \"66e829fe-01fb-48bc-a708-f1d35e050228\") "
Sep 18 14:08:26.161060 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:26.160925    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66e829fe-01fb-48bc-a708-f1d35e050228-tls-certs\") pod \"66e829fe-01fb-48bc-a708-f1d35e050228\" (UID: \"66e829fe-01fb-48bc-a708-f1d35e050228\") "
Sep 18 14:08:26.161060 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:26.160960    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66e829fe-01fb-48bc-a708-f1d35e050228-model-cache\") pod \"66e829fe-01fb-48bc-a708-f1d35e050228\" (UID: \"66e829fe-01fb-48bc-a708-f1d35e050228\") "
Sep 18 14:08:26.161060 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:26.161004    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66e829fe-01fb-48bc-a708-f1d35e050228-home" (OuterVolumeSpecName: "home") pod "66e829fe-01fb-48bc-a708-f1d35e050228" (UID: "66e829fe-01fb-48bc-a708-f1d35e050228"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:08:26.161060 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:26.161022    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d030a33-c49f-4697-9968-47f837e16bea-home\") pod \"1d030a33-c49f-4697-9968-47f837e16bea\" (UID: \"1d030a33-c49f-4697-9968-47f837e16bea\") "
Sep 18 14:08:26.161877 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:26.161184    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d030a33-c49f-4697-9968-47f837e16bea-model-cache" (OuterVolumeSpecName: "model-cache") pod "1d030a33-c49f-4697-9968-47f837e16bea" (UID: "1d030a33-c49f-4697-9968-47f837e16bea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:08:26.161877 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:26.161356    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d030a33-c49f-4697-9968-47f837e16bea-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1d030a33-c49f-4697-9968-47f837e16bea" (UID: "1d030a33-c49f-4697-9968-47f837e16bea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:08:26.161877 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:26.161514    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d030a33-c49f-4697-9968-47f837e16bea-model-cache\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:08:26.161877 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:26.161533    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66e829fe-01fb-48bc-a708-f1d35e050228-home\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:08:26.161877 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:26.161546    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d030a33-c49f-4697-9968-47f837e16bea-tmp-dir\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:08:26.161877 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:26.161558    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e-tmp-dir\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:08:26.161877 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:26.161640    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66e829fe-01fb-48bc-a708-f1d35e050228-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "66e829fe-01fb-48bc-a708-f1d35e050228" (UID: "66e829fe-01fb-48bc-a708-f1d35e050228"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:08:26.161877 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:26.161796    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e-home" (OuterVolumeSpecName: "home") pod "3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e" (UID: "3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:08:26.162326 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:26.162301    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e-model-cache" (OuterVolumeSpecName: "model-cache") pod "3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e" (UID: "3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:08:26.162915 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:26.162607    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d030a33-c49f-4697-9968-47f837e16bea-home" (OuterVolumeSpecName: "home") pod "1d030a33-c49f-4697-9968-47f837e16bea" (UID: "1d030a33-c49f-4697-9968-47f837e16bea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:08:26.164724 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:26.164692    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66e829fe-01fb-48bc-a708-f1d35e050228-model-cache" (OuterVolumeSpecName: "model-cache") pod "66e829fe-01fb-48bc-a708-f1d35e050228" (UID: "66e829fe-01fb-48bc-a708-f1d35e050228"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:08:26.164858 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:26.164766    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e" (UID: "3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:08:26.165309 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:26.165270    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d030a33-c49f-4697-9968-47f837e16bea-dshm" (OuterVolumeSpecName: "dshm") pod "1d030a33-c49f-4697-9968-47f837e16bea" (UID: "1d030a33-c49f-4697-9968-47f837e16bea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:08:26.165430 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:26.165408    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66e829fe-01fb-48bc-a708-f1d35e050228-dshm" (OuterVolumeSpecName: "dshm") pod "66e829fe-01fb-48bc-a708-f1d35e050228" (UID: "66e829fe-01fb-48bc-a708-f1d35e050228"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:08:26.166140 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:26.166097    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d030a33-c49f-4697-9968-47f837e16bea-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1d030a33-c49f-4697-9968-47f837e16bea" (UID: "1d030a33-c49f-4697-9968-47f837e16bea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:08:26.166140 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:26.166104    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d030a33-c49f-4697-9968-47f837e16bea-kube-api-access-p7q4t" (OuterVolumeSpecName: "kube-api-access-p7q4t") pod "1d030a33-c49f-4697-9968-47f837e16bea" (UID: "1d030a33-c49f-4697-9968-47f837e16bea"). InnerVolumeSpecName "kube-api-access-p7q4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:08:26.166964 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:26.166929    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66e829fe-01fb-48bc-a708-f1d35e050228-kube-api-access-7jb7g" (OuterVolumeSpecName: "kube-api-access-7jb7g") pod "66e829fe-01fb-48bc-a708-f1d35e050228" (UID: "66e829fe-01fb-48bc-a708-f1d35e050228"). InnerVolumeSpecName "kube-api-access-7jb7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:08:26.167184 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:26.167158    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66e829fe-01fb-48bc-a708-f1d35e050228-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "66e829fe-01fb-48bc-a708-f1d35e050228" (UID: "66e829fe-01fb-48bc-a708-f1d35e050228"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:08:26.167343 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:26.167321    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e-dshm" (OuterVolumeSpecName: "dshm") pod "3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e" (UID: "3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:08:26.167755 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:26.167729    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e-kube-api-access-4xh26" (OuterVolumeSpecName: "kube-api-access-4xh26") pod "3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e" (UID: "3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e"). InnerVolumeSpecName "kube-api-access-4xh26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:08:26.263119 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:26.263031    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66e829fe-01fb-48bc-a708-f1d35e050228-model-cache\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:08:26.263119 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:26.263061    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d030a33-c49f-4697-9968-47f837e16bea-home\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:08:26.263119 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:26.263070    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d030a33-c49f-4697-9968-47f837e16bea-tls-certs\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:08:26.263119 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:26.263079    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7q4t\" (UniqueName: \"kubernetes.io/projected/1d030a33-c49f-4697-9968-47f837e16bea-kube-api-access-p7q4t\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:08:26.263119 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:26.263088    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e-dshm\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:08:26.263119 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:26.263098    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66e829fe-01fb-48bc-a708-f1d35e050228-tmp-dir\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:08:26.263119 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:26.263110    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e-home\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:08:26.263119 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:26.263118    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jb7g\" (UniqueName: \"kubernetes.io/projected/66e829fe-01fb-48bc-a708-f1d35e050228-kube-api-access-7jb7g\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:08:26.263119 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:26.263125    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d030a33-c49f-4697-9968-47f837e16bea-dshm\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:08:26.263513 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:26.263136    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e-tls-certs\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:08:26.263513 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:26.263144    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e-model-cache\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:08:26.263513 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:26.263151    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xh26\" (UniqueName: \"kubernetes.io/projected/3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e-kube-api-access-4xh26\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:08:26.263513 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:26.263159    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66e829fe-01fb-48bc-a708-f1d35e050228-dshm\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:08:26.263513 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:26.263167    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66e829fe-01fb-48bc-a708-f1d35e050228-tls-certs\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:08:26.472981 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:26.472946    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:08:27.072091 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:27.072006    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ed4c53/distinct-2-w0-ed4c53-kserve-7ccfc9d644-hjn2r"
Sep 18 14:08:27.072091 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:27.072032    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ed4c53/distinct-2-w0-ed4c53-kserve-7ccfc9d644-hjn2r" event={"ID":"1d030a33-c49f-4697-9968-47f837e16bea","Type":"ContainerDied","Data":"a44f727e6912f5a06bd97c8a07911755020088a7bb5f287bb640eb378aed9dcc"}
Sep 18 14:08:27.072091 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:27.072089    2577 scope.go:117] "RemoveContainer" containerID="9ce7e11c667a658dc3f38ef824a538bebb9e5b27094fc5a65f5d6780b615346f"
Sep 18 14:08:27.073493 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:27.073413    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ed4c53/premium-w0-ed4c53-kserve-565c6b654d-mcfqz"
Sep 18 14:08:27.073493 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:27.073439    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ed4c53/premium-w0-ed4c53-kserve-565c6b654d-mcfqz" event={"ID":"66e829fe-01fb-48bc-a708-f1d35e050228","Type":"ContainerDied","Data":"24b5d99fd09a61502855b86a38f6a81b95099e406048f000451f6d7748e50adc"}
Sep 18 14:08:27.075152 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:27.075128    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ed4c53/simulator-w0-ed4c53-kserve-989f79bd5-z4lwd"
Sep 18 14:08:27.075276 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:27.075126    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ed4c53/simulator-w0-ed4c53-kserve-989f79bd5-z4lwd" event={"ID":"3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e","Type":"ContainerDied","Data":"4efcde98fa0633697d78afb9f3f4ed8d62f2cc2bc0a5ef1e5c483bc9b109ffeb"}
Sep 18 14:08:27.083979 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:27.083963    2577 scope.go:117] "RemoveContainer" containerID="9ea591b988b67c650993843d8474dac58dd6ddfafee71cf40666ba2594fb8f96"
Sep 18 14:08:27.092651 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:27.092633    2577 scope.go:117] "RemoveContainer" containerID="27d4b8142f62fac170c9503503fbb18a7469eaf10731424c23a6e5f4114031c8"
Sep 18 14:08:27.100767 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:27.100658    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ed4c53/distinct-2-w0-ed4c53-kserve-7ccfc9d644-hjn2r"]
Sep 18 14:08:27.111454 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:27.111429    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-ed4c53/distinct-2-w0-ed4c53-kserve-7ccfc9d644-hjn2r"]
Sep 18 14:08:27.127226 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:27.127202    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ed4c53/simulator-w0-ed4c53-kserve-989f79bd5-z4lwd"]
Sep 18 14:08:27.134679 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:27.134641    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-ed4c53/simulator-w0-ed4c53-kserve-989f79bd5-z4lwd"]
Sep 18 14:08:27.152125 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:27.152100    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ed4c53/premium-w0-ed4c53-kserve-565c6b654d-mcfqz"]
Sep 18 14:08:27.157783 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:27.157757    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-ed4c53/premium-w0-ed4c53-kserve-565c6b654d-mcfqz"]
Sep 18 14:08:27.205917 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:27.205887    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ca1331-66c7d84454-ztxcz"]
Sep 18 14:08:27.206480 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:27.206464    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d030a33-c49f-4697-9968-47f837e16bea" containerName="main"
Sep 18 14:08:27.206528 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:27.206484    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d030a33-c49f-4697-9968-47f837e16bea" containerName="main"
Sep 18 14:08:27.206528 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:27.206506    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6342501-29e4-4f74-936a-94f796d7889f" containerName="main"
Sep 18 14:08:27.206528 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:27.206516    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6342501-29e4-4f74-936a-94f796d7889f" containerName="main"
Sep 18 14:08:27.206624 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:27.206529    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d4b52c6-fb44-431d-adff-446bbd7860ea" containerName="payload-pre-processing"
Sep 18 14:08:27.206624 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:27.206539    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d4b52c6-fb44-431d-adff-446bbd7860ea" containerName="payload-pre-processing"
Sep 18 14:08:27.206624 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:27.206550    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66e829fe-01fb-48bc-a708-f1d35e050228" containerName="main"
Sep 18 14:08:27.206624 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:27.206559    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="66e829fe-01fb-48bc-a708-f1d35e050228" containerName="main"
Sep 18 14:08:27.206624 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:27.206577    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e" containerName="main"
Sep 18 14:08:27.206624 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:27.206585    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e" containerName="main"
Sep 18 14:08:27.206801 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:27.206682    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6342501-29e4-4f74-936a-94f796d7889f" containerName="main"
Sep 18 14:08:27.206801 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:27.206693    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d4b52c6-fb44-431d-adff-446bbd7860ea" containerName="payload-pre-processing"
Sep 18 14:08:27.206801 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:27.206704    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e" containerName="main"
Sep 18 14:08:27.206801 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:27.206716    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="66e829fe-01fb-48bc-a708-f1d35e050228" containerName="main"
Sep 18 14:08:27.206801 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:27.206727    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d030a33-c49f-4697-9968-47f837e16bea" containerName="main"
Sep 18 14:08:27.211681 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:27.211659    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ca1331-66c7d84454-ztxcz"
Sep 18 14:08:27.214478 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:27.214453    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-ca1331-serving-cert\""
Sep 18 14:08:27.230147 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:27.230125    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ca1331-66c7d84454-ztxcz"]
Sep 18 14:08:27.262282 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:27.262244    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:08:27.272461 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:27.272435    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abf81e5c-1248-4af3-bdc0-4b2928175a62-maas-api-tls\") pod \"maas-api-e2e-api-a-ca1331-66c7d84454-ztxcz\" (UID: \"abf81e5c-1248-4af3-bdc0-4b2928175a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ca1331-66c7d84454-ztxcz"
Sep 18 14:08:27.272578 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:27.272500    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abf81e5c-1248-4af3-bdc0-4b2928175a62-metrics-certs\") pod \"maas-api-e2e-api-a-ca1331-66c7d84454-ztxcz\" (UID: \"abf81e5c-1248-4af3-bdc0-4b2928175a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ca1331-66c7d84454-ztxcz"
Sep 18 14:08:27.272578 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:27.272533    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abf81e5c-1248-4af3-bdc0-4b2928175a62-openshift-service-ca\") pod \"maas-api-e2e-api-a-ca1331-66c7d84454-ztxcz\" (UID: \"abf81e5c-1248-4af3-bdc0-4b2928175a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ca1331-66c7d84454-ztxcz"
Sep 18 14:08:27.272578 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:27.272564    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rqdx\" (UniqueName: \"kubernetes.io/projected/abf81e5c-1248-4af3-bdc0-4b2928175a62-kube-api-access-7rqdx\") pod \"maas-api-e2e-api-a-ca1331-66c7d84454-ztxcz\" (UID: \"abf81e5c-1248-4af3-bdc0-4b2928175a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ca1331-66c7d84454-ztxcz"
Sep 18 14:08:27.360528 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:27.360493    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:08:27.373783 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:27.373744    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abf81e5c-1248-4af3-bdc0-4b2928175a62-maas-api-tls\") pod \"maas-api-e2e-api-a-ca1331-66c7d84454-ztxcz\" (UID: \"abf81e5c-1248-4af3-bdc0-4b2928175a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ca1331-66c7d84454-ztxcz"
Sep 18 14:08:27.373967 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:27.373812    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abf81e5c-1248-4af3-bdc0-4b2928175a62-metrics-certs\") pod \"maas-api-e2e-api-a-ca1331-66c7d84454-ztxcz\" (UID: \"abf81e5c-1248-4af3-bdc0-4b2928175a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ca1331-66c7d84454-ztxcz"
Sep 18 14:08:27.373967 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:27.373867    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abf81e5c-1248-4af3-bdc0-4b2928175a62-openshift-service-ca\") pod \"maas-api-e2e-api-a-ca1331-66c7d84454-ztxcz\" (UID: \"abf81e5c-1248-4af3-bdc0-4b2928175a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ca1331-66c7d84454-ztxcz"
Sep 18 14:08:27.373967 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:27.373903    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rqdx\" (UniqueName: \"kubernetes.io/projected/abf81e5c-1248-4af3-bdc0-4b2928175a62-kube-api-access-7rqdx\") pod \"maas-api-e2e-api-a-ca1331-66c7d84454-ztxcz\" (UID: \"abf81e5c-1248-4af3-bdc0-4b2928175a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ca1331-66c7d84454-ztxcz"
Sep 18 14:08:27.373967 ip-10-0-141-222 kubenswrapper[2577]: E0918 14:08:27.373928    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-ca1331-serving-cert: secret "maas-api-e2e-api-a-ca1331-serving-cert" not found
Sep 18 14:08:27.374187 ip-10-0-141-222 kubenswrapper[2577]: E0918 14:08:27.374006    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abf81e5c-1248-4af3-bdc0-4b2928175a62-maas-api-tls podName:abf81e5c-1248-4af3-bdc0-4b2928175a62 nodeName:}" failed. No retries permitted until 2026-09-18 14:08:27.873985533 +0000 UTC m=+1456.926644852 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/abf81e5c-1248-4af3-bdc0-4b2928175a62-maas-api-tls") pod "maas-api-e2e-api-a-ca1331-66c7d84454-ztxcz" (UID: "abf81e5c-1248-4af3-bdc0-4b2928175a62") : secret "maas-api-e2e-api-a-ca1331-serving-cert" not found
Sep 18 14:08:27.374579 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:27.374561    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abf81e5c-1248-4af3-bdc0-4b2928175a62-openshift-service-ca\") pod \"maas-api-e2e-api-a-ca1331-66c7d84454-ztxcz\" (UID: \"abf81e5c-1248-4af3-bdc0-4b2928175a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ca1331-66c7d84454-ztxcz"
Sep 18 14:08:27.376260 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:27.376244    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abf81e5c-1248-4af3-bdc0-4b2928175a62-metrics-certs\") pod \"maas-api-e2e-api-a-ca1331-66c7d84454-ztxcz\" (UID: \"abf81e5c-1248-4af3-bdc0-4b2928175a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ca1331-66c7d84454-ztxcz"
Sep 18 14:08:27.386118 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:27.386096    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rqdx\" (UniqueName: \"kubernetes.io/projected/abf81e5c-1248-4af3-bdc0-4b2928175a62-kube-api-access-7rqdx\") pod \"maas-api-e2e-api-a-ca1331-66c7d84454-ztxcz\" (UID: \"abf81e5c-1248-4af3-bdc0-4b2928175a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ca1331-66c7d84454-ztxcz"
Sep 18 14:08:27.534052 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:27.534017    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:08:27.878666 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:27.878632    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abf81e5c-1248-4af3-bdc0-4b2928175a62-maas-api-tls\") pod \"maas-api-e2e-api-a-ca1331-66c7d84454-ztxcz\" (UID: \"abf81e5c-1248-4af3-bdc0-4b2928175a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ca1331-66c7d84454-ztxcz"
Sep 18 14:08:27.881117 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:27.881093    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abf81e5c-1248-4af3-bdc0-4b2928175a62-maas-api-tls\") pod \"maas-api-e2e-api-a-ca1331-66c7d84454-ztxcz\" (UID: \"abf81e5c-1248-4af3-bdc0-4b2928175a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ca1331-66c7d84454-ztxcz"
Sep 18 14:08:27.915260 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:27.915225    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:08:28.015544 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:28.015508    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:08:28.089202 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:28.089174    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ca1331-5cfd7b7d78-tttxp"]
Sep 18 14:08:28.116592 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:28.093523    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ca1331-5cfd7b7d78-tttxp"
Sep 18 14:08:28.116592 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:28.095952    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-ca1331-dockercfg-zfxd4\""
Sep 18 14:08:28.116592 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:28.095983    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-ca1331\""
Sep 18 14:08:28.116592 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:28.101192    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ca1331-5cfd7b7d78-tttxp"]
Sep 18 14:08:28.122413 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:28.122383    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ca1331-66c7d84454-ztxcz"
Sep 18 14:08:28.180966 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:28.180931    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlx7w\" (UniqueName: \"kubernetes.io/projected/6273d451-8a5f-4c31-90be-ba6c0aa10853-kube-api-access-wlx7w\") pod \"payload-pre-processing-e2e-api-a-ca1331-5cfd7b7d78-tttxp\" (UID: \"6273d451-8a5f-4c31-90be-ba6c0aa10853\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ca1331-5cfd7b7d78-tttxp"
Sep 18 14:08:28.181154 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:28.181034    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6273d451-8a5f-4c31-90be-ba6c0aa10853-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ca1331-5cfd7b7d78-tttxp\" (UID: \"6273d451-8a5f-4c31-90be-ba6c0aa10853\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ca1331-5cfd7b7d78-tttxp"
Sep 18 14:08:28.282249 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:28.282214    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6273d451-8a5f-4c31-90be-ba6c0aa10853-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ca1331-5cfd7b7d78-tttxp\" (UID: \"6273d451-8a5f-4c31-90be-ba6c0aa10853\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ca1331-5cfd7b7d78-tttxp"
Sep 18 14:08:28.282503 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:28.282310    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlx7w\" (UniqueName: \"kubernetes.io/projected/6273d451-8a5f-4c31-90be-ba6c0aa10853-kube-api-access-wlx7w\") pod \"payload-pre-processing-e2e-api-a-ca1331-5cfd7b7d78-tttxp\" (UID: \"6273d451-8a5f-4c31-90be-ba6c0aa10853\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ca1331-5cfd7b7d78-tttxp"
Sep 18 14:08:28.282939 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:28.282920    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6273d451-8a5f-4c31-90be-ba6c0aa10853-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ca1331-5cfd7b7d78-tttxp\" (UID: \"6273d451-8a5f-4c31-90be-ba6c0aa10853\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ca1331-5cfd7b7d78-tttxp"
Sep 18 14:08:28.290916 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:28.290886    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlx7w\" (UniqueName: \"kubernetes.io/projected/6273d451-8a5f-4c31-90be-ba6c0aa10853-kube-api-access-wlx7w\") pod \"payload-pre-processing-e2e-api-a-ca1331-5cfd7b7d78-tttxp\" (UID: \"6273d451-8a5f-4c31-90be-ba6c0aa10853\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ca1331-5cfd7b7d78-tttxp"
Sep 18 14:08:28.342395 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:28.342343    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:08:28.405613 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:28.405442    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ca1331-5cfd7b7d78-tttxp"
Sep 18 14:08:28.409266 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:28.409226    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ca1331-66c7d84454-ztxcz"]
Sep 18 14:08:28.412322 ip-10-0-141-222 kubenswrapper[2577]: W0918 14:08:28.412289    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabf81e5c_1248_4af3_bdc0_4b2928175a62.slice/crio-69b03e6947c3d4f1fe5d49de9070e540e5da81d6784800643c0d79f3cbbc61c0 WatchSource:0}: Error finding container 69b03e6947c3d4f1fe5d49de9070e540e5da81d6784800643c0d79f3cbbc61c0: Status 404 returned error can't find the container with id 69b03e6947c3d4f1fe5d49de9070e540e5da81d6784800643c0d79f3cbbc61c0
Sep 18 14:08:28.689965 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:28.689943    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ca1331-5cfd7b7d78-tttxp"]
Sep 18 14:08:28.692139 ip-10-0-141-222 kubenswrapper[2577]: W0918 14:08:28.692110    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6273d451_8a5f_4c31_90be_ba6c0aa10853.slice/crio-95e15eeb9d9a9e323f6435e000c19d45f0c891942f5988690e0b3600f0c31602 WatchSource:0}: Error finding container 95e15eeb9d9a9e323f6435e000c19d45f0c891942f5988690e0b3600f0c31602: Status 404 returned error can't find the container with id 95e15eeb9d9a9e323f6435e000c19d45f0c891942f5988690e0b3600f0c31602
Sep 18 14:08:29.092728 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:29.092692    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ca1331-5cfd7b7d78-tttxp" event={"ID":"6273d451-8a5f-4c31-90be-ba6c0aa10853","Type":"ContainerStarted","Data":"cfebbcf4a50fd22b4b317e97ee6ccaa7227789174ff3126500c6f420211280cf"}
Sep 18 14:08:29.093231 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:29.092762    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ca1331-5cfd7b7d78-tttxp" event={"ID":"6273d451-8a5f-4c31-90be-ba6c0aa10853","Type":"ContainerStarted","Data":"95e15eeb9d9a9e323f6435e000c19d45f0c891942f5988690e0b3600f0c31602"}
Sep 18 14:08:29.093231 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:29.092789    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ca1331-5cfd7b7d78-tttxp"
Sep 18 14:08:29.094603 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:29.094576    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ca1331-66c7d84454-ztxcz" event={"ID":"abf81e5c-1248-4af3-bdc0-4b2928175a62","Type":"ContainerStarted","Data":"35197cbd94506d71af6b03716394cae496b89d181ced804ec9afc3dcaeee6f24"}
Sep 18 14:08:29.094738 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:29.094621    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ca1331-66c7d84454-ztxcz" event={"ID":"abf81e5c-1248-4af3-bdc0-4b2928175a62","Type":"ContainerStarted","Data":"69b03e6947c3d4f1fe5d49de9070e540e5da81d6784800643c0d79f3cbbc61c0"}
Sep 18 14:08:29.094738 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:29.094653    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ca1331-66c7d84454-ztxcz"
Sep 18 14:08:29.109343 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:29.109313    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d030a33-c49f-4697-9968-47f837e16bea" path="/var/lib/kubelet/pods/1d030a33-c49f-4697-9968-47f837e16bea/volumes"
Sep 18 14:08:29.109857 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:29.109840    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e" path="/var/lib/kubelet/pods/3c6a9f4b-ddd1-4ff7-a981-36f74c8e941e/volumes"
Sep 18 14:08:29.110350 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:29.110331    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66e829fe-01fb-48bc-a708-f1d35e050228" path="/var/lib/kubelet/pods/66e829fe-01fb-48bc-a708-f1d35e050228/volumes"
Sep 18 14:08:29.138349 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:29.138302    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ca1331-5cfd7b7d78-tttxp" podStartSLOduration=1.138287113 podStartE2EDuration="1.138287113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:08:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:08:29.136121886 +0000 UTC m=+1458.188781215" watchObservedRunningTime="2026-09-18 14:08:29.138287113 +0000 UTC m=+1458.190946441"
Sep 18 14:08:29.202833 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:29.202772    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ca1331-66c7d84454-ztxcz" podStartSLOduration=1.735819566 podStartE2EDuration="2.202751047s" totalImagesPullingTime="466.931481ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:08:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:08:29.200268241 +0000 UTC m=+1458.252927568" watchObservedRunningTime="2026-09-18 14:08:29.202751047 +0000 UTC m=+1458.255410376"
Sep 18 14:08:29.440935 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:29.440836    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:08:30.164841 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:30.164789    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:08:30.463470 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:30.463382    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:08:30.614407 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:30.614353    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:08:30.726540 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:30.726400    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:08:31.012978 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:31.012896    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:08:35.107938 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:35.107904    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ca1331-66c7d84454-ztxcz"
Sep 18 14:08:37.302162 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:37.302128    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 14:08:40.103795 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:40.103761    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ca1331-5cfd7b7d78-tttxp"
Sep 18 14:08:48.569675 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:48.569639    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:08:48.671958 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:48.671923    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:08:48.848745 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:48.848711    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:08:49.311629 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:49.311554    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:08:49.380016 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:49.379988    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:08:49.721520 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:49.721489    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:08:49.899186 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:49.899141    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:08:50.325950 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:50.325911    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:08:50.439873 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:50.439840    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:08:50.683530 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:50.683493    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:08:51.177687 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:51.177645    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:08:51.364168 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:51.364138    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:08:52.231638 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:52.231607    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:08:52.933764 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:52.933730    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:08:53.433564 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:53.433529    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:08:53.525034 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:53.525000    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:08:53.759948 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:53.759850    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:08:54.492466 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:54.492435    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:08:55.263933 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:55.263896    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:08:55.340667 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:55.340635    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:08:55.755412 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:55.755375    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:08:55.847936 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:55.847899    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:08:55.929467 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:55.929432    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-eb3054-b95bb9dcd-qqqqk"]
Sep 18 14:08:55.933232 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:55.933215    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-eb3054-b95bb9dcd-qqqqk"
Sep 18 14:08:55.935315 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:55.935295    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-eb3054\""
Sep 18 14:08:55.935459 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:55.935443    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-eb3054-dockercfg-wrqlm\""
Sep 18 14:08:55.940225 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:55.940202    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0611c055-7a5e-4503-b076-22ac855233ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-eb3054-b95bb9dcd-qqqqk\" (UID: \"0611c055-7a5e-4503-b076-22ac855233ac\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-eb3054-b95bb9dcd-qqqqk"
Sep 18 14:08:55.940753 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:55.940725    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7ssj\" (UniqueName: \"kubernetes.io/projected/0611c055-7a5e-4503-b076-22ac855233ac-kube-api-access-w7ssj\") pod \"payload-pre-processing-e2e-api-b-eb3054-b95bb9dcd-qqqqk\" (UID: \"0611c055-7a5e-4503-b076-22ac855233ac\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-eb3054-b95bb9dcd-qqqqk"
Sep 18 14:08:55.942873 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:55.942850    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-eb3054-b95bb9dcd-qqqqk"]
Sep 18 14:08:56.041765 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:56.041675    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0611c055-7a5e-4503-b076-22ac855233ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-eb3054-b95bb9dcd-qqqqk\" (UID: \"0611c055-7a5e-4503-b076-22ac855233ac\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-eb3054-b95bb9dcd-qqqqk"
Sep 18 14:08:56.041765 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:56.041746    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7ssj\" (UniqueName: \"kubernetes.io/projected/0611c055-7a5e-4503-b076-22ac855233ac-kube-api-access-w7ssj\") pod \"payload-pre-processing-e2e-api-b-eb3054-b95bb9dcd-qqqqk\" (UID: \"0611c055-7a5e-4503-b076-22ac855233ac\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-eb3054-b95bb9dcd-qqqqk"
Sep 18 14:08:56.042552 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:56.042533    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0611c055-7a5e-4503-b076-22ac855233ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-eb3054-b95bb9dcd-qqqqk\" (UID: \"0611c055-7a5e-4503-b076-22ac855233ac\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-eb3054-b95bb9dcd-qqqqk"
Sep 18 14:08:56.049963 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:56.049932    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7ssj\" (UniqueName: \"kubernetes.io/projected/0611c055-7a5e-4503-b076-22ac855233ac-kube-api-access-w7ssj\") pod \"payload-pre-processing-e2e-api-b-eb3054-b95bb9dcd-qqqqk\" (UID: \"0611c055-7a5e-4503-b076-22ac855233ac\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-eb3054-b95bb9dcd-qqqqk"
Sep 18 14:08:56.084337 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:56.084310    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:08:56.185479 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:56.185423    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:08:56.244810 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:56.244782    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-eb3054-b95bb9dcd-qqqqk"
Sep 18 14:08:56.285872 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:56.285812    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:08:56.519233 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:56.519202    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-eb3054-b95bb9dcd-qqqqk"]
Sep 18 14:08:56.521324 ip-10-0-141-222 kubenswrapper[2577]: W0918 14:08:56.521296    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0611c055_7a5e_4503_b076_22ac855233ac.slice/crio-706c11c226c94a165c40aec484d211535bd541e81b5741834436bbba5b1d9f8a WatchSource:0}: Error finding container 706c11c226c94a165c40aec484d211535bd541e81b5741834436bbba5b1d9f8a: Status 404 returned error can't find the container with id 706c11c226c94a165c40aec484d211535bd541e81b5741834436bbba5b1d9f8a
Sep 18 14:08:56.892509 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:56.892472    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:08:57.259751 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:57.259656    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-eb3054-b95bb9dcd-qqqqk" event={"ID":"0611c055-7a5e-4503-b076-22ac855233ac","Type":"ContainerStarted","Data":"161d97361c3c2e83d65541f0f11912420e5d6b66ef2192c3035a5da2bb2237a1"}
Sep 18 14:08:57.259751 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:57.259720    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-eb3054-b95bb9dcd-qqqqk" event={"ID":"0611c055-7a5e-4503-b076-22ac855233ac","Type":"ContainerStarted","Data":"706c11c226c94a165c40aec484d211535bd541e81b5741834436bbba5b1d9f8a"}
Sep 18 14:08:57.259751 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:57.259750    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-eb3054-b95bb9dcd-qqqqk"
Sep 18 14:08:57.276403 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:57.276348    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-eb3054-b95bb9dcd-qqqqk" podStartSLOduration=2.276330023 podStartE2EDuration="2.276330023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:08:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:08:57.27403054 +0000 UTC m=+1486.326689867" watchObservedRunningTime="2026-09-18 14:08:57.276330023 +0000 UTC m=+1486.328989349"
Sep 18 14:08:57.549700 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:57.549613    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:08:58.338500 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:58.338452    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:08:59.143316 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:08:59.143285    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:09:00.138179 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:00.138136    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:09:02.091087 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:02.091057    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:09:07.562056 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:07.562021    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:09:08.269012 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:08.268979    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-eb3054-b95bb9dcd-qqqqk"
Sep 18 14:09:13.209643 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:13.209613    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-8tnxl_69d3178a-5769-4b5f-a243-30a1bedc5bd6/console-operator/1.log"
Sep 18 14:09:13.220004 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:13.219974    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-8tnxl_69d3178a-5769-4b5f-a243-30a1bedc5bd6/console-operator/1.log"
Sep 18 14:09:13.360357 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:13.360324    2577 scope.go:117] "RemoveContainer" containerID="8b3581ba1551b1e9ec9f89df626d2eb839eefd84e404993f1ceede0584d4ba4b"
Sep 18 14:09:17.092198 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:17.092163    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:09:17.821506 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:17.821463    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:09:18.361760 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:18.361721    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:09:18.656673 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:18.656588    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:09:19.473251 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:19.473213    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:09:19.906200 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:19.906165    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 14:09:20.318584 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:20.318472    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:09:21.132771 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:21.132731    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:09:21.937735 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:21.937693    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:09:22.658130 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:22.658099    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:09:23.523040 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:23.522991    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:09:24.229190 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:24.229129    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:09:25.126170 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:25.126085    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:09:25.784137 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:25.784083    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:09:26.506900 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:26.506870    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:09:27.261632 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:27.261585    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:09:31.360846 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:31.360789    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:09:36.565652 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:36.565614    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:09:37.092305 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:37.092268    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:09:37.751166 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:37.751131    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ca1331-66c7d84454-ztxcz"]
Sep 18 14:09:37.751649 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:37.751386    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ca1331-66c7d84454-ztxcz" podUID="abf81e5c-1248-4af3-bdc0-4b2928175a62" containerName="maas-api" containerID="cri-o://35197cbd94506d71af6b03716394cae496b89d181ced804ec9afc3dcaeee6f24" gracePeriod=30
Sep 18 14:09:37.781686 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:37.781656    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:09:37.812333 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:37.812298    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ca1331-5cfd7b7d78-tttxp"]
Sep 18 14:09:37.812535 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:37.812509    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ca1331-5cfd7b7d78-tttxp" podUID="6273d451-8a5f-4c31-90be-ba6c0aa10853" containerName="payload-pre-processing" containerID="cri-o://cfebbcf4a50fd22b4b317e97ee6ccaa7227789174ff3126500c6f420211280cf" gracePeriod=30
Sep 18 14:09:37.882572 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:37.882535    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:09:37.985385 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:37.985306    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:09:38.055287 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:38.055249    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:09:38.057470 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:38.057451    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ca1331-66c7d84454-ztxcz"
Sep 18 14:09:38.084937 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:38.084907    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ca1331-5cfd7b7d78-tttxp"
Sep 18 14:09:38.219650 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:38.219613    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:09:38.234946 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:38.234919    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abf81e5c-1248-4af3-bdc0-4b2928175a62-metrics-certs\") pod \"abf81e5c-1248-4af3-bdc0-4b2928175a62\" (UID: \"abf81e5c-1248-4af3-bdc0-4b2928175a62\") "
Sep 18 14:09:38.235098 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:38.234950    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abf81e5c-1248-4af3-bdc0-4b2928175a62-openshift-service-ca\") pod \"abf81e5c-1248-4af3-bdc0-4b2928175a62\" (UID: \"abf81e5c-1248-4af3-bdc0-4b2928175a62\") "
Sep 18 14:09:38.235098 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:38.234978    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6273d451-8a5f-4c31-90be-ba6c0aa10853-plugins-config-volume\") pod \"6273d451-8a5f-4c31-90be-ba6c0aa10853\" (UID: \"6273d451-8a5f-4c31-90be-ba6c0aa10853\") "
Sep 18 14:09:38.235198 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:38.235179    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abf81e5c-1248-4af3-bdc0-4b2928175a62-maas-api-tls\") pod \"abf81e5c-1248-4af3-bdc0-4b2928175a62\" (UID: \"abf81e5c-1248-4af3-bdc0-4b2928175a62\") "
Sep 18 14:09:38.235273 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:38.235254    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7rqdx\" (UniqueName: \"kubernetes.io/projected/abf81e5c-1248-4af3-bdc0-4b2928175a62-kube-api-access-7rqdx\") pod \"abf81e5c-1248-4af3-bdc0-4b2928175a62\" (UID: \"abf81e5c-1248-4af3-bdc0-4b2928175a62\") "
Sep 18 14:09:38.235339 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:38.235283    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abf81e5c-1248-4af3-bdc0-4b2928175a62-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "abf81e5c-1248-4af3-bdc0-4b2928175a62" (UID: "abf81e5c-1248-4af3-bdc0-4b2928175a62"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:09:38.235397 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:38.235333    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6273d451-8a5f-4c31-90be-ba6c0aa10853-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6273d451-8a5f-4c31-90be-ba6c0aa10853" (UID: "6273d451-8a5f-4c31-90be-ba6c0aa10853"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:09:38.235397 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:38.235344    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wlx7w\" (UniqueName: \"kubernetes.io/projected/6273d451-8a5f-4c31-90be-ba6c0aa10853-kube-api-access-wlx7w\") pod \"6273d451-8a5f-4c31-90be-ba6c0aa10853\" (UID: \"6273d451-8a5f-4c31-90be-ba6c0aa10853\") "
Sep 18 14:09:38.235727 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:38.235695    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abf81e5c-1248-4af3-bdc0-4b2928175a62-openshift-service-ca\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:09:38.235727 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:38.235717    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6273d451-8a5f-4c31-90be-ba6c0aa10853-plugins-config-volume\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:09:38.237317 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:38.237293    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abf81e5c-1248-4af3-bdc0-4b2928175a62-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "abf81e5c-1248-4af3-bdc0-4b2928175a62" (UID: "abf81e5c-1248-4af3-bdc0-4b2928175a62"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:09:38.237423 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:38.237311    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6273d451-8a5f-4c31-90be-ba6c0aa10853-kube-api-access-wlx7w" (OuterVolumeSpecName: "kube-api-access-wlx7w") pod "6273d451-8a5f-4c31-90be-ba6c0aa10853" (UID: "6273d451-8a5f-4c31-90be-ba6c0aa10853"). InnerVolumeSpecName "kube-api-access-wlx7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:09:38.237681 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:38.237661    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abf81e5c-1248-4af3-bdc0-4b2928175a62-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "abf81e5c-1248-4af3-bdc0-4b2928175a62" (UID: "abf81e5c-1248-4af3-bdc0-4b2928175a62"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:09:38.237777 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:38.237675    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abf81e5c-1248-4af3-bdc0-4b2928175a62-kube-api-access-7rqdx" (OuterVolumeSpecName: "kube-api-access-7rqdx") pod "abf81e5c-1248-4af3-bdc0-4b2928175a62" (UID: "abf81e5c-1248-4af3-bdc0-4b2928175a62"). InnerVolumeSpecName "kube-api-access-7rqdx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:09:38.337189 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:38.337091    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abf81e5c-1248-4af3-bdc0-4b2928175a62-maas-api-tls\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:09:38.337189 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:38.337125    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rqdx\" (UniqueName: \"kubernetes.io/projected/abf81e5c-1248-4af3-bdc0-4b2928175a62-kube-api-access-7rqdx\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:09:38.337189 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:38.337136    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlx7w\" (UniqueName: \"kubernetes.io/projected/6273d451-8a5f-4c31-90be-ba6c0aa10853-kube-api-access-wlx7w\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:09:38.337189 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:38.337145    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abf81e5c-1248-4af3-bdc0-4b2928175a62-metrics-certs\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:09:38.426394 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:38.426363    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:09:38.503937 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:38.503901    2577 generic.go:356] "Generic (PLEG): container finished" podID="abf81e5c-1248-4af3-bdc0-4b2928175a62" containerID="35197cbd94506d71af6b03716394cae496b89d181ced804ec9afc3dcaeee6f24" exitCode=0
Sep 18 14:09:38.504140 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:38.503953    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ca1331-66c7d84454-ztxcz"
Sep 18 14:09:38.504140 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:38.503988    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ca1331-66c7d84454-ztxcz" event={"ID":"abf81e5c-1248-4af3-bdc0-4b2928175a62","Type":"ContainerDied","Data":"35197cbd94506d71af6b03716394cae496b89d181ced804ec9afc3dcaeee6f24"}
Sep 18 14:09:38.504140 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:38.504073    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ca1331-66c7d84454-ztxcz" event={"ID":"abf81e5c-1248-4af3-bdc0-4b2928175a62","Type":"ContainerDied","Data":"69b03e6947c3d4f1fe5d49de9070e540e5da81d6784800643c0d79f3cbbc61c0"}
Sep 18 14:09:38.504140 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:38.504101    2577 scope.go:117] "RemoveContainer" containerID="35197cbd94506d71af6b03716394cae496b89d181ced804ec9afc3dcaeee6f24"
Sep 18 14:09:38.505565 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:38.505539    2577 generic.go:356] "Generic (PLEG): container finished" podID="6273d451-8a5f-4c31-90be-ba6c0aa10853" containerID="cfebbcf4a50fd22b4b317e97ee6ccaa7227789174ff3126500c6f420211280cf" exitCode=0
Sep 18 14:09:38.505696 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:38.505568    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ca1331-5cfd7b7d78-tttxp" event={"ID":"6273d451-8a5f-4c31-90be-ba6c0aa10853","Type":"ContainerDied","Data":"cfebbcf4a50fd22b4b317e97ee6ccaa7227789174ff3126500c6f420211280cf"}
Sep 18 14:09:38.505696 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:38.505602    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ca1331-5cfd7b7d78-tttxp"
Sep 18 14:09:38.505696 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:38.505631    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ca1331-5cfd7b7d78-tttxp" event={"ID":"6273d451-8a5f-4c31-90be-ba6c0aa10853","Type":"ContainerDied","Data":"95e15eeb9d9a9e323f6435e000c19d45f0c891942f5988690e0b3600f0c31602"}
Sep 18 14:09:38.514000 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:38.513983    2577 scope.go:117] "RemoveContainer" containerID="35197cbd94506d71af6b03716394cae496b89d181ced804ec9afc3dcaeee6f24"
Sep 18 14:09:38.514315 ip-10-0-141-222 kubenswrapper[2577]: E0918 14:09:38.514294    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35197cbd94506d71af6b03716394cae496b89d181ced804ec9afc3dcaeee6f24\": container with ID starting with 35197cbd94506d71af6b03716394cae496b89d181ced804ec9afc3dcaeee6f24 not found: ID does not exist" containerID="35197cbd94506d71af6b03716394cae496b89d181ced804ec9afc3dcaeee6f24"
Sep 18 14:09:38.514382 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:38.514323    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35197cbd94506d71af6b03716394cae496b89d181ced804ec9afc3dcaeee6f24"} err="failed to get container status \"35197cbd94506d71af6b03716394cae496b89d181ced804ec9afc3dcaeee6f24\": rpc error: code = NotFound desc = could not find container \"35197cbd94506d71af6b03716394cae496b89d181ced804ec9afc3dcaeee6f24\": container with ID starting with 35197cbd94506d71af6b03716394cae496b89d181ced804ec9afc3dcaeee6f24 not found: ID does not exist"
Sep 18 14:09:38.514382 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:38.514361    2577 scope.go:117] "RemoveContainer" containerID="cfebbcf4a50fd22b4b317e97ee6ccaa7227789174ff3126500c6f420211280cf"
Sep 18 14:09:38.523087 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:38.523070    2577 scope.go:117] "RemoveContainer" containerID="cfebbcf4a50fd22b4b317e97ee6ccaa7227789174ff3126500c6f420211280cf"
Sep 18 14:09:38.523376 ip-10-0-141-222 kubenswrapper[2577]: E0918 14:09:38.523358    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfebbcf4a50fd22b4b317e97ee6ccaa7227789174ff3126500c6f420211280cf\": container with ID starting with cfebbcf4a50fd22b4b317e97ee6ccaa7227789174ff3126500c6f420211280cf not found: ID does not exist" containerID="cfebbcf4a50fd22b4b317e97ee6ccaa7227789174ff3126500c6f420211280cf"
Sep 18 14:09:38.523421 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:38.523385    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfebbcf4a50fd22b4b317e97ee6ccaa7227789174ff3126500c6f420211280cf"} err="failed to get container status \"cfebbcf4a50fd22b4b317e97ee6ccaa7227789174ff3126500c6f420211280cf\": rpc error: code = NotFound desc = could not find container \"cfebbcf4a50fd22b4b317e97ee6ccaa7227789174ff3126500c6f420211280cf\": container with ID starting with cfebbcf4a50fd22b4b317e97ee6ccaa7227789174ff3126500c6f420211280cf not found: ID does not exist"
Sep 18 14:09:38.529241 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:38.529219    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ca1331-5cfd7b7d78-tttxp"]
Sep 18 14:09:38.535447 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:38.535428    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ca1331-5cfd7b7d78-tttxp"]
Sep 18 14:09:38.549577 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:38.549537    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ca1331-66c7d84454-ztxcz"]
Sep 18 14:09:38.552001 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:38.551980    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ca1331-66c7d84454-ztxcz"]
Sep 18 14:09:38.892456 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:38.892417    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:09:39.108027 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:39.107995    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6273d451-8a5f-4c31-90be-ba6c0aa10853" path="/var/lib/kubelet/pods/6273d451-8a5f-4c31-90be-ba6c0aa10853/volumes"
Sep 18 14:09:39.108425 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:39.108412    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abf81e5c-1248-4af3-bdc0-4b2928175a62" path="/var/lib/kubelet/pods/abf81e5c-1248-4af3-bdc0-4b2928175a62/volumes"
Sep 18 14:09:39.758691 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:39.758652    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:09:40.419188 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:40.419160    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:09:41.191549 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:41.191514    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 14:09:41.256186 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:41.256154    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:09:41.554625 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:41.554529    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-377ca2/distinct-w4-377ca2-kserve-5f8574bffb-lhk8c"]
Sep 18 14:09:41.566639 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:41.566606    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-377ca2/premium-w4-377ca2-kserve-759c89bc45-4r9k4"]
Sep 18 14:09:41.575150 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:41.575094    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-377ca2/simulator-w4-377ca2-kserve-797f87bbfb-x986s"]
Sep 18 14:09:41.644482 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:41.644447    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:09:42.139123 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:42.139089    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:09:42.923964 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:42.923923    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:09:43.834206 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:43.834175    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:09:44.334859 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:44.334810    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7627-518dbc"]
Sep 18 14:09:44.335235 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:44.335198    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6273d451-8a5f-4c31-90be-ba6c0aa10853" containerName="payload-pre-processing"
Sep 18 14:09:44.335235 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:44.335210    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6273d451-8a5f-4c31-90be-ba6c0aa10853" containerName="payload-pre-processing"
Sep 18 14:09:44.335235 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:44.335231    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abf81e5c-1248-4af3-bdc0-4b2928175a62" containerName="maas-api"
Sep 18 14:09:44.335346 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:44.335237    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="abf81e5c-1248-4af3-bdc0-4b2928175a62" containerName="maas-api"
Sep 18 14:09:44.335346 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:44.335301    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6273d451-8a5f-4c31-90be-ba6c0aa10853" containerName="payload-pre-processing"
Sep 18 14:09:44.335346 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:44.335311    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="abf81e5c-1248-4af3-bdc0-4b2928175a62" containerName="maas-api"
Sep 18 14:09:44.339775 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:44.339754    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7627-518dbc"
Sep 18 14:09:44.342406 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:44.342383    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-knszx\""
Sep 18 14:09:44.348372 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:44.348346    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7627-518dbc"]
Sep 18 14:09:44.391082 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:44.391041    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drwjj\" (UniqueName: \"kubernetes.io/projected/7f40c363-5fbd-47dd-a3d7-8a62b5f8e5ad-kube-api-access-drwjj\") pod \"test-curl-7627-518dbc\" (UID: \"7f40c363-5fbd-47dd-a3d7-8a62b5f8e5ad\") " pod="opendatahub/test-curl-7627-518dbc"
Sep 18 14:09:44.492100 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:44.492062    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drwjj\" (UniqueName: \"kubernetes.io/projected/7f40c363-5fbd-47dd-a3d7-8a62b5f8e5ad-kube-api-access-drwjj\") pod \"test-curl-7627-518dbc\" (UID: \"7f40c363-5fbd-47dd-a3d7-8a62b5f8e5ad\") " pod="opendatahub/test-curl-7627-518dbc"
Sep 18 14:09:44.501144 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:44.501120    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drwjj\" (UniqueName: \"kubernetes.io/projected/7f40c363-5fbd-47dd-a3d7-8a62b5f8e5ad-kube-api-access-drwjj\") pod \"test-curl-7627-518dbc\" (UID: \"7f40c363-5fbd-47dd-a3d7-8a62b5f8e5ad\") " pod="opendatahub/test-curl-7627-518dbc"
Sep 18 14:09:44.650287 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:44.650249    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7627-518dbc"
Sep 18 14:09:44.929787 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:44.929755    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7627-518dbc"]
Sep 18 14:09:44.932216 ip-10-0-141-222 kubenswrapper[2577]: W0918 14:09:44.932190    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7f40c363_5fbd_47dd_a3d7_8a62b5f8e5ad.slice/crio-07913c075336fb784df31784ce67e1c431b9d52b659fe3358b57b669d838ff61 WatchSource:0}: Error finding container 07913c075336fb784df31784ce67e1c431b9d52b659fe3358b57b669d838ff61: Status 404 returned error can't find the container with id 07913c075336fb784df31784ce67e1c431b9d52b659fe3358b57b669d838ff61
Sep 18 14:09:45.174463 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:45.174422    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:09:45.251889 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:45.251790    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:09:45.454505 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:45.454470    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:09:45.548776 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:45.548679    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7627-518dbc" event={"ID":"7f40c363-5fbd-47dd-a3d7-8a62b5f8e5ad","Type":"ContainerStarted","Data":"ae9d33f129a0917d70b657602c1eb4a75bdac6b313a4118f2ff9fc1dccdfe01d"}
Sep 18 14:09:45.548776 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:45.548728    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7627-518dbc" event={"ID":"7f40c363-5fbd-47dd-a3d7-8a62b5f8e5ad","Type":"ContainerStarted","Data":"07913c075336fb784df31784ce67e1c431b9d52b659fe3358b57b669d838ff61"}
Sep 18 14:09:45.573337 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:45.573281    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7627-518dbc" podStartSLOduration=1.573265727 podStartE2EDuration="1.573265727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:09:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:09:45.563225274 +0000 UTC m=+1534.615884601" watchObservedRunningTime="2026-09-18 14:09:45.573265727 +0000 UTC m=+1534.625925054"
Sep 18 14:09:45.870375 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:45.870326    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:09:46.170246 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:46.170150    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7627-518dbc"]
Sep 18 14:09:46.174436 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:46.174404    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7627-518dbc"]
Sep 18 14:09:46.554663 ip-10-0-141-222 kubenswrapper[2577]: E0918 14:09:46.554558    2577 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7627-518dbc" container="test-curl-7627-518dbc"
Sep 18 14:09:46.720331 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:46.720298    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:09:47.534296 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:47.534261    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:09:47.561715 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:47.561671    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7627-518dbc" podUID="7f40c363-5fbd-47dd-a3d7-8a62b5f8e5ad" containerName="test-curl-7627-518dbc" containerID="cri-o://ae9d33f129a0917d70b657602c1eb4a75bdac6b313a4118f2ff9fc1dccdfe01d" gracePeriod=2
Sep 18 14:09:47.613354 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:47.613317    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:09:48.204468 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:48.204431    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:09:48.704276 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:48.704241    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:09:48.927958 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:48.927922    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:09:49.742063 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:49.742038    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7627-518dbc"
Sep 18 14:09:49.769356 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:49.769326    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:09:49.839461 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:49.839430    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-drwjj\" (UniqueName: \"kubernetes.io/projected/7f40c363-5fbd-47dd-a3d7-8a62b5f8e5ad-kube-api-access-drwjj\") pod \"7f40c363-5fbd-47dd-a3d7-8a62b5f8e5ad\" (UID: \"7f40c363-5fbd-47dd-a3d7-8a62b5f8e5ad\") "
Sep 18 14:09:49.841611 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:49.841543    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f40c363-5fbd-47dd-a3d7-8a62b5f8e5ad-kube-api-access-drwjj" (OuterVolumeSpecName: "kube-api-access-drwjj") pod "7f40c363-5fbd-47dd-a3d7-8a62b5f8e5ad" (UID: "7f40c363-5fbd-47dd-a3d7-8a62b5f8e5ad"). InnerVolumeSpecName "kube-api-access-drwjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:09:49.940327 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:49.940291    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drwjj\" (UniqueName: \"kubernetes.io/projected/7f40c363-5fbd-47dd-a3d7-8a62b5f8e5ad-kube-api-access-drwjj\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:09:50.551843 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:50.551785    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:09:50.586539 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:50.586501    2577 generic.go:356] "Generic (PLEG): container finished" podID="7f40c363-5fbd-47dd-a3d7-8a62b5f8e5ad" containerID="ae9d33f129a0917d70b657602c1eb4a75bdac6b313a4118f2ff9fc1dccdfe01d" exitCode=137
Sep 18 14:09:50.586713 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:50.586556    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7627-518dbc"
Sep 18 14:09:50.586713 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:50.586567    2577 scope.go:117] "RemoveContainer" containerID="ae9d33f129a0917d70b657602c1eb4a75bdac6b313a4118f2ff9fc1dccdfe01d"
Sep 18 14:09:50.598084 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:50.598057    2577 scope.go:117] "RemoveContainer" containerID="ae9d33f129a0917d70b657602c1eb4a75bdac6b313a4118f2ff9fc1dccdfe01d"
Sep 18 14:09:50.598416 ip-10-0-141-222 kubenswrapper[2577]: E0918 14:09:50.598397    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae9d33f129a0917d70b657602c1eb4a75bdac6b313a4118f2ff9fc1dccdfe01d\": container with ID starting with ae9d33f129a0917d70b657602c1eb4a75bdac6b313a4118f2ff9fc1dccdfe01d not found: ID does not exist" containerID="ae9d33f129a0917d70b657602c1eb4a75bdac6b313a4118f2ff9fc1dccdfe01d"
Sep 18 14:09:50.598468 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:50.598425    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae9d33f129a0917d70b657602c1eb4a75bdac6b313a4118f2ff9fc1dccdfe01d"} err="failed to get container status \"ae9d33f129a0917d70b657602c1eb4a75bdac6b313a4118f2ff9fc1dccdfe01d\": rpc error: code = NotFound desc = could not find container \"ae9d33f129a0917d70b657602c1eb4a75bdac6b313a4118f2ff9fc1dccdfe01d\": container with ID starting with ae9d33f129a0917d70b657602c1eb4a75bdac6b313a4118f2ff9fc1dccdfe01d not found: ID does not exist"
Sep 18 14:09:50.600240 ip-10-0-141-222 kubenswrapper[2577]: E0918 14:09:50.600223    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7627-518dbc\" is forbidden: User \"system:node:ip-10-0-141-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-222.ec2.internal' and this object" podUID="7f40c363-5fbd-47dd-a3d7-8a62b5f8e5ad" pod="opendatahub/test-curl-7627-518dbc"
Sep 18 14:09:51.108130 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:51.108096    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f40c363-5fbd-47dd-a3d7-8a62b5f8e5ad" path="/var/lib/kubelet/pods/7f40c363-5fbd-47dd-a3d7-8a62b5f8e5ad/volumes"
Sep 18 14:09:55.408523 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:55.408492    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:09:55.412585 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:55.412560    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-377ca2-29829000-rlwmp"]
Sep 18 14:09:55.422336 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:55.416670    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-377ca2-29829000-rlwmp"]
Sep 18 14:09:55.504190 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:55.504161    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:09:55.605720 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:55.605683    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:09:55.706068 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:55.705983    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:09:55.984771 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:55.984677    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:09:56.576238 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:56.576173    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-377ca2/distinct-w4-377ca2-kserve-5f8574bffb-lhk8c" podUID="a58f638f-72bd-4d81-bdba-5b58e146ae90" containerName="main" containerID="cri-o://e0d2362d7c83ad74f0d67972bea11cb73e8ccd3f723d9d9b43b0b290f236fd84" gracePeriod=45
Sep 18 14:09:56.587646 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:56.587610    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-377ca2/premium-w4-377ca2-kserve-759c89bc45-4r9k4" podUID="d81fc1dd-14e8-4f3c-9cfd-56731b661718" containerName="main" containerID="cri-o://4788c5e924a73b728097dd08d088131ceb7dcd156be8470d919aa3be3918ac78" gracePeriod=45
Sep 18 14:09:56.594245 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:56.594194    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-377ca2/simulator-w4-377ca2-kserve-797f87bbfb-x986s" podUID="e3a3b935-cb2b-4a63-ba38-2906228f737a" containerName="main" containerID="cri-o://4e3096a5929537d65614b7dbfb62bf40f0c6305620cbba1e4dedcaa41d6ae321" gracePeriod=45
Sep 18 14:09:56.631270 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:56.631237    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:09:56.895336 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:56.895306    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-377ca2/distinct-w4-377ca2-kserve-5f8574bffb-lhk8c"
Sep 18 14:09:56.898966 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:56.898938    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-377ca2/simulator-w4-377ca2-kserve-797f87bbfb-x986s"
Sep 18 14:09:56.906198 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:56.906174    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-377ca2/premium-w4-377ca2-kserve-759c89bc45-4r9k4"
Sep 18 14:09:57.001086 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.001052    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d81fc1dd-14e8-4f3c-9cfd-56731b661718-model-cache\") pod \"d81fc1dd-14e8-4f3c-9cfd-56731b661718\" (UID: \"d81fc1dd-14e8-4f3c-9cfd-56731b661718\") "
Sep 18 14:09:57.001086 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.001089    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a58f638f-72bd-4d81-bdba-5b58e146ae90-model-cache\") pod \"a58f638f-72bd-4d81-bdba-5b58e146ae90\" (UID: \"a58f638f-72bd-4d81-bdba-5b58e146ae90\") "
Sep 18 14:09:57.001335 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.001108    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3a3b935-cb2b-4a63-ba38-2906228f737a-tmp-dir\") pod \"e3a3b935-cb2b-4a63-ba38-2906228f737a\" (UID: \"e3a3b935-cb2b-4a63-ba38-2906228f737a\") "
Sep 18 14:09:57.001335 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.001128    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3a3b935-cb2b-4a63-ba38-2906228f737a-tls-certs\") pod \"e3a3b935-cb2b-4a63-ba38-2906228f737a\" (UID: \"e3a3b935-cb2b-4a63-ba38-2906228f737a\") "
Sep 18 14:09:57.001335 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.001152    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a58f638f-72bd-4d81-bdba-5b58e146ae90-tmp-dir\") pod \"a58f638f-72bd-4d81-bdba-5b58e146ae90\" (UID: \"a58f638f-72bd-4d81-bdba-5b58e146ae90\") "
Sep 18 14:09:57.001335 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.001191    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d81fc1dd-14e8-4f3c-9cfd-56731b661718-home\") pod \"d81fc1dd-14e8-4f3c-9cfd-56731b661718\" (UID: \"d81fc1dd-14e8-4f3c-9cfd-56731b661718\") "
Sep 18 14:09:57.001335 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.001217    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3a3b935-cb2b-4a63-ba38-2906228f737a-model-cache\") pod \"e3a3b935-cb2b-4a63-ba38-2906228f737a\" (UID: \"e3a3b935-cb2b-4a63-ba38-2906228f737a\") "
Sep 18 14:09:57.001335 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.001240    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a58f638f-72bd-4d81-bdba-5b58e146ae90-dshm\") pod \"a58f638f-72bd-4d81-bdba-5b58e146ae90\" (UID: \"a58f638f-72bd-4d81-bdba-5b58e146ae90\") "
Sep 18 14:09:57.001335 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.001316    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d81fc1dd-14e8-4f3c-9cfd-56731b661718-dshm\") pod \"d81fc1dd-14e8-4f3c-9cfd-56731b661718\" (UID: \"d81fc1dd-14e8-4f3c-9cfd-56731b661718\") "
Sep 18 14:09:57.001690 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.001343    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5jgfc\" (UniqueName: \"kubernetes.io/projected/e3a3b935-cb2b-4a63-ba38-2906228f737a-kube-api-access-5jgfc\") pod \"e3a3b935-cb2b-4a63-ba38-2906228f737a\" (UID: \"e3a3b935-cb2b-4a63-ba38-2906228f737a\") "
Sep 18 14:09:57.001690 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.001376    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-krlnk\" (UniqueName: \"kubernetes.io/projected/d81fc1dd-14e8-4f3c-9cfd-56731b661718-kube-api-access-krlnk\") pod \"d81fc1dd-14e8-4f3c-9cfd-56731b661718\" (UID: \"d81fc1dd-14e8-4f3c-9cfd-56731b661718\") "
Sep 18 14:09:57.001690 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.001388    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3a3b935-cb2b-4a63-ba38-2906228f737a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e3a3b935-cb2b-4a63-ba38-2906228f737a" (UID: "e3a3b935-cb2b-4a63-ba38-2906228f737a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:09:57.001690 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.001388    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d81fc1dd-14e8-4f3c-9cfd-56731b661718-model-cache" (OuterVolumeSpecName: "model-cache") pod "d81fc1dd-14e8-4f3c-9cfd-56731b661718" (UID: "d81fc1dd-14e8-4f3c-9cfd-56731b661718"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:09:57.001690 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.001400    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3a3b935-cb2b-4a63-ba38-2906228f737a-home\") pod \"e3a3b935-cb2b-4a63-ba38-2906228f737a\" (UID: \"e3a3b935-cb2b-4a63-ba38-2906228f737a\") "
Sep 18 14:09:57.001690 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.001512    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a58f638f-72bd-4d81-bdba-5b58e146ae90-home\") pod \"a58f638f-72bd-4d81-bdba-5b58e146ae90\" (UID: \"a58f638f-72bd-4d81-bdba-5b58e146ae90\") "
Sep 18 14:09:57.001690 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.001560    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d81fc1dd-14e8-4f3c-9cfd-56731b661718-tmp-dir\") pod \"d81fc1dd-14e8-4f3c-9cfd-56731b661718\" (UID: \"d81fc1dd-14e8-4f3c-9cfd-56731b661718\") "
Sep 18 14:09:57.001690 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.001589    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d81fc1dd-14e8-4f3c-9cfd-56731b661718-tls-certs\") pod \"d81fc1dd-14e8-4f3c-9cfd-56731b661718\" (UID: \"d81fc1dd-14e8-4f3c-9cfd-56731b661718\") "
Sep 18 14:09:57.002293 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.002087    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3a3b935-cb2b-4a63-ba38-2906228f737a-dshm\") pod \"e3a3b935-cb2b-4a63-ba38-2906228f737a\" (UID: \"e3a3b935-cb2b-4a63-ba38-2906228f737a\") "
Sep 18 14:09:57.002293 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.002122    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a58f638f-72bd-4d81-bdba-5b58e146ae90-tls-certs\") pod \"a58f638f-72bd-4d81-bdba-5b58e146ae90\" (UID: \"a58f638f-72bd-4d81-bdba-5b58e146ae90\") "
Sep 18 14:09:57.002293 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.002162    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lt6l2\" (UniqueName: \"kubernetes.io/projected/a58f638f-72bd-4d81-bdba-5b58e146ae90-kube-api-access-lt6l2\") pod \"a58f638f-72bd-4d81-bdba-5b58e146ae90\" (UID: \"a58f638f-72bd-4d81-bdba-5b58e146ae90\") "
Sep 18 14:09:57.002460 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.001392    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a58f638f-72bd-4d81-bdba-5b58e146ae90-model-cache" (OuterVolumeSpecName: "model-cache") pod "a58f638f-72bd-4d81-bdba-5b58e146ae90" (UID: "a58f638f-72bd-4d81-bdba-5b58e146ae90"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:09:57.002460 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.001669    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d81fc1dd-14e8-4f3c-9cfd-56731b661718-home" (OuterVolumeSpecName: "home") pod "d81fc1dd-14e8-4f3c-9cfd-56731b661718" (UID: "d81fc1dd-14e8-4f3c-9cfd-56731b661718"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:09:57.002460 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.001841    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a58f638f-72bd-4d81-bdba-5b58e146ae90-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a58f638f-72bd-4d81-bdba-5b58e146ae90" (UID: "a58f638f-72bd-4d81-bdba-5b58e146ae90"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:09:57.002460 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.002129    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3a3b935-cb2b-4a63-ba38-2906228f737a-home" (OuterVolumeSpecName: "home") pod "e3a3b935-cb2b-4a63-ba38-2906228f737a" (UID: "e3a3b935-cb2b-4a63-ba38-2906228f737a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:09:57.002460 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.002306    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d81fc1dd-14e8-4f3c-9cfd-56731b661718-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d81fc1dd-14e8-4f3c-9cfd-56731b661718" (UID: "d81fc1dd-14e8-4f3c-9cfd-56731b661718"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:09:57.002735 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.002517    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3a3b935-cb2b-4a63-ba38-2906228f737a-model-cache" (OuterVolumeSpecName: "model-cache") pod "e3a3b935-cb2b-4a63-ba38-2906228f737a" (UID: "e3a3b935-cb2b-4a63-ba38-2906228f737a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:09:57.002735 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.002525    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a58f638f-72bd-4d81-bdba-5b58e146ae90-home" (OuterVolumeSpecName: "home") pod "a58f638f-72bd-4d81-bdba-5b58e146ae90" (UID: "a58f638f-72bd-4d81-bdba-5b58e146ae90"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:09:57.003442 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.003256    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3a3b935-cb2b-4a63-ba38-2906228f737a-home\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:09:57.003442 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.003283    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a58f638f-72bd-4d81-bdba-5b58e146ae90-home\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:09:57.003442 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.003297    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d81fc1dd-14e8-4f3c-9cfd-56731b661718-tmp-dir\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:09:57.003442 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.003313    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d81fc1dd-14e8-4f3c-9cfd-56731b661718-model-cache\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:09:57.003442 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.003327    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a58f638f-72bd-4d81-bdba-5b58e146ae90-model-cache\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:09:57.003442 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.003342    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3a3b935-cb2b-4a63-ba38-2906228f737a-tmp-dir\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:09:57.003442 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.003354    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a58f638f-72bd-4d81-bdba-5b58e146ae90-tmp-dir\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:09:57.003442 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.003367    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d81fc1dd-14e8-4f3c-9cfd-56731b661718-home\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:09:57.003442 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.003379    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3a3b935-cb2b-4a63-ba38-2906228f737a-model-cache\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:09:57.004456 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.004132    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a58f638f-72bd-4d81-bdba-5b58e146ae90-dshm" (OuterVolumeSpecName: "dshm") pod "a58f638f-72bd-4d81-bdba-5b58e146ae90" (UID: "a58f638f-72bd-4d81-bdba-5b58e146ae90"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:09:57.004456 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.004415    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3a3b935-cb2b-4a63-ba38-2906228f737a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e3a3b935-cb2b-4a63-ba38-2906228f737a" (UID: "e3a3b935-cb2b-4a63-ba38-2906228f737a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:09:57.004456 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.004429    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3a3b935-cb2b-4a63-ba38-2906228f737a-kube-api-access-5jgfc" (OuterVolumeSpecName: "kube-api-access-5jgfc") pod "e3a3b935-cb2b-4a63-ba38-2906228f737a" (UID: "e3a3b935-cb2b-4a63-ba38-2906228f737a"). InnerVolumeSpecName "kube-api-access-5jgfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:09:57.004648 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.004587    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d81fc1dd-14e8-4f3c-9cfd-56731b661718-dshm" (OuterVolumeSpecName: "dshm") pod "d81fc1dd-14e8-4f3c-9cfd-56731b661718" (UID: "d81fc1dd-14e8-4f3c-9cfd-56731b661718"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:09:57.004940 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.004919    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a58f638f-72bd-4d81-bdba-5b58e146ae90-kube-api-access-lt6l2" (OuterVolumeSpecName: "kube-api-access-lt6l2") pod "a58f638f-72bd-4d81-bdba-5b58e146ae90" (UID: "a58f638f-72bd-4d81-bdba-5b58e146ae90"). InnerVolumeSpecName "kube-api-access-lt6l2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:09:57.005533 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.005508    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a58f638f-72bd-4d81-bdba-5b58e146ae90-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a58f638f-72bd-4d81-bdba-5b58e146ae90" (UID: "a58f638f-72bd-4d81-bdba-5b58e146ae90"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:09:57.005631 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.005596    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d81fc1dd-14e8-4f3c-9cfd-56731b661718-kube-api-access-krlnk" (OuterVolumeSpecName: "kube-api-access-krlnk") pod "d81fc1dd-14e8-4f3c-9cfd-56731b661718" (UID: "d81fc1dd-14e8-4f3c-9cfd-56731b661718"). InnerVolumeSpecName "kube-api-access-krlnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:09:57.005694 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.005678    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3a3b935-cb2b-4a63-ba38-2906228f737a-dshm" (OuterVolumeSpecName: "dshm") pod "e3a3b935-cb2b-4a63-ba38-2906228f737a" (UID: "e3a3b935-cb2b-4a63-ba38-2906228f737a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:09:57.005834 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.005801    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d81fc1dd-14e8-4f3c-9cfd-56731b661718-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d81fc1dd-14e8-4f3c-9cfd-56731b661718" (UID: "d81fc1dd-14e8-4f3c-9cfd-56731b661718"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:09:57.104435 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.104400    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d81fc1dd-14e8-4f3c-9cfd-56731b661718-dshm\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:09:57.104435 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.104430    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jgfc\" (UniqueName: \"kubernetes.io/projected/e3a3b935-cb2b-4a63-ba38-2906228f737a-kube-api-access-5jgfc\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:09:57.104435 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.104441    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krlnk\" (UniqueName: \"kubernetes.io/projected/d81fc1dd-14e8-4f3c-9cfd-56731b661718-kube-api-access-krlnk\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:09:57.104693 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.104451    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d81fc1dd-14e8-4f3c-9cfd-56731b661718-tls-certs\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:09:57.104693 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.104462    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3a3b935-cb2b-4a63-ba38-2906228f737a-dshm\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:09:57.104693 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.104470    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a58f638f-72bd-4d81-bdba-5b58e146ae90-tls-certs\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:09:57.104693 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.104477    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lt6l2\" (UniqueName: \"kubernetes.io/projected/a58f638f-72bd-4d81-bdba-5b58e146ae90-kube-api-access-lt6l2\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:09:57.104693 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.104485    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3a3b935-cb2b-4a63-ba38-2906228f737a-tls-certs\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:09:57.104693 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.104493    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a58f638f-72bd-4d81-bdba-5b58e146ae90-dshm\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:09:57.108427 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.108363    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8df24333-37c6-435f-b858-de1708adbf0f" path="/var/lib/kubelet/pods/8df24333-37c6-435f-b858-de1708adbf0f/volumes"
Sep 18 14:09:57.342624 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.342591    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:09:57.629180 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.629142    2577 generic.go:356] "Generic (PLEG): container finished" podID="a58f638f-72bd-4d81-bdba-5b58e146ae90" containerID="e0d2362d7c83ad74f0d67972bea11cb73e8ccd3f723d9d9b43b0b290f236fd84" exitCode=0
Sep 18 14:09:57.629180 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.629187    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-377ca2/distinct-w4-377ca2-kserve-5f8574bffb-lhk8c"
Sep 18 14:09:57.629729 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.629222    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-377ca2/distinct-w4-377ca2-kserve-5f8574bffb-lhk8c" event={"ID":"a58f638f-72bd-4d81-bdba-5b58e146ae90","Type":"ContainerDied","Data":"e0d2362d7c83ad74f0d67972bea11cb73e8ccd3f723d9d9b43b0b290f236fd84"}
Sep 18 14:09:57.629729 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.629277    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-377ca2/distinct-w4-377ca2-kserve-5f8574bffb-lhk8c" event={"ID":"a58f638f-72bd-4d81-bdba-5b58e146ae90","Type":"ContainerDied","Data":"a5138d9ba5458552f482269e49c0c62baac0fd457989531a8fb7b786842d3fc0"}
Sep 18 14:09:57.629729 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.629302    2577 scope.go:117] "RemoveContainer" containerID="e0d2362d7c83ad74f0d67972bea11cb73e8ccd3f723d9d9b43b0b290f236fd84"
Sep 18 14:09:57.630855 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.630814    2577 generic.go:356] "Generic (PLEG): container finished" podID="e3a3b935-cb2b-4a63-ba38-2906228f737a" containerID="4e3096a5929537d65614b7dbfb62bf40f0c6305620cbba1e4dedcaa41d6ae321" exitCode=0
Sep 18 14:09:57.630969 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.630866    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-377ca2/simulator-w4-377ca2-kserve-797f87bbfb-x986s"
Sep 18 14:09:57.630969 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.630918    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-377ca2/simulator-w4-377ca2-kserve-797f87bbfb-x986s" event={"ID":"e3a3b935-cb2b-4a63-ba38-2906228f737a","Type":"ContainerDied","Data":"4e3096a5929537d65614b7dbfb62bf40f0c6305620cbba1e4dedcaa41d6ae321"}
Sep 18 14:09:57.630969 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.630960    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-377ca2/simulator-w4-377ca2-kserve-797f87bbfb-x986s" event={"ID":"e3a3b935-cb2b-4a63-ba38-2906228f737a","Type":"ContainerDied","Data":"624d0185f943de016d52278dbebd66ba174e2135f352c17f8a33814b7fe80f09"}
Sep 18 14:09:57.632957 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.632890    2577 generic.go:356] "Generic (PLEG): container finished" podID="d81fc1dd-14e8-4f3c-9cfd-56731b661718" containerID="4788c5e924a73b728097dd08d088131ceb7dcd156be8470d919aa3be3918ac78" exitCode=0
Sep 18 14:09:57.632957 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.632921    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-377ca2/premium-w4-377ca2-kserve-759c89bc45-4r9k4" event={"ID":"d81fc1dd-14e8-4f3c-9cfd-56731b661718","Type":"ContainerDied","Data":"4788c5e924a73b728097dd08d088131ceb7dcd156be8470d919aa3be3918ac78"}
Sep 18 14:09:57.632957 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.632945    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-377ca2/premium-w4-377ca2-kserve-759c89bc45-4r9k4" event={"ID":"d81fc1dd-14e8-4f3c-9cfd-56731b661718","Type":"ContainerDied","Data":"582908eae9e74b73f8983caa5f5bcc1571877706115dc729e106d09e0ff06e1f"}
Sep 18 14:09:57.632957 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.632953    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-377ca2/premium-w4-377ca2-kserve-759c89bc45-4r9k4"
Sep 18 14:09:57.640073 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.640055    2577 scope.go:117] "RemoveContainer" containerID="e0d2362d7c83ad74f0d67972bea11cb73e8ccd3f723d9d9b43b0b290f236fd84"
Sep 18 14:09:57.640455 ip-10-0-141-222 kubenswrapper[2577]: E0918 14:09:57.640377    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0d2362d7c83ad74f0d67972bea11cb73e8ccd3f723d9d9b43b0b290f236fd84\": container with ID starting with e0d2362d7c83ad74f0d67972bea11cb73e8ccd3f723d9d9b43b0b290f236fd84 not found: ID does not exist" containerID="e0d2362d7c83ad74f0d67972bea11cb73e8ccd3f723d9d9b43b0b290f236fd84"
Sep 18 14:09:57.640455 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.640416    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0d2362d7c83ad74f0d67972bea11cb73e8ccd3f723d9d9b43b0b290f236fd84"} err="failed to get container status \"e0d2362d7c83ad74f0d67972bea11cb73e8ccd3f723d9d9b43b0b290f236fd84\": rpc error: code = NotFound desc = could not find container \"e0d2362d7c83ad74f0d67972bea11cb73e8ccd3f723d9d9b43b0b290f236fd84\": container with ID starting with e0d2362d7c83ad74f0d67972bea11cb73e8ccd3f723d9d9b43b0b290f236fd84 not found: ID does not exist"
Sep 18 14:09:57.640455 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.640438    2577 scope.go:117] "RemoveContainer" containerID="4e3096a5929537d65614b7dbfb62bf40f0c6305620cbba1e4dedcaa41d6ae321"
Sep 18 14:09:57.649897 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.649877    2577 scope.go:117] "RemoveContainer" containerID="4e3096a5929537d65614b7dbfb62bf40f0c6305620cbba1e4dedcaa41d6ae321"
Sep 18 14:09:57.650151 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.650132    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-377ca2/distinct-w4-377ca2-kserve-5f8574bffb-lhk8c"]
Sep 18 14:09:57.650194 ip-10-0-141-222 kubenswrapper[2577]: E0918 14:09:57.650167    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e3096a5929537d65614b7dbfb62bf40f0c6305620cbba1e4dedcaa41d6ae321\": container with ID starting with 4e3096a5929537d65614b7dbfb62bf40f0c6305620cbba1e4dedcaa41d6ae321 not found: ID does not exist" containerID="4e3096a5929537d65614b7dbfb62bf40f0c6305620cbba1e4dedcaa41d6ae321"
Sep 18 14:09:57.650231 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.650193    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e3096a5929537d65614b7dbfb62bf40f0c6305620cbba1e4dedcaa41d6ae321"} err="failed to get container status \"4e3096a5929537d65614b7dbfb62bf40f0c6305620cbba1e4dedcaa41d6ae321\": rpc error: code = NotFound desc = could not find container \"4e3096a5929537d65614b7dbfb62bf40f0c6305620cbba1e4dedcaa41d6ae321\": container with ID starting with 4e3096a5929537d65614b7dbfb62bf40f0c6305620cbba1e4dedcaa41d6ae321 not found: ID does not exist"
Sep 18 14:09:57.650231 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.650210    2577 scope.go:117] "RemoveContainer" containerID="4788c5e924a73b728097dd08d088131ceb7dcd156be8470d919aa3be3918ac78"
Sep 18 14:09:57.654206 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.654178    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-377ca2/distinct-w4-377ca2-kserve-5f8574bffb-lhk8c"]
Sep 18 14:09:57.659003 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.658986    2577 scope.go:117] "RemoveContainer" containerID="4788c5e924a73b728097dd08d088131ceb7dcd156be8470d919aa3be3918ac78"
Sep 18 14:09:57.659259 ip-10-0-141-222 kubenswrapper[2577]: E0918 14:09:57.659240    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4788c5e924a73b728097dd08d088131ceb7dcd156be8470d919aa3be3918ac78\": container with ID starting with 4788c5e924a73b728097dd08d088131ceb7dcd156be8470d919aa3be3918ac78 not found: ID does not exist" containerID="4788c5e924a73b728097dd08d088131ceb7dcd156be8470d919aa3be3918ac78"
Sep 18 14:09:57.659308 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.659267    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4788c5e924a73b728097dd08d088131ceb7dcd156be8470d919aa3be3918ac78"} err="failed to get container status \"4788c5e924a73b728097dd08d088131ceb7dcd156be8470d919aa3be3918ac78\": rpc error: code = NotFound desc = could not find container \"4788c5e924a73b728097dd08d088131ceb7dcd156be8470d919aa3be3918ac78\": container with ID starting with 4788c5e924a73b728097dd08d088131ceb7dcd156be8470d919aa3be3918ac78 not found: ID does not exist"
Sep 18 14:09:57.671366 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.671338    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-377ca2/simulator-w4-377ca2-kserve-797f87bbfb-x986s"]
Sep 18 14:09:57.675747 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.675718    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-377ca2/simulator-w4-377ca2-kserve-797f87bbfb-x986s"]
Sep 18 14:09:57.686209 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.686169    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-377ca2/premium-w4-377ca2-kserve-759c89bc45-4r9k4"]
Sep 18 14:09:57.689305 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:57.689275    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-377ca2/premium-w4-377ca2-kserve-759c89bc45-4r9k4"]
Sep 18 14:09:58.160457 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:58.160422    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:09:58.843717 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:58.843679    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:09:59.112164 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:59.112079    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a58f638f-72bd-4d81-bdba-5b58e146ae90" path="/var/lib/kubelet/pods/a58f638f-72bd-4d81-bdba-5b58e146ae90/volumes"
Sep 18 14:09:59.112559 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:59.112540    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d81fc1dd-14e8-4f3c-9cfd-56731b661718" path="/var/lib/kubelet/pods/d81fc1dd-14e8-4f3c-9cfd-56731b661718/volumes"
Sep 18 14:09:59.112923 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:59.112910    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3a3b935-cb2b-4a63-ba38-2906228f737a" path="/var/lib/kubelet/pods/e3a3b935-cb2b-4a63-ba38-2906228f737a/volumes"
Sep 18 14:09:59.735396 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:09:59.735359    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:10:02.134966 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:02.134929    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:10:02.239308 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:02.239274    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:10:02.408926 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:02.408792    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:10:05.500483 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:05.500452    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:10:05.526934 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:05.526903    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-eb3054-b95bb9dcd-qqqqk"]
Sep 18 14:10:05.527143 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:05.527120    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-eb3054-b95bb9dcd-qqqqk" podUID="0611c055-7a5e-4503-b076-22ac855233ac" containerName="payload-pre-processing" containerID="cri-o://161d97361c3c2e83d65541f0f11912420e5d6b66ef2192c3035a5da2bb2237a1" gracePeriod=30
Sep 18 14:10:05.569535 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:05.569501    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:10:05.670922 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:05.670866    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:10:05.682441 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:05.682409    2577 generic.go:356] "Generic (PLEG): container finished" podID="0611c055-7a5e-4503-b076-22ac855233ac" containerID="161d97361c3c2e83d65541f0f11912420e5d6b66ef2192c3035a5da2bb2237a1" exitCode=0
Sep 18 14:10:05.682642 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:05.682487    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-eb3054-b95bb9dcd-qqqqk" event={"ID":"0611c055-7a5e-4503-b076-22ac855233ac","Type":"ContainerDied","Data":"161d97361c3c2e83d65541f0f11912420e5d6b66ef2192c3035a5da2bb2237a1"}
Sep 18 14:10:05.742567 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:05.742533    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:10:05.789291 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:05.789263    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-eb3054-b95bb9dcd-qqqqk"
Sep 18 14:10:05.885642 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:05.885603    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w7ssj\" (UniqueName: \"kubernetes.io/projected/0611c055-7a5e-4503-b076-22ac855233ac-kube-api-access-w7ssj\") pod \"0611c055-7a5e-4503-b076-22ac855233ac\" (UID: \"0611c055-7a5e-4503-b076-22ac855233ac\") "
Sep 18 14:10:05.885862 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:05.885658    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0611c055-7a5e-4503-b076-22ac855233ac-plugins-config-volume\") pod \"0611c055-7a5e-4503-b076-22ac855233ac\" (UID: \"0611c055-7a5e-4503-b076-22ac855233ac\") "
Sep 18 14:10:05.886041 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:05.886018    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0611c055-7a5e-4503-b076-22ac855233ac-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0611c055-7a5e-4503-b076-22ac855233ac" (UID: "0611c055-7a5e-4503-b076-22ac855233ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:10:05.887762 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:05.887738    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0611c055-7a5e-4503-b076-22ac855233ac-kube-api-access-w7ssj" (OuterVolumeSpecName: "kube-api-access-w7ssj") pod "0611c055-7a5e-4503-b076-22ac855233ac" (UID: "0611c055-7a5e-4503-b076-22ac855233ac"). InnerVolumeSpecName "kube-api-access-w7ssj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:10:05.986800 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:05.986758    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7ssj\" (UniqueName: \"kubernetes.io/projected/0611c055-7a5e-4503-b076-22ac855233ac-kube-api-access-w7ssj\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:10:05.986800 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:05.986793    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0611c055-7a5e-4503-b076-22ac855233ac-plugins-config-volume\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:10:06.045600 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:06.045514    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:10:06.124483 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:06.124438    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:10:06.689358 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:06.689321    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-eb3054-b95bb9dcd-qqqqk"
Sep 18 14:10:06.689850 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:06.689356    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-eb3054-b95bb9dcd-qqqqk" event={"ID":"0611c055-7a5e-4503-b076-22ac855233ac","Type":"ContainerDied","Data":"706c11c226c94a165c40aec484d211535bd541e81b5741834436bbba5b1d9f8a"}
Sep 18 14:10:06.689850 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:06.689431    2577 scope.go:117] "RemoveContainer" containerID="161d97361c3c2e83d65541f0f11912420e5d6b66ef2192c3035a5da2bb2237a1"
Sep 18 14:10:06.721911 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:06.719893    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-eb3054-b95bb9dcd-qqqqk"]
Sep 18 14:10:06.725738 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:06.725710    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-eb3054-b95bb9dcd-qqqqk"]
Sep 18 14:10:06.807648 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:06.807613    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:10:07.107343 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:07.107304    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0611c055-7a5e-4503-b076-22ac855233ac" path="/var/lib/kubelet/pods/0611c055-7a5e-4503-b076-22ac855233ac/volumes"
Sep 18 14:10:07.882691 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:07.882653    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:10:08.201621 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:08.201518    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 14:10:08.725404 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:08.725367    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:10:09.425366 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:09.425332    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:10:12.231747 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:12.231690    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:10:12.306188 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:12.306160    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:10:12.503585 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:12.503495    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:10:13.138133 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:13.138101    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:10:13.398323 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:13.398237    2577 scope.go:117] "RemoveContainer" containerID="a6029672d44e4c1954bbd60e36c6546a24ad26322c8f547b948d9e4b8029bba1"
Sep 18 14:10:13.818481 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:13.818389    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:10:14.460597 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:14.460568    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:10:15.087579 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:15.087544    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-783872/sub-test-model-783872-kserve-59c9f8df9c-j27z7"]
Sep 18 14:10:15.088149 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:15.088127    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d81fc1dd-14e8-4f3c-9cfd-56731b661718" containerName="main"
Sep 18 14:10:15.088211 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:15.088154    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d81fc1dd-14e8-4f3c-9cfd-56731b661718" containerName="main"
Sep 18 14:10:15.088211 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:15.088170    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3a3b935-cb2b-4a63-ba38-2906228f737a" containerName="main"
Sep 18 14:10:15.088211 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:15.088180    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3a3b935-cb2b-4a63-ba38-2906228f737a" containerName="main"
Sep 18 14:10:15.088211 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:15.088199    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a58f638f-72bd-4d81-bdba-5b58e146ae90" containerName="main"
Sep 18 14:10:15.088211 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:15.088208    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a58f638f-72bd-4d81-bdba-5b58e146ae90" containerName="main"
Sep 18 14:10:15.088368 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:15.088224    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0611c055-7a5e-4503-b076-22ac855233ac" containerName="payload-pre-processing"
Sep 18 14:10:15.088368 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:15.088233    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0611c055-7a5e-4503-b076-22ac855233ac" containerName="payload-pre-processing"
Sep 18 14:10:15.088368 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:15.088254    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f40c363-5fbd-47dd-a3d7-8a62b5f8e5ad" containerName="test-curl-7627-518dbc"
Sep 18 14:10:15.088368 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:15.088263    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f40c363-5fbd-47dd-a3d7-8a62b5f8e5ad" containerName="test-curl-7627-518dbc"
Sep 18 14:10:15.088368 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:15.088356    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d81fc1dd-14e8-4f3c-9cfd-56731b661718" containerName="main"
Sep 18 14:10:15.088521 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:15.088371    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f40c363-5fbd-47dd-a3d7-8a62b5f8e5ad" containerName="test-curl-7627-518dbc"
Sep 18 14:10:15.088521 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:15.088384    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0611c055-7a5e-4503-b076-22ac855233ac" containerName="payload-pre-processing"
Sep 18 14:10:15.088521 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:15.088395    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3a3b935-cb2b-4a63-ba38-2906228f737a" containerName="main"
Sep 18 14:10:15.088521 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:15.088405    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a58f638f-72bd-4d81-bdba-5b58e146ae90" containerName="main"
Sep 18 14:10:15.093340 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:15.093314    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-783872/sub-test-model-783872-kserve-59c9f8df9c-j27z7"
Sep 18 14:10:15.096017 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:15.095991    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-783872\"/\"sub-test-model-783872-kserve-self-signed-certs\""
Sep 18 14:10:15.096161 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:15.096069    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-783872\"/\"kube-root-ca.crt\""
Sep 18 14:10:15.096161 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:15.096088    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-783872\"/\"default-dockercfg-cwdxt\""
Sep 18 14:10:15.096161 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:15.096137    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-783872\"/\"openshift-service-ca.crt\""
Sep 18 14:10:15.100330 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:15.100303    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-783872/sub-test-model-783872-kserve-59c9f8df9c-j27z7"]
Sep 18 14:10:15.122698 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:15.122652    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:10:15.170298 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:15.170255    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80-home\") pod \"sub-test-model-783872-kserve-59c9f8df9c-j27z7\" (UID: \"f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80\") " pod="ai-tenant-e2e-shared-a-w3-783872/sub-test-model-783872-kserve-59c9f8df9c-j27z7"
Sep 18 14:10:15.170298 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:15.170295    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjbr8\" (UniqueName: \"kubernetes.io/projected/f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80-kube-api-access-mjbr8\") pod \"sub-test-model-783872-kserve-59c9f8df9c-j27z7\" (UID: \"f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80\") " pod="ai-tenant-e2e-shared-a-w3-783872/sub-test-model-783872-kserve-59c9f8df9c-j27z7"
Sep 18 14:10:15.170543 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:15.170424    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80-dshm\") pod \"sub-test-model-783872-kserve-59c9f8df9c-j27z7\" (UID: \"f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80\") " pod="ai-tenant-e2e-shared-a-w3-783872/sub-test-model-783872-kserve-59c9f8df9c-j27z7"
Sep 18 14:10:15.170543 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:15.170461    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80-tmp-dir\") pod \"sub-test-model-783872-kserve-59c9f8df9c-j27z7\" (UID: \"f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80\") " pod="ai-tenant-e2e-shared-a-w3-783872/sub-test-model-783872-kserve-59c9f8df9c-j27z7"
Sep 18 14:10:15.170543 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:15.170494    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80-tls-certs\") pod \"sub-test-model-783872-kserve-59c9f8df9c-j27z7\" (UID: \"f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80\") " pod="ai-tenant-e2e-shared-a-w3-783872/sub-test-model-783872-kserve-59c9f8df9c-j27z7"
Sep 18 14:10:15.170543 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:15.170518    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80-model-cache\") pod \"sub-test-model-783872-kserve-59c9f8df9c-j27z7\" (UID: \"f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80\") " pod="ai-tenant-e2e-shared-a-w3-783872/sub-test-model-783872-kserve-59c9f8df9c-j27z7"
Sep 18 14:10:15.230140 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:15.230106    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:10:15.271511 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:15.271476    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80-dshm\") pod \"sub-test-model-783872-kserve-59c9f8df9c-j27z7\" (UID: \"f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80\") " pod="ai-tenant-e2e-shared-a-w3-783872/sub-test-model-783872-kserve-59c9f8df9c-j27z7"
Sep 18 14:10:15.271698 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:15.271531    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80-tmp-dir\") pod \"sub-test-model-783872-kserve-59c9f8df9c-j27z7\" (UID: \"f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80\") " pod="ai-tenant-e2e-shared-a-w3-783872/sub-test-model-783872-kserve-59c9f8df9c-j27z7"
Sep 18 14:10:15.271698 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:15.271571    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80-tls-certs\") pod \"sub-test-model-783872-kserve-59c9f8df9c-j27z7\" (UID: \"f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80\") " pod="ai-tenant-e2e-shared-a-w3-783872/sub-test-model-783872-kserve-59c9f8df9c-j27z7"
Sep 18 14:10:15.271698 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:15.271601    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80-model-cache\") pod \"sub-test-model-783872-kserve-59c9f8df9c-j27z7\" (UID: \"f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80\") " pod="ai-tenant-e2e-shared-a-w3-783872/sub-test-model-783872-kserve-59c9f8df9c-j27z7"
Sep 18 14:10:15.271698 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:15.271655    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80-home\") pod \"sub-test-model-783872-kserve-59c9f8df9c-j27z7\" (UID: \"f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80\") " pod="ai-tenant-e2e-shared-a-w3-783872/sub-test-model-783872-kserve-59c9f8df9c-j27z7"
Sep 18 14:10:15.271698 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:15.271681    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjbr8\" (UniqueName: \"kubernetes.io/projected/f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80-kube-api-access-mjbr8\") pod \"sub-test-model-783872-kserve-59c9f8df9c-j27z7\" (UID: \"f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80\") " pod="ai-tenant-e2e-shared-a-w3-783872/sub-test-model-783872-kserve-59c9f8df9c-j27z7"
Sep 18 14:10:15.272095 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:15.272073    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80-model-cache\") pod \"sub-test-model-783872-kserve-59c9f8df9c-j27z7\" (UID: \"f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80\") " pod="ai-tenant-e2e-shared-a-w3-783872/sub-test-model-783872-kserve-59c9f8df9c-j27z7"
Sep 18 14:10:15.272166 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:15.272073    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80-tmp-dir\") pod \"sub-test-model-783872-kserve-59c9f8df9c-j27z7\" (UID: \"f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80\") " pod="ai-tenant-e2e-shared-a-w3-783872/sub-test-model-783872-kserve-59c9f8df9c-j27z7"
Sep 18 14:10:15.272166 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:15.272125    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80-home\") pod \"sub-test-model-783872-kserve-59c9f8df9c-j27z7\" (UID: \"f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80\") " pod="ai-tenant-e2e-shared-a-w3-783872/sub-test-model-783872-kserve-59c9f8df9c-j27z7"
Sep 18 14:10:15.274014 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:15.273991    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80-dshm\") pod \"sub-test-model-783872-kserve-59c9f8df9c-j27z7\" (UID: \"f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80\") " pod="ai-tenant-e2e-shared-a-w3-783872/sub-test-model-783872-kserve-59c9f8df9c-j27z7"
Sep 18 14:10:15.274177 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:15.274162    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80-tls-certs\") pod \"sub-test-model-783872-kserve-59c9f8df9c-j27z7\" (UID: \"f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80\") " pod="ai-tenant-e2e-shared-a-w3-783872/sub-test-model-783872-kserve-59c9f8df9c-j27z7"
Sep 18 14:10:15.279054 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:15.279033    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjbr8\" (UniqueName: \"kubernetes.io/projected/f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80-kube-api-access-mjbr8\") pod \"sub-test-model-783872-kserve-59c9f8df9c-j27z7\" (UID: \"f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80\") " pod="ai-tenant-e2e-shared-a-w3-783872/sub-test-model-783872-kserve-59c9f8df9c-j27z7"
Sep 18 14:10:15.405094 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:15.404998    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-783872/sub-test-model-783872-kserve-59c9f8df9c-j27z7"
Sep 18 14:10:15.681119 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:15.681092    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-783872/sub-test-model-783872-kserve-59c9f8df9c-j27z7"]
Sep 18 14:10:15.683408 ip-10-0-141-222 kubenswrapper[2577]: W0918 14:10:15.683377    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf3fb07f2_b49b_4dad_a9db_cb6ab8a56e80.slice/crio-86d48fccee68038a9096a75fd73497767b88aee1f97c4778eddca4c81f3350ac WatchSource:0}: Error finding container 86d48fccee68038a9096a75fd73497767b88aee1f97c4778eddca4c81f3350ac: Status 404 returned error can't find the container with id 86d48fccee68038a9096a75fd73497767b88aee1f97c4778eddca4c81f3350ac
Sep 18 14:10:15.745316 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:15.745287    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-783872/sub-test-model-783872-kserve-59c9f8df9c-j27z7" event={"ID":"f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80","Type":"ContainerStarted","Data":"86d48fccee68038a9096a75fd73497767b88aee1f97c4778eddca4c81f3350ac"}
Sep 18 14:10:15.916930 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:15.916895    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:10:16.724494 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:16.724463    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:10:16.751787 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:16.751756    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-783872/sub-test-model-783872-kserve-59c9f8df9c-j27z7" event={"ID":"f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80","Type":"ContainerStarted","Data":"1deeeabe11c5bf74d16565e53f5059a802835f4b8c5e4eee594c47f0ccfa374a"}
Sep 18 14:10:16.772282 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:16.772226    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-783872/sub-test-model-783872-kserve-59c9f8df9c-j27z7" podStartSLOduration=1.7722087819999999 podStartE2EDuration="1.772208782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:10:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:10:16.768892723 +0000 UTC m=+1565.821552050" watchObservedRunningTime="2026-09-18 14:10:16.772208782 +0000 UTC m=+1565.824868109"
Sep 18 14:10:18.195779 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:18.195745    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:10:18.292884 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:18.292849    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:10:19.900216 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:19.900182    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:10:23.409271 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:23.409224    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:10:23.431068 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:23.431030    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-e46fbf-7d645454dc-nmptd"]
Sep 18 14:10:23.431364 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:23.431333    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e46fbf-7d645454dc-nmptd" podUID="7cb59069-38f5-4972-9df5-99f51514d2e4" containerName="payload-processing" containerID="cri-o://6d6f719b5eea9d083bf0c80081e18ac81b82c9ae7b4c002d6986c169f070727b" gracePeriod=30
Sep 18 14:10:23.477765 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:23.477733    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:10:23.579083 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:23.579048    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:10:23.678502 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:23.678459    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:10:23.704367 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:23.704339    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e46fbf-7d645454dc-nmptd"
Sep 18 14:10:23.798599 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:23.798563    2577 generic.go:356] "Generic (PLEG): container finished" podID="7cb59069-38f5-4972-9df5-99f51514d2e4" containerID="6d6f719b5eea9d083bf0c80081e18ac81b82c9ae7b4c002d6986c169f070727b" exitCode=0
Sep 18 14:10:23.798850 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:23.798611    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e46fbf-7d645454dc-nmptd"
Sep 18 14:10:23.798850 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:23.798623    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e46fbf-7d645454dc-nmptd" event={"ID":"7cb59069-38f5-4972-9df5-99f51514d2e4","Type":"ContainerDied","Data":"6d6f719b5eea9d083bf0c80081e18ac81b82c9ae7b4c002d6986c169f070727b"}
Sep 18 14:10:23.798850 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:23.798668    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e46fbf-7d645454dc-nmptd" event={"ID":"7cb59069-38f5-4972-9df5-99f51514d2e4","Type":"ContainerDied","Data":"9a27e3d0ca7616921ff3f759b88f2e7c31c1834eda9b9c87c8297917bae7d936"}
Sep 18 14:10:23.798850 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:23.798691    2577 scope.go:117] "RemoveContainer" containerID="6d6f719b5eea9d083bf0c80081e18ac81b82c9ae7b4c002d6986c169f070727b"
Sep 18 14:10:23.808050 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:23.808029    2577 scope.go:117] "RemoveContainer" containerID="6d6f719b5eea9d083bf0c80081e18ac81b82c9ae7b4c002d6986c169f070727b"
Sep 18 14:10:23.808368 ip-10-0-141-222 kubenswrapper[2577]: E0918 14:10:23.808340    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d6f719b5eea9d083bf0c80081e18ac81b82c9ae7b4c002d6986c169f070727b\": container with ID starting with 6d6f719b5eea9d083bf0c80081e18ac81b82c9ae7b4c002d6986c169f070727b not found: ID does not exist" containerID="6d6f719b5eea9d083bf0c80081e18ac81b82c9ae7b4c002d6986c169f070727b"
Sep 18 14:10:23.808433 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:23.808376    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d6f719b5eea9d083bf0c80081e18ac81b82c9ae7b4c002d6986c169f070727b"} err="failed to get container status \"6d6f719b5eea9d083bf0c80081e18ac81b82c9ae7b4c002d6986c169f070727b\": rpc error: code = NotFound desc = could not find container \"6d6f719b5eea9d083bf0c80081e18ac81b82c9ae7b4c002d6986c169f070727b\": container with ID starting with 6d6f719b5eea9d083bf0c80081e18ac81b82c9ae7b4c002d6986c169f070727b not found: ID does not exist"
Sep 18 14:10:23.852544 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:23.852508    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cb59069-38f5-4972-9df5-99f51514d2e4-plugins-config-volume\") pod \"7cb59069-38f5-4972-9df5-99f51514d2e4\" (UID: \"7cb59069-38f5-4972-9df5-99f51514d2e4\") "
Sep 18 14:10:23.852771 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:23.852577    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fvgtc\" (UniqueName: \"kubernetes.io/projected/7cb59069-38f5-4972-9df5-99f51514d2e4-kube-api-access-fvgtc\") pod \"7cb59069-38f5-4972-9df5-99f51514d2e4\" (UID: \"7cb59069-38f5-4972-9df5-99f51514d2e4\") "
Sep 18 14:10:23.853079 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:23.853045    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cb59069-38f5-4972-9df5-99f51514d2e4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7cb59069-38f5-4972-9df5-99f51514d2e4" (UID: "7cb59069-38f5-4972-9df5-99f51514d2e4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:10:23.854917 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:23.854881    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cb59069-38f5-4972-9df5-99f51514d2e4-kube-api-access-fvgtc" (OuterVolumeSpecName: "kube-api-access-fvgtc") pod "7cb59069-38f5-4972-9df5-99f51514d2e4" (UID: "7cb59069-38f5-4972-9df5-99f51514d2e4"). InnerVolumeSpecName "kube-api-access-fvgtc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:10:23.953400 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:23.953300    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvgtc\" (UniqueName: \"kubernetes.io/projected/7cb59069-38f5-4972-9df5-99f51514d2e4-kube-api-access-fvgtc\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:10:23.953400 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:23.953331    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cb59069-38f5-4972-9df5-99f51514d2e4-plugins-config-volume\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:10:23.959307 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:23.959273    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:10:24.059101 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:24.059057    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:10:24.121926 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:24.121891    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-e46fbf-7d645454dc-nmptd"]
Sep 18 14:10:24.129007 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:24.128975    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-e46fbf-7d645454dc-nmptd"]
Sep 18 14:10:24.762979 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:24.762939    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:10:25.107520 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:25.107485    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7cb59069-38f5-4972-9df5-99f51514d2e4" path="/var/lib/kubelet/pods/7cb59069-38f5-4972-9df5-99f51514d2e4/volumes"
Sep 18 14:10:25.405436 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:25.405332    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-783872/sub-test-model-783872-kserve-59c9f8df9c-j27z7"
Sep 18 14:10:25.405436 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:25.405389    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-783872/sub-test-model-783872-kserve-59c9f8df9c-j27z7"
Sep 18 14:10:25.418087 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:25.418056    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-783872/sub-test-model-783872-kserve-59c9f8df9c-j27z7"
Sep 18 14:10:25.443591 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:25.443558    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:10:25.823005 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:25.822915    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-783872/sub-test-model-783872-kserve-59c9f8df9c-j27z7"
Sep 18 14:10:26.247700 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:26.247663    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:10:29.592428 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:29.592384    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 14:10:29.992457 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:29.992372    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:10:30.067251 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:30.067211    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:10:30.270075 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:30.269990    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:10:31.337306 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:31.337272    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:10:32.115734 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:32.115689    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:10:32.770440 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:32.770399    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:10:33.408856 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:33.408803    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:10:34.994420 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:34.994369    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-1a4b6d/sub-test-model-1a4b6d-kserve-57ff7dd955-75v7m"]
Sep 18 14:10:34.994837 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:34.994727    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cb59069-38f5-4972-9df5-99f51514d2e4" containerName="payload-processing"
Sep 18 14:10:34.994837 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:34.994738    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cb59069-38f5-4972-9df5-99f51514d2e4" containerName="payload-processing"
Sep 18 14:10:34.994837 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:34.994835    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cb59069-38f5-4972-9df5-99f51514d2e4" containerName="payload-processing"
Sep 18 14:10:34.998089 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:34.998068    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-1a4b6d/sub-test-model-1a4b6d-kserve-57ff7dd955-75v7m"
Sep 18 14:10:35.000550 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:35.000517    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-1a4b6d\"/\"openshift-service-ca.crt\""
Sep 18 14:10:35.001064 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:35.001033    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-1a4b6d\"/\"kube-root-ca.crt\""
Sep 18 14:10:35.001186 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:35.001084    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-1a4b6d\"/\"sub-test-model-1a4b6d-kserve-self-signed-certs\""
Sep 18 14:10:35.001186 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:35.001104    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-1a4b6d\"/\"default-dockercfg-2qgqp\""
Sep 18 14:10:35.008919 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:35.008895    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1a4b6d/sub-test-model-1a4b6d-kserve-57ff7dd955-75v7m"]
Sep 18 14:10:35.033800 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:35.033764    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:10:35.161188 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:35.161142    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9gzl\" (UniqueName: \"kubernetes.io/projected/7bb7da67-bf04-4716-bd98-5f5c35b236a8-kube-api-access-c9gzl\") pod \"sub-test-model-1a4b6d-kserve-57ff7dd955-75v7m\" (UID: \"7bb7da67-bf04-4716-bd98-5f5c35b236a8\") " pod="ai-tenant-e2e-shared-b-w3-1a4b6d/sub-test-model-1a4b6d-kserve-57ff7dd955-75v7m"
Sep 18 14:10:35.161392 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:35.161215    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bb7da67-bf04-4716-bd98-5f5c35b236a8-model-cache\") pod \"sub-test-model-1a4b6d-kserve-57ff7dd955-75v7m\" (UID: \"7bb7da67-bf04-4716-bd98-5f5c35b236a8\") " pod="ai-tenant-e2e-shared-b-w3-1a4b6d/sub-test-model-1a4b6d-kserve-57ff7dd955-75v7m"
Sep 18 14:10:35.161392 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:35.161265    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bb7da67-bf04-4716-bd98-5f5c35b236a8-tmp-dir\") pod \"sub-test-model-1a4b6d-kserve-57ff7dd955-75v7m\" (UID: \"7bb7da67-bf04-4716-bd98-5f5c35b236a8\") " pod="ai-tenant-e2e-shared-b-w3-1a4b6d/sub-test-model-1a4b6d-kserve-57ff7dd955-75v7m"
Sep 18 14:10:35.161392 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:35.161328    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bb7da67-bf04-4716-bd98-5f5c35b236a8-tls-certs\") pod \"sub-test-model-1a4b6d-kserve-57ff7dd955-75v7m\" (UID: \"7bb7da67-bf04-4716-bd98-5f5c35b236a8\") " pod="ai-tenant-e2e-shared-b-w3-1a4b6d/sub-test-model-1a4b6d-kserve-57ff7dd955-75v7m"
Sep 18 14:10:35.161558 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:35.161406    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bb7da67-bf04-4716-bd98-5f5c35b236a8-home\") pod \"sub-test-model-1a4b6d-kserve-57ff7dd955-75v7m\" (UID: \"7bb7da67-bf04-4716-bd98-5f5c35b236a8\") " pod="ai-tenant-e2e-shared-b-w3-1a4b6d/sub-test-model-1a4b6d-kserve-57ff7dd955-75v7m"
Sep 18 14:10:35.161558 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:35.161469    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bb7da67-bf04-4716-bd98-5f5c35b236a8-dshm\") pod \"sub-test-model-1a4b6d-kserve-57ff7dd955-75v7m\" (UID: \"7bb7da67-bf04-4716-bd98-5f5c35b236a8\") " pod="ai-tenant-e2e-shared-b-w3-1a4b6d/sub-test-model-1a4b6d-kserve-57ff7dd955-75v7m"
Sep 18 14:10:35.262599 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:35.262511    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bb7da67-bf04-4716-bd98-5f5c35b236a8-tls-certs\") pod \"sub-test-model-1a4b6d-kserve-57ff7dd955-75v7m\" (UID: \"7bb7da67-bf04-4716-bd98-5f5c35b236a8\") " pod="ai-tenant-e2e-shared-b-w3-1a4b6d/sub-test-model-1a4b6d-kserve-57ff7dd955-75v7m"
Sep 18 14:10:35.262599 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:35.262555    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bb7da67-bf04-4716-bd98-5f5c35b236a8-home\") pod \"sub-test-model-1a4b6d-kserve-57ff7dd955-75v7m\" (UID: \"7bb7da67-bf04-4716-bd98-5f5c35b236a8\") " pod="ai-tenant-e2e-shared-b-w3-1a4b6d/sub-test-model-1a4b6d-kserve-57ff7dd955-75v7m"
Sep 18 14:10:35.262599 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:35.262595    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bb7da67-bf04-4716-bd98-5f5c35b236a8-dshm\") pod \"sub-test-model-1a4b6d-kserve-57ff7dd955-75v7m\" (UID: \"7bb7da67-bf04-4716-bd98-5f5c35b236a8\") " pod="ai-tenant-e2e-shared-b-w3-1a4b6d/sub-test-model-1a4b6d-kserve-57ff7dd955-75v7m"
Sep 18 14:10:35.262925 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:35.262617    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9gzl\" (UniqueName: \"kubernetes.io/projected/7bb7da67-bf04-4716-bd98-5f5c35b236a8-kube-api-access-c9gzl\") pod \"sub-test-model-1a4b6d-kserve-57ff7dd955-75v7m\" (UID: \"7bb7da67-bf04-4716-bd98-5f5c35b236a8\") " pod="ai-tenant-e2e-shared-b-w3-1a4b6d/sub-test-model-1a4b6d-kserve-57ff7dd955-75v7m"
Sep 18 14:10:35.262925 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:35.262657    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bb7da67-bf04-4716-bd98-5f5c35b236a8-model-cache\") pod \"sub-test-model-1a4b6d-kserve-57ff7dd955-75v7m\" (UID: \"7bb7da67-bf04-4716-bd98-5f5c35b236a8\") " pod="ai-tenant-e2e-shared-b-w3-1a4b6d/sub-test-model-1a4b6d-kserve-57ff7dd955-75v7m"
Sep 18 14:10:35.262925 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:35.262694    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bb7da67-bf04-4716-bd98-5f5c35b236a8-tmp-dir\") pod \"sub-test-model-1a4b6d-kserve-57ff7dd955-75v7m\" (UID: \"7bb7da67-bf04-4716-bd98-5f5c35b236a8\") " pod="ai-tenant-e2e-shared-b-w3-1a4b6d/sub-test-model-1a4b6d-kserve-57ff7dd955-75v7m"
Sep 18 14:10:35.263120 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:35.263097    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bb7da67-bf04-4716-bd98-5f5c35b236a8-home\") pod \"sub-test-model-1a4b6d-kserve-57ff7dd955-75v7m\" (UID: \"7bb7da67-bf04-4716-bd98-5f5c35b236a8\") " pod="ai-tenant-e2e-shared-b-w3-1a4b6d/sub-test-model-1a4b6d-kserve-57ff7dd955-75v7m"
Sep 18 14:10:35.263177 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:35.263100    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bb7da67-bf04-4716-bd98-5f5c35b236a8-model-cache\") pod \"sub-test-model-1a4b6d-kserve-57ff7dd955-75v7m\" (UID: \"7bb7da67-bf04-4716-bd98-5f5c35b236a8\") " pod="ai-tenant-e2e-shared-b-w3-1a4b6d/sub-test-model-1a4b6d-kserve-57ff7dd955-75v7m"
Sep 18 14:10:35.263177 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:35.263161    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bb7da67-bf04-4716-bd98-5f5c35b236a8-tmp-dir\") pod \"sub-test-model-1a4b6d-kserve-57ff7dd955-75v7m\" (UID: \"7bb7da67-bf04-4716-bd98-5f5c35b236a8\") " pod="ai-tenant-e2e-shared-b-w3-1a4b6d/sub-test-model-1a4b6d-kserve-57ff7dd955-75v7m"
Sep 18 14:10:35.264951 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:35.264933    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bb7da67-bf04-4716-bd98-5f5c35b236a8-dshm\") pod \"sub-test-model-1a4b6d-kserve-57ff7dd955-75v7m\" (UID: \"7bb7da67-bf04-4716-bd98-5f5c35b236a8\") " pod="ai-tenant-e2e-shared-b-w3-1a4b6d/sub-test-model-1a4b6d-kserve-57ff7dd955-75v7m"
Sep 18 14:10:35.265131 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:35.265114    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bb7da67-bf04-4716-bd98-5f5c35b236a8-tls-certs\") pod \"sub-test-model-1a4b6d-kserve-57ff7dd955-75v7m\" (UID: \"7bb7da67-bf04-4716-bd98-5f5c35b236a8\") " pod="ai-tenant-e2e-shared-b-w3-1a4b6d/sub-test-model-1a4b6d-kserve-57ff7dd955-75v7m"
Sep 18 14:10:35.271321 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:35.271285    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9gzl\" (UniqueName: \"kubernetes.io/projected/7bb7da67-bf04-4716-bd98-5f5c35b236a8-kube-api-access-c9gzl\") pod \"sub-test-model-1a4b6d-kserve-57ff7dd955-75v7m\" (UID: \"7bb7da67-bf04-4716-bd98-5f5c35b236a8\") " pod="ai-tenant-e2e-shared-b-w3-1a4b6d/sub-test-model-1a4b6d-kserve-57ff7dd955-75v7m"
Sep 18 14:10:35.308156 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:35.308120    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-1a4b6d/sub-test-model-1a4b6d-kserve-57ff7dd955-75v7m"
Sep 18 14:10:35.584758 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:35.584730    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1a4b6d/sub-test-model-1a4b6d-kserve-57ff7dd955-75v7m"]
Sep 18 14:10:35.587067 ip-10-0-141-222 kubenswrapper[2577]: W0918 14:10:35.587034    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7bb7da67_bf04_4716_bd98_5f5c35b236a8.slice/crio-9211e2675fd30fe39dd8bcd4e26144b5fac016a8f5d9c9cccbe4064bc411d77b WatchSource:0}: Error finding container 9211e2675fd30fe39dd8bcd4e26144b5fac016a8f5d9c9cccbe4064bc411d77b: Status 404 returned error can't find the container with id 9211e2675fd30fe39dd8bcd4e26144b5fac016a8f5d9c9cccbe4064bc411d77b
Sep 18 14:10:35.865481 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:35.865445    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-1a4b6d/sub-test-model-1a4b6d-kserve-57ff7dd955-75v7m" event={"ID":"7bb7da67-bf04-4716-bd98-5f5c35b236a8","Type":"ContainerStarted","Data":"16656b7f2be7733d3564417b9f0a56293481d3b4294730115507963618b7b71c"}
Sep 18 14:10:35.865674 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:35.865491    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-1a4b6d/sub-test-model-1a4b6d-kserve-57ff7dd955-75v7m" event={"ID":"7bb7da67-bf04-4716-bd98-5f5c35b236a8","Type":"ContainerStarted","Data":"9211e2675fd30fe39dd8bcd4e26144b5fac016a8f5d9c9cccbe4064bc411d77b"}
Sep 18 14:10:35.882521 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:35.882452    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-1a4b6d/sub-test-model-1a4b6d-kserve-57ff7dd955-75v7m" podStartSLOduration=1.8824320989999999 podStartE2EDuration="1.882432099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:10:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:10:35.882096594 +0000 UTC m=+1584.934755922" watchObservedRunningTime="2026-09-18 14:10:35.882432099 +0000 UTC m=+1584.935091427"
Sep 18 14:10:35.968594 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:35.968558    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:10:36.273718 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:36.273619    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:10:39.902117 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:39.902079    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 14:10:41.072969 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:41.072936    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:10:41.175072 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:41.175035    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:10:41.274423 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:41.274377    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:10:41.340057 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:41.340026    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:10:41.656802 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:41.656712    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:10:41.756749 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:41.756714    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:10:42.754757 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:42.754725    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:10:43.450034 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:43.449986    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:10:45.308638 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:45.308596    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-1a4b6d/sub-test-model-1a4b6d-kserve-57ff7dd955-75v7m"
Sep 18 14:10:45.309151 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:45.308665    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-1a4b6d/sub-test-model-1a4b6d-kserve-57ff7dd955-75v7m"
Sep 18 14:10:45.321877 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:45.321844    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-1a4b6d/sub-test-model-1a4b6d-kserve-57ff7dd955-75v7m"
Sep 18 14:10:45.937526 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:45.937496    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-1a4b6d/sub-test-model-1a4b6d-kserve-57ff7dd955-75v7m"
Sep 18 14:10:48.180900 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:48.180857    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:10:48.281037 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:48.281001    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:10:48.424170 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:48.424140    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6"]
Sep 18 14:10:48.424382 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:48.424356    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6" podUID="1b4e8091-639f-434e-8bf3-4411ef5b0b7e" containerName="istio-proxy" containerID="cri-o://df549ff85096cbcbbd24faded92a397abb3a94433a0e1db79e1dd98358dd4143" gracePeriod=30
Sep 18 14:10:48.444188 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:48.444097    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:10:51.192881 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:51.192844    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 14:10:53.684586 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:53.684553    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6"
Sep 18 14:10:53.724729 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:53.724638    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bgb7k\" (UniqueName: \"kubernetes.io/projected/1b4e8091-639f-434e-8bf3-4411ef5b0b7e-kube-api-access-bgb7k\") pod \"1b4e8091-639f-434e-8bf3-4411ef5b0b7e\" (UID: \"1b4e8091-639f-434e-8bf3-4411ef5b0b7e\") "
Sep 18 14:10:53.724729 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:53.724680    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b4e8091-639f-434e-8bf3-4411ef5b0b7e-istiod-ca-cert\") pod \"1b4e8091-639f-434e-8bf3-4411ef5b0b7e\" (UID: \"1b4e8091-639f-434e-8bf3-4411ef5b0b7e\") "
Sep 18 14:10:53.724729 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:53.724705    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b4e8091-639f-434e-8bf3-4411ef5b0b7e-istio-token\") pod \"1b4e8091-639f-434e-8bf3-4411ef5b0b7e\" (UID: \"1b4e8091-639f-434e-8bf3-4411ef5b0b7e\") "
Sep 18 14:10:53.725043 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:53.724875    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b4e8091-639f-434e-8bf3-4411ef5b0b7e-istio-data\") pod \"1b4e8091-639f-434e-8bf3-4411ef5b0b7e\" (UID: \"1b4e8091-639f-434e-8bf3-4411ef5b0b7e\") "
Sep 18 14:10:53.725043 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:53.724915    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b4e8091-639f-434e-8bf3-4411ef5b0b7e-workload-certs\") pod \"1b4e8091-639f-434e-8bf3-4411ef5b0b7e\" (UID: \"1b4e8091-639f-434e-8bf3-4411ef5b0b7e\") "
Sep 18 14:10:53.725219 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:53.725191    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b4e8091-639f-434e-8bf3-4411ef5b0b7e-istio-data" (OuterVolumeSpecName: "istio-data") pod "1b4e8091-639f-434e-8bf3-4411ef5b0b7e" (UID: "1b4e8091-639f-434e-8bf3-4411ef5b0b7e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:10:53.725284 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:53.725210    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b4e8091-639f-434e-8bf3-4411ef5b0b7e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1b4e8091-639f-434e-8bf3-4411ef5b0b7e" (UID: "1b4e8091-639f-434e-8bf3-4411ef5b0b7e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:10:53.725284 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:53.725164    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b4e8091-639f-434e-8bf3-4411ef5b0b7e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1b4e8091-639f-434e-8bf3-4411ef5b0b7e" (UID: "1b4e8091-639f-434e-8bf3-4411ef5b0b7e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:10:53.725363 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:53.725340    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b4e8091-639f-434e-8bf3-4411ef5b0b7e-credential-socket\") pod \"1b4e8091-639f-434e-8bf3-4411ef5b0b7e\" (UID: \"1b4e8091-639f-434e-8bf3-4411ef5b0b7e\") "
Sep 18 14:10:53.725413 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:53.725383    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b4e8091-639f-434e-8bf3-4411ef5b0b7e-istio-envoy\") pod \"1b4e8091-639f-434e-8bf3-4411ef5b0b7e\" (UID: \"1b4e8091-639f-434e-8bf3-4411ef5b0b7e\") "
Sep 18 14:10:53.725413 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:53.725409    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b4e8091-639f-434e-8bf3-4411ef5b0b7e-workload-socket\") pod \"1b4e8091-639f-434e-8bf3-4411ef5b0b7e\" (UID: \"1b4e8091-639f-434e-8bf3-4411ef5b0b7e\") "
Sep 18 14:10:53.725521 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:53.725436    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b4e8091-639f-434e-8bf3-4411ef5b0b7e-istio-podinfo\") pod \"1b4e8091-639f-434e-8bf3-4411ef5b0b7e\" (UID: \"1b4e8091-639f-434e-8bf3-4411ef5b0b7e\") "
Sep 18 14:10:53.725627 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:53.725607    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b4e8091-639f-434e-8bf3-4411ef5b0b7e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1b4e8091-639f-434e-8bf3-4411ef5b0b7e" (UID: "1b4e8091-639f-434e-8bf3-4411ef5b0b7e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:10:53.725700 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:53.725657    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b4e8091-639f-434e-8bf3-4411ef5b0b7e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1b4e8091-639f-434e-8bf3-4411ef5b0b7e" (UID: "1b4e8091-639f-434e-8bf3-4411ef5b0b7e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:10:53.725876 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:53.725853    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b4e8091-639f-434e-8bf3-4411ef5b0b7e-credential-socket\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:10:53.725977 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:53.725881    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b4e8091-639f-434e-8bf3-4411ef5b0b7e-workload-socket\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:10:53.725977 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:53.725896    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b4e8091-639f-434e-8bf3-4411ef5b0b7e-istiod-ca-cert\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:10:53.725977 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:53.725909    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b4e8091-639f-434e-8bf3-4411ef5b0b7e-istio-data\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:10:53.725977 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:53.725921    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b4e8091-639f-434e-8bf3-4411ef5b0b7e-workload-certs\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:10:53.727131 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:53.727094    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b4e8091-639f-434e-8bf3-4411ef5b0b7e-istio-token" (OuterVolumeSpecName: "istio-token") pod "1b4e8091-639f-434e-8bf3-4411ef5b0b7e" (UID: "1b4e8091-639f-434e-8bf3-4411ef5b0b7e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:10:53.727131 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:53.727108    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b4e8091-639f-434e-8bf3-4411ef5b0b7e-kube-api-access-bgb7k" (OuterVolumeSpecName: "kube-api-access-bgb7k") pod "1b4e8091-639f-434e-8bf3-4411ef5b0b7e" (UID: "1b4e8091-639f-434e-8bf3-4411ef5b0b7e"). InnerVolumeSpecName "kube-api-access-bgb7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:10:53.727534 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:53.727514    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b4e8091-639f-434e-8bf3-4411ef5b0b7e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1b4e8091-639f-434e-8bf3-4411ef5b0b7e" (UID: "1b4e8091-639f-434e-8bf3-4411ef5b0b7e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 14:10:53.727981 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:53.727962    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b4e8091-639f-434e-8bf3-4411ef5b0b7e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1b4e8091-639f-434e-8bf3-4411ef5b0b7e" (UID: "1b4e8091-639f-434e-8bf3-4411ef5b0b7e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:10:53.826435 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:53.826384    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgb7k\" (UniqueName: \"kubernetes.io/projected/1b4e8091-639f-434e-8bf3-4411ef5b0b7e-kube-api-access-bgb7k\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:10:53.826435 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:53.826428    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b4e8091-639f-434e-8bf3-4411ef5b0b7e-istio-token\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:10:53.826435 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:53.826438    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b4e8091-639f-434e-8bf3-4411ef5b0b7e-istio-envoy\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:10:53.826435 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:53.826445    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b4e8091-639f-434e-8bf3-4411ef5b0b7e-istio-podinfo\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:10:53.978835 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:53.978717    2577 generic.go:356] "Generic (PLEG): container finished" podID="1b4e8091-639f-434e-8bf3-4411ef5b0b7e" containerID="df549ff85096cbcbbd24faded92a397abb3a94433a0e1db79e1dd98358dd4143" exitCode=0
Sep 18 14:10:53.978835 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:53.978772    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6"
Sep 18 14:10:53.978835 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:53.978804    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6" event={"ID":"1b4e8091-639f-434e-8bf3-4411ef5b0b7e","Type":"ContainerDied","Data":"df549ff85096cbcbbd24faded92a397abb3a94433a0e1db79e1dd98358dd4143"}
Sep 18 14:10:53.979127 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:53.978882    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6" event={"ID":"1b4e8091-639f-434e-8bf3-4411ef5b0b7e","Type":"ContainerDied","Data":"3c077352c8655533f7a4ba0fc8f2c83755dd8c02c66a33e46eb7b84408b6b062"}
Sep 18 14:10:53.979127 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:53.978910    2577 scope.go:117] "RemoveContainer" containerID="df549ff85096cbcbbd24faded92a397abb3a94433a0e1db79e1dd98358dd4143"
Sep 18 14:10:53.989670 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:53.989647    2577 scope.go:117] "RemoveContainer" containerID="d2238084965e842fce7e00c218dd42f2e36814d052f5374cf387c7890c11474e"
Sep 18 14:10:53.998530 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:53.998508    2577 scope.go:117] "RemoveContainer" containerID="df549ff85096cbcbbd24faded92a397abb3a94433a0e1db79e1dd98358dd4143"
Sep 18 14:10:53.998882 ip-10-0-141-222 kubenswrapper[2577]: E0918 14:10:53.998861    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df549ff85096cbcbbd24faded92a397abb3a94433a0e1db79e1dd98358dd4143\": container with ID starting with df549ff85096cbcbbd24faded92a397abb3a94433a0e1db79e1dd98358dd4143 not found: ID does not exist" containerID="df549ff85096cbcbbd24faded92a397abb3a94433a0e1db79e1dd98358dd4143"
Sep 18 14:10:53.998947 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:53.998892    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df549ff85096cbcbbd24faded92a397abb3a94433a0e1db79e1dd98358dd4143"} err="failed to get container status \"df549ff85096cbcbbd24faded92a397abb3a94433a0e1db79e1dd98358dd4143\": rpc error: code = NotFound desc = could not find container \"df549ff85096cbcbbd24faded92a397abb3a94433a0e1db79e1dd98358dd4143\": container with ID starting with df549ff85096cbcbbd24faded92a397abb3a94433a0e1db79e1dd98358dd4143 not found: ID does not exist"
Sep 18 14:10:53.998947 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:53.998911    2577 scope.go:117] "RemoveContainer" containerID="d2238084965e842fce7e00c218dd42f2e36814d052f5374cf387c7890c11474e"
Sep 18 14:10:53.999180 ip-10-0-141-222 kubenswrapper[2577]: E0918 14:10:53.999158    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2238084965e842fce7e00c218dd42f2e36814d052f5374cf387c7890c11474e\": container with ID starting with d2238084965e842fce7e00c218dd42f2e36814d052f5374cf387c7890c11474e not found: ID does not exist" containerID="d2238084965e842fce7e00c218dd42f2e36814d052f5374cf387c7890c11474e"
Sep 18 14:10:53.999230 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:53.999186    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2238084965e842fce7e00c218dd42f2e36814d052f5374cf387c7890c11474e"} err="failed to get container status \"d2238084965e842fce7e00c218dd42f2e36814d052f5374cf387c7890c11474e\": rpc error: code = NotFound desc = could not find container \"d2238084965e842fce7e00c218dd42f2e36814d052f5374cf387c7890c11474e\": container with ID starting with d2238084965e842fce7e00c218dd42f2e36814d052f5374cf387c7890c11474e not found: ID does not exist"
Sep 18 14:10:54.002983 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:54.002958    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6"]
Sep 18 14:10:54.010423 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:54.010396    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-55bd4d-openshift-default-5b975c7b4d-n89f6"]
Sep 18 14:10:55.107885 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:10:55.107783    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b4e8091-639f-434e-8bf3-4411ef5b0b7e" path="/var/lib/kubelet/pods/1b4e8091-639f-434e-8bf3-4411ef5b0b7e/volumes"
Sep 18 14:11:04.496918 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:04.496880    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 14:11:14.937191 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:14.937154    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-783872/sub-test-model-783872-kserve-59c9f8df9c-j27z7"]
Sep 18 14:11:14.960765 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:14.960730    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:15.892316 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:15.892276    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1a4b6d/sub-test-model-1a4b6d-kserve-57ff7dd955-75v7m"]
Sep 18 14:11:15.912713 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:15.912666    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:17.508299 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:17.508260    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 14:11:23.280566 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:23.280528    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:27.599789 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:27.599742    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 14:11:29.956546 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:29.956506    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-783872/sub-test-model-783872-kserve-59c9f8df9c-j27z7" podUID="f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80" containerName="main" containerID="cri-o://1deeeabe11c5bf74d16565e53f5059a802835f4b8c5e4eee594c47f0ccfa374a" gracePeriod=45
Sep 18 14:11:30.181528 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:30.181496    2577 generic.go:356] "Generic (PLEG): container finished" podID="f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80" containerID="1deeeabe11c5bf74d16565e53f5059a802835f4b8c5e4eee594c47f0ccfa374a" exitCode=0
Sep 18 14:11:30.181713 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:30.181585    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-783872/sub-test-model-783872-kserve-59c9f8df9c-j27z7" event={"ID":"f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80","Type":"ContainerDied","Data":"1deeeabe11c5bf74d16565e53f5059a802835f4b8c5e4eee594c47f0ccfa374a"}
Sep 18 14:11:30.206198 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:30.206168    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-783872/sub-test-model-783872-kserve-59c9f8df9c-j27z7"
Sep 18 14:11:30.257913 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:30.257804    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80-home\") pod \"f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80\" (UID: \"f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80\") "
Sep 18 14:11:30.257913 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:30.257869    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80-tmp-dir\") pod \"f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80\" (UID: \"f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80\") "
Sep 18 14:11:30.257913 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:30.257894    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80-model-cache\") pod \"f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80\" (UID: \"f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80\") "
Sep 18 14:11:30.257913 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:30.257909    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80-dshm\") pod \"f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80\" (UID: \"f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80\") "
Sep 18 14:11:30.258256 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:30.257955    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80-tls-certs\") pod \"f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80\" (UID: \"f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80\") "
Sep 18 14:11:30.258256 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:30.257985    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mjbr8\" (UniqueName: \"kubernetes.io/projected/f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80-kube-api-access-mjbr8\") pod \"f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80\" (UID: \"f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80\") "
Sep 18 14:11:30.258256 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:30.258154    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80-home" (OuterVolumeSpecName: "home") pod "f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80" (UID: "f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:11:30.258256 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:30.258239    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80" (UID: "f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:11:30.258455 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:30.258259    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80-model-cache" (OuterVolumeSpecName: "model-cache") pod "f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80" (UID: "f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:11:30.258455 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:30.258288    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80-home\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:11:30.260336 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:30.260303    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80-dshm" (OuterVolumeSpecName: "dshm") pod "f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80" (UID: "f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:11:30.260459 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:30.260346    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80-kube-api-access-mjbr8" (OuterVolumeSpecName: "kube-api-access-mjbr8") pod "f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80" (UID: "f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80"). InnerVolumeSpecName "kube-api-access-mjbr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:11:30.260459 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:30.260374    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80" (UID: "f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:11:30.359746 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:30.359711    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80-tmp-dir\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:11:30.359746 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:30.359746    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80-model-cache\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:11:30.360034 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:30.359757    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80-dshm\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:11:30.360034 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:30.359766    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80-tls-certs\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:11:30.360034 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:30.359774    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjbr8\" (UniqueName: \"kubernetes.io/projected/f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80-kube-api-access-mjbr8\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:11:30.911473 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:30.911414    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-1a4b6d/sub-test-model-1a4b6d-kserve-57ff7dd955-75v7m" podUID="7bb7da67-bf04-4716-bd98-5f5c35b236a8" containerName="main" containerID="cri-o://16656b7f2be7733d3564417b9f0a56293481d3b4294730115507963618b7b71c" gracePeriod=45
Sep 18 14:11:31.165492 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:31.165428    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-1a4b6d/sub-test-model-1a4b6d-kserve-57ff7dd955-75v7m"
Sep 18 14:11:31.189901 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:31.189863    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-783872/sub-test-model-783872-kserve-59c9f8df9c-j27z7" event={"ID":"f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80","Type":"ContainerDied","Data":"86d48fccee68038a9096a75fd73497767b88aee1f97c4778eddca4c81f3350ac"}
Sep 18 14:11:31.190094 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:31.189933    2577 scope.go:117] "RemoveContainer" containerID="1deeeabe11c5bf74d16565e53f5059a802835f4b8c5e4eee594c47f0ccfa374a"
Sep 18 14:11:31.190094 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:31.189881    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-783872/sub-test-model-783872-kserve-59c9f8df9c-j27z7"
Sep 18 14:11:31.191865 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:31.191755    2577 generic.go:356] "Generic (PLEG): container finished" podID="7bb7da67-bf04-4716-bd98-5f5c35b236a8" containerID="16656b7f2be7733d3564417b9f0a56293481d3b4294730115507963618b7b71c" exitCode=0
Sep 18 14:11:31.191865 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:31.191813    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-1a4b6d/sub-test-model-1a4b6d-kserve-57ff7dd955-75v7m"
Sep 18 14:11:31.192022 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:31.191872    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-1a4b6d/sub-test-model-1a4b6d-kserve-57ff7dd955-75v7m" event={"ID":"7bb7da67-bf04-4716-bd98-5f5c35b236a8","Type":"ContainerDied","Data":"16656b7f2be7733d3564417b9f0a56293481d3b4294730115507963618b7b71c"}
Sep 18 14:11:31.192022 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:31.191915    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-1a4b6d/sub-test-model-1a4b6d-kserve-57ff7dd955-75v7m" event={"ID":"7bb7da67-bf04-4716-bd98-5f5c35b236a8","Type":"ContainerDied","Data":"9211e2675fd30fe39dd8bcd4e26144b5fac016a8f5d9c9cccbe4064bc411d77b"}
Sep 18 14:11:31.200477 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:31.200459    2577 scope.go:117] "RemoveContainer" containerID="16656b7f2be7733d3564417b9f0a56293481d3b4294730115507963618b7b71c"
Sep 18 14:11:31.205948 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:31.205921    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-783872/sub-test-model-783872-kserve-59c9f8df9c-j27z7"]
Sep 18 14:11:31.210239 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:31.210211    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-783872/sub-test-model-783872-kserve-59c9f8df9c-j27z7"]
Sep 18 14:11:31.211246 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:31.211228    2577 scope.go:117] "RemoveContainer" containerID="16656b7f2be7733d3564417b9f0a56293481d3b4294730115507963618b7b71c"
Sep 18 14:11:31.211591 ip-10-0-141-222 kubenswrapper[2577]: E0918 14:11:31.211571    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16656b7f2be7733d3564417b9f0a56293481d3b4294730115507963618b7b71c\": container with ID starting with 16656b7f2be7733d3564417b9f0a56293481d3b4294730115507963618b7b71c not found: ID does not exist" containerID="16656b7f2be7733d3564417b9f0a56293481d3b4294730115507963618b7b71c"
Sep 18 14:11:31.211661 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:31.211604    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16656b7f2be7733d3564417b9f0a56293481d3b4294730115507963618b7b71c"} err="failed to get container status \"16656b7f2be7733d3564417b9f0a56293481d3b4294730115507963618b7b71c\": rpc error: code = NotFound desc = could not find container \"16656b7f2be7733d3564417b9f0a56293481d3b4294730115507963618b7b71c\": container with ID starting with 16656b7f2be7733d3564417b9f0a56293481d3b4294730115507963618b7b71c not found: ID does not exist"
Sep 18 14:11:31.267083 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:31.267037    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bb7da67-bf04-4716-bd98-5f5c35b236a8-model-cache\") pod \"7bb7da67-bf04-4716-bd98-5f5c35b236a8\" (UID: \"7bb7da67-bf04-4716-bd98-5f5c35b236a8\") "
Sep 18 14:11:31.267083 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:31.267088    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c9gzl\" (UniqueName: \"kubernetes.io/projected/7bb7da67-bf04-4716-bd98-5f5c35b236a8-kube-api-access-c9gzl\") pod \"7bb7da67-bf04-4716-bd98-5f5c35b236a8\" (UID: \"7bb7da67-bf04-4716-bd98-5f5c35b236a8\") "
Sep 18 14:11:31.267347 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:31.267121    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bb7da67-bf04-4716-bd98-5f5c35b236a8-home\") pod \"7bb7da67-bf04-4716-bd98-5f5c35b236a8\" (UID: \"7bb7da67-bf04-4716-bd98-5f5c35b236a8\") "
Sep 18 14:11:31.267347 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:31.267178    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bb7da67-bf04-4716-bd98-5f5c35b236a8-dshm\") pod \"7bb7da67-bf04-4716-bd98-5f5c35b236a8\" (UID: \"7bb7da67-bf04-4716-bd98-5f5c35b236a8\") "
Sep 18 14:11:31.267347 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:31.267247    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bb7da67-bf04-4716-bd98-5f5c35b236a8-tmp-dir\") pod \"7bb7da67-bf04-4716-bd98-5f5c35b236a8\" (UID: \"7bb7da67-bf04-4716-bd98-5f5c35b236a8\") "
Sep 18 14:11:31.267347 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:31.267270    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bb7da67-bf04-4716-bd98-5f5c35b236a8-tls-certs\") pod \"7bb7da67-bf04-4716-bd98-5f5c35b236a8\" (UID: \"7bb7da67-bf04-4716-bd98-5f5c35b236a8\") "
Sep 18 14:11:31.267550 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:31.267371    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bb7da67-bf04-4716-bd98-5f5c35b236a8-model-cache" (OuterVolumeSpecName: "model-cache") pod "7bb7da67-bf04-4716-bd98-5f5c35b236a8" (UID: "7bb7da67-bf04-4716-bd98-5f5c35b236a8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:11:31.267550 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:31.267414    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bb7da67-bf04-4716-bd98-5f5c35b236a8-home" (OuterVolumeSpecName: "home") pod "7bb7da67-bf04-4716-bd98-5f5c35b236a8" (UID: "7bb7da67-bf04-4716-bd98-5f5c35b236a8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:11:31.267550 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:31.267507    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bb7da67-bf04-4716-bd98-5f5c35b236a8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7bb7da67-bf04-4716-bd98-5f5c35b236a8" (UID: "7bb7da67-bf04-4716-bd98-5f5c35b236a8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:11:31.267751 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:31.267560    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bb7da67-bf04-4716-bd98-5f5c35b236a8-tmp-dir\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:11:31.267751 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:31.267572    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bb7da67-bf04-4716-bd98-5f5c35b236a8-model-cache\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:11:31.267751 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:31.267583    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bb7da67-bf04-4716-bd98-5f5c35b236a8-home\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:11:31.269509 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:31.269481    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bb7da67-bf04-4716-bd98-5f5c35b236a8-dshm" (OuterVolumeSpecName: "dshm") pod "7bb7da67-bf04-4716-bd98-5f5c35b236a8" (UID: "7bb7da67-bf04-4716-bd98-5f5c35b236a8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:11:31.269599 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:31.269475    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bb7da67-bf04-4716-bd98-5f5c35b236a8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7bb7da67-bf04-4716-bd98-5f5c35b236a8" (UID: "7bb7da67-bf04-4716-bd98-5f5c35b236a8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:11:31.269599 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:31.269577    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bb7da67-bf04-4716-bd98-5f5c35b236a8-kube-api-access-c9gzl" (OuterVolumeSpecName: "kube-api-access-c9gzl") pod "7bb7da67-bf04-4716-bd98-5f5c35b236a8" (UID: "7bb7da67-bf04-4716-bd98-5f5c35b236a8"). InnerVolumeSpecName "kube-api-access-c9gzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:11:31.368257 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:31.368207    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bb7da67-bf04-4716-bd98-5f5c35b236a8-tls-certs\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:11:31.368257 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:31.368255    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9gzl\" (UniqueName: \"kubernetes.io/projected/7bb7da67-bf04-4716-bd98-5f5c35b236a8-kube-api-access-c9gzl\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:11:31.368257 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:31.368266    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bb7da67-bf04-4716-bd98-5f5c35b236a8-dshm\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:11:31.513923 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:31.513887    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1a4b6d/sub-test-model-1a4b6d-kserve-57ff7dd955-75v7m"]
Sep 18 14:11:31.516034 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:31.516009    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1a4b6d/sub-test-model-1a4b6d-kserve-57ff7dd955-75v7m"]
Sep 18 14:11:33.107794 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:33.107759    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bb7da67-bf04-4716-bd98-5f5c35b236a8" path="/var/lib/kubelet/pods/7bb7da67-bf04-4716-bd98-5f5c35b236a8/volumes"
Sep 18 14:11:33.108213 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:33.108199    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80" path="/var/lib/kubelet/pods/f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80/volumes"
Sep 18 14:11:55.305414 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:55.305381    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-1a4b6d/rate-test-model-1a4b6d-kserve-744947c7f5-xx577"]
Sep 18 14:11:55.305837 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:55.305733    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bb7da67-bf04-4716-bd98-5f5c35b236a8" containerName="main"
Sep 18 14:11:55.305837 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:55.305745    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bb7da67-bf04-4716-bd98-5f5c35b236a8" containerName="main"
Sep 18 14:11:55.305837 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:55.305759    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b4e8091-639f-434e-8bf3-4411ef5b0b7e" containerName="istio-proxy"
Sep 18 14:11:55.305837 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:55.305764    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b4e8091-639f-434e-8bf3-4411ef5b0b7e" containerName="istio-proxy"
Sep 18 14:11:55.305837 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:55.305773    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80" containerName="main"
Sep 18 14:11:55.305837 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:55.305779    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80" containerName="main"
Sep 18 14:11:55.305837 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:55.305789    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b4e8091-639f-434e-8bf3-4411ef5b0b7e" containerName="istio-proxy"
Sep 18 14:11:55.305837 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:55.305794    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b4e8091-639f-434e-8bf3-4411ef5b0b7e" containerName="istio-proxy"
Sep 18 14:11:55.306096 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:55.305876    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3fb07f2-b49b-4dad-a9db-cb6ab8a56e80" containerName="main"
Sep 18 14:11:55.306096 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:55.305886    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b4e8091-639f-434e-8bf3-4411ef5b0b7e" containerName="istio-proxy"
Sep 18 14:11:55.306096 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:55.305894    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b4e8091-639f-434e-8bf3-4411ef5b0b7e" containerName="istio-proxy"
Sep 18 14:11:55.306096 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:55.305899    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bb7da67-bf04-4716-bd98-5f5c35b236a8" containerName="main"
Sep 18 14:11:55.308817 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:55.308800    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-1a4b6d/rate-test-model-1a4b6d-kserve-744947c7f5-xx577"
Sep 18 14:11:55.311365 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:55.311343    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-1a4b6d\"/\"kube-root-ca.crt\""
Sep 18 14:11:55.311505 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:55.311381    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-1a4b6d\"/\"openshift-service-ca.crt\""
Sep 18 14:11:55.311879 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:55.311864    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-1a4b6d\"/\"default-dockercfg-2qgqp\""
Sep 18 14:11:55.311972 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:55.311945    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-1a4b6d\"/\"rate-test-model-1a4b6d-kserve-self-signed-certs\""
Sep 18 14:11:55.318583 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:55.318549    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1a4b6d/rate-test-model-1a4b6d-kserve-744947c7f5-xx577"]
Sep 18 14:11:55.335959 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:55.335923    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:55.386056 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:55.386003    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51f02198-f392-4f35-b68e-9ad3fb6810df-dshm\") pod \"rate-test-model-1a4b6d-kserve-744947c7f5-xx577\" (UID: \"51f02198-f392-4f35-b68e-9ad3fb6810df\") " pod="ai-tenant-e2e-shared-b-w3-1a4b6d/rate-test-model-1a4b6d-kserve-744947c7f5-xx577"
Sep 18 14:11:55.386056 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:55.386063    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51f02198-f392-4f35-b68e-9ad3fb6810df-tmp-dir\") pod \"rate-test-model-1a4b6d-kserve-744947c7f5-xx577\" (UID: \"51f02198-f392-4f35-b68e-9ad3fb6810df\") " pod="ai-tenant-e2e-shared-b-w3-1a4b6d/rate-test-model-1a4b6d-kserve-744947c7f5-xx577"
Sep 18 14:11:55.386325 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:55.386155    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51f02198-f392-4f35-b68e-9ad3fb6810df-model-cache\") pod \"rate-test-model-1a4b6d-kserve-744947c7f5-xx577\" (UID: \"51f02198-f392-4f35-b68e-9ad3fb6810df\") " pod="ai-tenant-e2e-shared-b-w3-1a4b6d/rate-test-model-1a4b6d-kserve-744947c7f5-xx577"
Sep 18 14:11:55.386325 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:55.386197    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51f02198-f392-4f35-b68e-9ad3fb6810df-tls-certs\") pod \"rate-test-model-1a4b6d-kserve-744947c7f5-xx577\" (UID: \"51f02198-f392-4f35-b68e-9ad3fb6810df\") " pod="ai-tenant-e2e-shared-b-w3-1a4b6d/rate-test-model-1a4b6d-kserve-744947c7f5-xx577"
Sep 18 14:11:55.386325 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:55.386298    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zxsp\" (UniqueName: \"kubernetes.io/projected/51f02198-f392-4f35-b68e-9ad3fb6810df-kube-api-access-2zxsp\") pod \"rate-test-model-1a4b6d-kserve-744947c7f5-xx577\" (UID: \"51f02198-f392-4f35-b68e-9ad3fb6810df\") " pod="ai-tenant-e2e-shared-b-w3-1a4b6d/rate-test-model-1a4b6d-kserve-744947c7f5-xx577"
Sep 18 14:11:55.386488 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:55.386346    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51f02198-f392-4f35-b68e-9ad3fb6810df-home\") pod \"rate-test-model-1a4b6d-kserve-744947c7f5-xx577\" (UID: \"51f02198-f392-4f35-b68e-9ad3fb6810df\") " pod="ai-tenant-e2e-shared-b-w3-1a4b6d/rate-test-model-1a4b6d-kserve-744947c7f5-xx577"
Sep 18 14:11:55.487363 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:55.487323    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51f02198-f392-4f35-b68e-9ad3fb6810df-dshm\") pod \"rate-test-model-1a4b6d-kserve-744947c7f5-xx577\" (UID: \"51f02198-f392-4f35-b68e-9ad3fb6810df\") " pod="ai-tenant-e2e-shared-b-w3-1a4b6d/rate-test-model-1a4b6d-kserve-744947c7f5-xx577"
Sep 18 14:11:55.487363 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:55.487368    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51f02198-f392-4f35-b68e-9ad3fb6810df-tmp-dir\") pod \"rate-test-model-1a4b6d-kserve-744947c7f5-xx577\" (UID: \"51f02198-f392-4f35-b68e-9ad3fb6810df\") " pod="ai-tenant-e2e-shared-b-w3-1a4b6d/rate-test-model-1a4b6d-kserve-744947c7f5-xx577"
Sep 18 14:11:55.487593 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:55.487411    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51f02198-f392-4f35-b68e-9ad3fb6810df-model-cache\") pod \"rate-test-model-1a4b6d-kserve-744947c7f5-xx577\" (UID: \"51f02198-f392-4f35-b68e-9ad3fb6810df\") " pod="ai-tenant-e2e-shared-b-w3-1a4b6d/rate-test-model-1a4b6d-kserve-744947c7f5-xx577"
Sep 18 14:11:55.487593 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:55.487439    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51f02198-f392-4f35-b68e-9ad3fb6810df-tls-certs\") pod \"rate-test-model-1a4b6d-kserve-744947c7f5-xx577\" (UID: \"51f02198-f392-4f35-b68e-9ad3fb6810df\") " pod="ai-tenant-e2e-shared-b-w3-1a4b6d/rate-test-model-1a4b6d-kserve-744947c7f5-xx577"
Sep 18 14:11:55.487687 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:55.487591    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zxsp\" (UniqueName: \"kubernetes.io/projected/51f02198-f392-4f35-b68e-9ad3fb6810df-kube-api-access-2zxsp\") pod \"rate-test-model-1a4b6d-kserve-744947c7f5-xx577\" (UID: \"51f02198-f392-4f35-b68e-9ad3fb6810df\") " pod="ai-tenant-e2e-shared-b-w3-1a4b6d/rate-test-model-1a4b6d-kserve-744947c7f5-xx577"
Sep 18 14:11:55.487687 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:55.487647    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51f02198-f392-4f35-b68e-9ad3fb6810df-home\") pod \"rate-test-model-1a4b6d-kserve-744947c7f5-xx577\" (UID: \"51f02198-f392-4f35-b68e-9ad3fb6810df\") " pod="ai-tenant-e2e-shared-b-w3-1a4b6d/rate-test-model-1a4b6d-kserve-744947c7f5-xx577"
Sep 18 14:11:55.487851 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:55.487809    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51f02198-f392-4f35-b68e-9ad3fb6810df-tmp-dir\") pod \"rate-test-model-1a4b6d-kserve-744947c7f5-xx577\" (UID: \"51f02198-f392-4f35-b68e-9ad3fb6810df\") " pod="ai-tenant-e2e-shared-b-w3-1a4b6d/rate-test-model-1a4b6d-kserve-744947c7f5-xx577"
Sep 18 14:11:55.487965 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:55.487943    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51f02198-f392-4f35-b68e-9ad3fb6810df-model-cache\") pod \"rate-test-model-1a4b6d-kserve-744947c7f5-xx577\" (UID: \"51f02198-f392-4f35-b68e-9ad3fb6810df\") " pod="ai-tenant-e2e-shared-b-w3-1a4b6d/rate-test-model-1a4b6d-kserve-744947c7f5-xx577"
Sep 18 14:11:55.488036 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:55.487972    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51f02198-f392-4f35-b68e-9ad3fb6810df-home\") pod \"rate-test-model-1a4b6d-kserve-744947c7f5-xx577\" (UID: \"51f02198-f392-4f35-b68e-9ad3fb6810df\") " pod="ai-tenant-e2e-shared-b-w3-1a4b6d/rate-test-model-1a4b6d-kserve-744947c7f5-xx577"
Sep 18 14:11:55.489686 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:55.489657    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51f02198-f392-4f35-b68e-9ad3fb6810df-dshm\") pod \"rate-test-model-1a4b6d-kserve-744947c7f5-xx577\" (UID: \"51f02198-f392-4f35-b68e-9ad3fb6810df\") " pod="ai-tenant-e2e-shared-b-w3-1a4b6d/rate-test-model-1a4b6d-kserve-744947c7f5-xx577"
Sep 18 14:11:55.490000 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:55.489981    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51f02198-f392-4f35-b68e-9ad3fb6810df-tls-certs\") pod \"rate-test-model-1a4b6d-kserve-744947c7f5-xx577\" (UID: \"51f02198-f392-4f35-b68e-9ad3fb6810df\") " pod="ai-tenant-e2e-shared-b-w3-1a4b6d/rate-test-model-1a4b6d-kserve-744947c7f5-xx577"
Sep 18 14:11:55.494950 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:55.494932    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zxsp\" (UniqueName: \"kubernetes.io/projected/51f02198-f392-4f35-b68e-9ad3fb6810df-kube-api-access-2zxsp\") pod \"rate-test-model-1a4b6d-kserve-744947c7f5-xx577\" (UID: \"51f02198-f392-4f35-b68e-9ad3fb6810df\") " pod="ai-tenant-e2e-shared-b-w3-1a4b6d/rate-test-model-1a4b6d-kserve-744947c7f5-xx577"
Sep 18 14:11:55.619405 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:55.619365    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-1a4b6d/rate-test-model-1a4b6d-kserve-744947c7f5-xx577"
Sep 18 14:11:55.895319 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:55.895278    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1a4b6d/rate-test-model-1a4b6d-kserve-744947c7f5-xx577"]
Sep 18 14:11:55.897572 ip-10-0-141-222 kubenswrapper[2577]: W0918 14:11:55.897545    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod51f02198_f392_4f35_b68e_9ad3fb6810df.slice/crio-b29d8f6876ba5dd2c9362e407fd3562ec882dea6f36f942c6ef9fdf20fbec922 WatchSource:0}: Error finding container b29d8f6876ba5dd2c9362e407fd3562ec882dea6f36f942c6ef9fdf20fbec922: Status 404 returned error can't find the container with id b29d8f6876ba5dd2c9362e407fd3562ec882dea6f36f942c6ef9fdf20fbec922
Sep 18 14:11:56.326142 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:56.326054    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-1a4b6d/rate-test-model-1a4b6d-kserve-744947c7f5-xx577" event={"ID":"51f02198-f392-4f35-b68e-9ad3fb6810df","Type":"ContainerStarted","Data":"73f4d348b2f3f9c5da59038db6ac2ca1c07b7d3ba3e4a77e434ef420b4878098"}
Sep 18 14:11:56.326142 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:56.326095    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-1a4b6d/rate-test-model-1a4b6d-kserve-744947c7f5-xx577" event={"ID":"51f02198-f392-4f35-b68e-9ad3fb6810df","Type":"ContainerStarted","Data":"b29d8f6876ba5dd2c9362e407fd3562ec882dea6f36f942c6ef9fdf20fbec922"}
Sep 18 14:11:56.346091 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:56.346034    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-1a4b6d/rate-test-model-1a4b6d-kserve-744947c7f5-xx577" podStartSLOduration=1.346012449 podStartE2EDuration="1.346012449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:11:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:11:56.344347615 +0000 UTC m=+1665.397006942" watchObservedRunningTime="2026-09-18 14:11:56.346012449 +0000 UTC m=+1665.398671776"
Sep 18 14:11:56.451662 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:56.451624    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:56.517541 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:56.517498    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:59.208988 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:11:59.208932    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 14:12:05.620124 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:12:05.620075    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-1a4b6d/rate-test-model-1a4b6d-kserve-744947c7f5-xx577"
Sep 18 14:12:05.620596 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:12:05.620145    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-1a4b6d/rate-test-model-1a4b6d-kserve-744947c7f5-xx577"
Sep 18 14:12:05.633209 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:12:05.633175    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-1a4b6d/rate-test-model-1a4b6d-kserve-744947c7f5-xx577"
Sep 18 14:12:06.391126 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:12:06.391092    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-1a4b6d/rate-test-model-1a4b6d-kserve-744947c7f5-xx577"
Sep 18 14:12:14.004769 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:12:14.004719    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 14:12:24.406239 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:12:24.406199    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 14:12:29.662605 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:12:29.662507    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:30.685196 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:12:30.685160    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1a4b6d/rate-test-model-1a4b6d-kserve-744947c7f5-xx577"]
Sep 18 14:12:30.894631 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:12:30.894594    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:31.125350 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:12:31.125317    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:31.468187 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:12:31.468089    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:31.792960 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:12:31.792865    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:33.935911 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:12:33.935878    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:34.747813 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:12:34.747763    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:34.847185 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:12:34.847128    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:34.948565 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:12:34.948528    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:35.229615 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:12:35.229578    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 14:12:35.973200 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:12:35.973163    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:36.826049 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:12:36.826007    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:36.987310 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:12:36.987281    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:37.179660 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:12:37.179630    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:37.772220 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:12:37.772154    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:37.899082 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:12:37.899051    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:45.395251 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:12:45.395212    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 14:12:45.704423 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:12:45.704313    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-1a4b6d/rate-test-model-1a4b6d-kserve-744947c7f5-xx577" podUID="51f02198-f392-4f35-b68e-9ad3fb6810df" containerName="main" containerID="cri-o://73f4d348b2f3f9c5da59038db6ac2ca1c07b7d3ba3e4a77e434ef420b4878098" gracePeriod=45
Sep 18 14:12:45.948246 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:12:45.948217    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-1a4b6d/rate-test-model-1a4b6d-kserve-744947c7f5-xx577"
Sep 18 14:12:46.018466 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:12:46.018379    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51f02198-f392-4f35-b68e-9ad3fb6810df-model-cache\") pod \"51f02198-f392-4f35-b68e-9ad3fb6810df\" (UID: \"51f02198-f392-4f35-b68e-9ad3fb6810df\") "
Sep 18 14:12:46.018466 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:12:46.018426    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2zxsp\" (UniqueName: \"kubernetes.io/projected/51f02198-f392-4f35-b68e-9ad3fb6810df-kube-api-access-2zxsp\") pod \"51f02198-f392-4f35-b68e-9ad3fb6810df\" (UID: \"51f02198-f392-4f35-b68e-9ad3fb6810df\") "
Sep 18 14:12:46.018466 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:12:46.018447    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51f02198-f392-4f35-b68e-9ad3fb6810df-tls-certs\") pod \"51f02198-f392-4f35-b68e-9ad3fb6810df\" (UID: \"51f02198-f392-4f35-b68e-9ad3fb6810df\") "
Sep 18 14:12:46.018757 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:12:46.018502    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51f02198-f392-4f35-b68e-9ad3fb6810df-dshm\") pod \"51f02198-f392-4f35-b68e-9ad3fb6810df\" (UID: \"51f02198-f392-4f35-b68e-9ad3fb6810df\") "
Sep 18 14:12:46.018757 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:12:46.018536    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51f02198-f392-4f35-b68e-9ad3fb6810df-tmp-dir\") pod \"51f02198-f392-4f35-b68e-9ad3fb6810df\" (UID: \"51f02198-f392-4f35-b68e-9ad3fb6810df\") "
Sep 18 14:12:46.018757 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:12:46.018558    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51f02198-f392-4f35-b68e-9ad3fb6810df-home\") pod \"51f02198-f392-4f35-b68e-9ad3fb6810df\" (UID: \"51f02198-f392-4f35-b68e-9ad3fb6810df\") "
Sep 18 14:12:46.018757 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:12:46.018706    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51f02198-f392-4f35-b68e-9ad3fb6810df-model-cache" (OuterVolumeSpecName: "model-cache") pod "51f02198-f392-4f35-b68e-9ad3fb6810df" (UID: "51f02198-f392-4f35-b68e-9ad3fb6810df"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:12:46.018951 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:12:46.018805    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51f02198-f392-4f35-b68e-9ad3fb6810df-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "51f02198-f392-4f35-b68e-9ad3fb6810df" (UID: "51f02198-f392-4f35-b68e-9ad3fb6810df"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:12:46.018951 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:12:46.018857    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51f02198-f392-4f35-b68e-9ad3fb6810df-model-cache\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:12:46.018951 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:12:46.018900    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51f02198-f392-4f35-b68e-9ad3fb6810df-home" (OuterVolumeSpecName: "home") pod "51f02198-f392-4f35-b68e-9ad3fb6810df" (UID: "51f02198-f392-4f35-b68e-9ad3fb6810df"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:12:46.020713 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:12:46.020691    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51f02198-f392-4f35-b68e-9ad3fb6810df-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "51f02198-f392-4f35-b68e-9ad3fb6810df" (UID: "51f02198-f392-4f35-b68e-9ad3fb6810df"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:12:46.020713 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:12:46.020698    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51f02198-f392-4f35-b68e-9ad3fb6810df-dshm" (OuterVolumeSpecName: "dshm") pod "51f02198-f392-4f35-b68e-9ad3fb6810df" (UID: "51f02198-f392-4f35-b68e-9ad3fb6810df"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:12:46.020857 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:12:46.020789    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51f02198-f392-4f35-b68e-9ad3fb6810df-kube-api-access-2zxsp" (OuterVolumeSpecName: "kube-api-access-2zxsp") pod "51f02198-f392-4f35-b68e-9ad3fb6810df" (UID: "51f02198-f392-4f35-b68e-9ad3fb6810df"). InnerVolumeSpecName "kube-api-access-2zxsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:12:46.119677 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:12:46.119631    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51f02198-f392-4f35-b68e-9ad3fb6810df-dshm\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:12:46.119677 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:12:46.119669    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51f02198-f392-4f35-b68e-9ad3fb6810df-tmp-dir\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:12:46.119677 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:12:46.119680    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51f02198-f392-4f35-b68e-9ad3fb6810df-home\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:12:46.119677 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:12:46.119688    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zxsp\" (UniqueName: \"kubernetes.io/projected/51f02198-f392-4f35-b68e-9ad3fb6810df-kube-api-access-2zxsp\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:12:46.120018 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:12:46.119697    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51f02198-f392-4f35-b68e-9ad3fb6810df-tls-certs\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:12:46.598947 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:12:46.598908    2577 generic.go:356] "Generic (PLEG): container finished" podID="51f02198-f392-4f35-b68e-9ad3fb6810df" containerID="73f4d348b2f3f9c5da59038db6ac2ca1c07b7d3ba3e4a77e434ef420b4878098" exitCode=0
Sep 18 14:12:46.598947 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:12:46.598952    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-1a4b6d/rate-test-model-1a4b6d-kserve-744947c7f5-xx577" event={"ID":"51f02198-f392-4f35-b68e-9ad3fb6810df","Type":"ContainerDied","Data":"73f4d348b2f3f9c5da59038db6ac2ca1c07b7d3ba3e4a77e434ef420b4878098"}
Sep 18 14:12:46.599558 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:12:46.598966    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-1a4b6d/rate-test-model-1a4b6d-kserve-744947c7f5-xx577"
Sep 18 14:12:46.599558 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:12:46.598990    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-1a4b6d/rate-test-model-1a4b6d-kserve-744947c7f5-xx577" event={"ID":"51f02198-f392-4f35-b68e-9ad3fb6810df","Type":"ContainerDied","Data":"b29d8f6876ba5dd2c9362e407fd3562ec882dea6f36f942c6ef9fdf20fbec922"}
Sep 18 14:12:46.599558 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:12:46.599012    2577 scope.go:117] "RemoveContainer" containerID="73f4d348b2f3f9c5da59038db6ac2ca1c07b7d3ba3e4a77e434ef420b4878098"
Sep 18 14:12:46.608667 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:12:46.608649    2577 scope.go:117] "RemoveContainer" containerID="73f4d348b2f3f9c5da59038db6ac2ca1c07b7d3ba3e4a77e434ef420b4878098"
Sep 18 14:12:46.608998 ip-10-0-141-222 kubenswrapper[2577]: E0918 14:12:46.608979    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73f4d348b2f3f9c5da59038db6ac2ca1c07b7d3ba3e4a77e434ef420b4878098\": container with ID starting with 73f4d348b2f3f9c5da59038db6ac2ca1c07b7d3ba3e4a77e434ef420b4878098 not found: ID does not exist" containerID="73f4d348b2f3f9c5da59038db6ac2ca1c07b7d3ba3e4a77e434ef420b4878098"
Sep 18 14:12:46.609061 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:12:46.609005    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73f4d348b2f3f9c5da59038db6ac2ca1c07b7d3ba3e4a77e434ef420b4878098"} err="failed to get container status \"73f4d348b2f3f9c5da59038db6ac2ca1c07b7d3ba3e4a77e434ef420b4878098\": rpc error: code = NotFound desc = could not find container \"73f4d348b2f3f9c5da59038db6ac2ca1c07b7d3ba3e4a77e434ef420b4878098\": container with ID starting with 73f4d348b2f3f9c5da59038db6ac2ca1c07b7d3ba3e4a77e434ef420b4878098 not found: ID does not exist"
Sep 18 14:12:46.620172 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:12:46.620144    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1a4b6d/rate-test-model-1a4b6d-kserve-744947c7f5-xx577"]
Sep 18 14:12:46.624021 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:12:46.623995    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1a4b6d/rate-test-model-1a4b6d-kserve-744947c7f5-xx577"]
Sep 18 14:12:47.107865 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:12:47.107807    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51f02198-f392-4f35-b68e-9ad3fb6810df" path="/var/lib/kubelet/pods/51f02198-f392-4f35-b68e-9ad3fb6810df/volumes"
Sep 18 14:12:59.653769 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:12:59.653719    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:59.753779 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:12:59.753742    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:00.293147 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:00.293114    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:00.459095 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:00.459061    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:00.586763 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:00.586671    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82"]
Sep 18 14:13:00.587160 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:00.587140    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51f02198-f392-4f35-b68e-9ad3fb6810df" containerName="main"
Sep 18 14:13:00.587257 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:00.587164    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="51f02198-f392-4f35-b68e-9ad3fb6810df" containerName="main"
Sep 18 14:13:00.587319 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:00.587267    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="51f02198-f392-4f35-b68e-9ad3fb6810df" containerName="main"
Sep 18 14:13:00.591782 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:00.591757    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82"
Sep 18 14:13:00.594273 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:00.594250    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-9d6301-openshift-default-dockercfg-lqt8p\""
Sep 18 14:13:00.611469 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:00.611441    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82"]
Sep 18 14:13:00.628632 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:00.628593    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:00.745221 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:00.745173    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/373fdc83-6fb7-45b4-aff8-907be2384680-istio-token\") pod \"e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82\" (UID: \"373fdc83-6fb7-45b4-aff8-907be2384680\") " pod="openshift-ingress/e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82"
Sep 18 14:13:00.745685 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:00.745230    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/373fdc83-6fb7-45b4-aff8-907be2384680-istio-podinfo\") pod \"e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82\" (UID: \"373fdc83-6fb7-45b4-aff8-907be2384680\") " pod="openshift-ingress/e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82"
Sep 18 14:13:00.745685 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:00.745287    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/373fdc83-6fb7-45b4-aff8-907be2384680-istiod-ca-cert\") pod \"e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82\" (UID: \"373fdc83-6fb7-45b4-aff8-907be2384680\") " pod="openshift-ingress/e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82"
Sep 18 14:13:00.745685 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:00.745367    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/373fdc83-6fb7-45b4-aff8-907be2384680-istio-envoy\") pod \"e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82\" (UID: \"373fdc83-6fb7-45b4-aff8-907be2384680\") " pod="openshift-ingress/e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82"
Sep 18 14:13:00.745685 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:00.745419    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/373fdc83-6fb7-45b4-aff8-907be2384680-workload-socket\") pod \"e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82\" (UID: \"373fdc83-6fb7-45b4-aff8-907be2384680\") " pod="openshift-ingress/e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82"
Sep 18 14:13:00.745685 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:00.745457    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/373fdc83-6fb7-45b4-aff8-907be2384680-istio-data\") pod \"e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82\" (UID: \"373fdc83-6fb7-45b4-aff8-907be2384680\") " pod="openshift-ingress/e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82"
Sep 18 14:13:00.745685 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:00.745487    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbrt4\" (UniqueName: \"kubernetes.io/projected/373fdc83-6fb7-45b4-aff8-907be2384680-kube-api-access-kbrt4\") pod \"e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82\" (UID: \"373fdc83-6fb7-45b4-aff8-907be2384680\") " pod="openshift-ingress/e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82"
Sep 18 14:13:00.745685 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:00.745521    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/373fdc83-6fb7-45b4-aff8-907be2384680-workload-certs\") pod \"e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82\" (UID: \"373fdc83-6fb7-45b4-aff8-907be2384680\") " pod="openshift-ingress/e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82"
Sep 18 14:13:00.745685 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:00.745550    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/373fdc83-6fb7-45b4-aff8-907be2384680-credential-socket\") pod \"e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82\" (UID: \"373fdc83-6fb7-45b4-aff8-907be2384680\") " pod="openshift-ingress/e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82"
Sep 18 14:13:00.846500 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:00.846464    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/373fdc83-6fb7-45b4-aff8-907be2384680-istio-envoy\") pod \"e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82\" (UID: \"373fdc83-6fb7-45b4-aff8-907be2384680\") " pod="openshift-ingress/e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82"
Sep 18 14:13:00.846712 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:00.846507    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/373fdc83-6fb7-45b4-aff8-907be2384680-workload-socket\") pod \"e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82\" (UID: \"373fdc83-6fb7-45b4-aff8-907be2384680\") " pod="openshift-ingress/e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82"
Sep 18 14:13:00.846712 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:00.846534    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/373fdc83-6fb7-45b4-aff8-907be2384680-istio-data\") pod \"e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82\" (UID: \"373fdc83-6fb7-45b4-aff8-907be2384680\") " pod="openshift-ingress/e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82"
Sep 18 14:13:00.846712 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:00.846642    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbrt4\" (UniqueName: \"kubernetes.io/projected/373fdc83-6fb7-45b4-aff8-907be2384680-kube-api-access-kbrt4\") pod \"e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82\" (UID: \"373fdc83-6fb7-45b4-aff8-907be2384680\") " pod="openshift-ingress/e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82"
Sep 18 14:13:00.846712 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:00.846702    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/373fdc83-6fb7-45b4-aff8-907be2384680-workload-certs\") pod \"e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82\" (UID: \"373fdc83-6fb7-45b4-aff8-907be2384680\") " pod="openshift-ingress/e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82"
Sep 18 14:13:00.846984 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:00.846725    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/373fdc83-6fb7-45b4-aff8-907be2384680-credential-socket\") pod \"e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82\" (UID: \"373fdc83-6fb7-45b4-aff8-907be2384680\") " pod="openshift-ingress/e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82"
Sep 18 14:13:00.846984 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:00.846757    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/373fdc83-6fb7-45b4-aff8-907be2384680-istio-token\") pod \"e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82\" (UID: \"373fdc83-6fb7-45b4-aff8-907be2384680\") " pod="openshift-ingress/e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82"
Sep 18 14:13:00.846984 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:00.846856    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/373fdc83-6fb7-45b4-aff8-907be2384680-istio-podinfo\") pod \"e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82\" (UID: \"373fdc83-6fb7-45b4-aff8-907be2384680\") " pod="openshift-ingress/e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82"
Sep 18 14:13:00.846984 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:00.846890    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/373fdc83-6fb7-45b4-aff8-907be2384680-istiod-ca-cert\") pod \"e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82\" (UID: \"373fdc83-6fb7-45b4-aff8-907be2384680\") " pod="openshift-ingress/e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82"
Sep 18 14:13:00.847231 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:00.847202    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/373fdc83-6fb7-45b4-aff8-907be2384680-workload-socket\") pod \"e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82\" (UID: \"373fdc83-6fb7-45b4-aff8-907be2384680\") " pod="openshift-ingress/e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82"
Sep 18 14:13:00.847370 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:00.847258    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/373fdc83-6fb7-45b4-aff8-907be2384680-credential-socket\") pod \"e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82\" (UID: \"373fdc83-6fb7-45b4-aff8-907be2384680\") " pod="openshift-ingress/e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82"
Sep 18 14:13:00.847370 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:00.847296    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/373fdc83-6fb7-45b4-aff8-907be2384680-istio-data\") pod \"e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82\" (UID: \"373fdc83-6fb7-45b4-aff8-907be2384680\") " pod="openshift-ingress/e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82"
Sep 18 14:13:00.847370 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:00.847322    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/373fdc83-6fb7-45b4-aff8-907be2384680-workload-certs\") pod \"e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82\" (UID: \"373fdc83-6fb7-45b4-aff8-907be2384680\") " pod="openshift-ingress/e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82"
Sep 18 14:13:00.847644 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:00.847624    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/373fdc83-6fb7-45b4-aff8-907be2384680-istiod-ca-cert\") pod \"e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82\" (UID: \"373fdc83-6fb7-45b4-aff8-907be2384680\") " pod="openshift-ingress/e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82"
Sep 18 14:13:00.848988 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:00.848966    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/373fdc83-6fb7-45b4-aff8-907be2384680-istio-envoy\") pod \"e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82\" (UID: \"373fdc83-6fb7-45b4-aff8-907be2384680\") " pod="openshift-ingress/e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82"
Sep 18 14:13:00.849542 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:00.849486    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/373fdc83-6fb7-45b4-aff8-907be2384680-istio-podinfo\") pod \"e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82\" (UID: \"373fdc83-6fb7-45b4-aff8-907be2384680\") " pod="openshift-ingress/e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82"
Sep 18 14:13:00.854745 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:00.854719    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbrt4\" (UniqueName: \"kubernetes.io/projected/373fdc83-6fb7-45b4-aff8-907be2384680-kube-api-access-kbrt4\") pod \"e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82\" (UID: \"373fdc83-6fb7-45b4-aff8-907be2384680\") " pod="openshift-ingress/e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82"
Sep 18 14:13:00.854880 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:00.854866    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/373fdc83-6fb7-45b4-aff8-907be2384680-istio-token\") pod \"e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82\" (UID: \"373fdc83-6fb7-45b4-aff8-907be2384680\") " pod="openshift-ingress/e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82"
Sep 18 14:13:00.901997 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:00.901962    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82"
Sep 18 14:13:01.003749 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:01.003717    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:01.185522 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:01.185438    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82"]
Sep 18 14:13:01.189177 ip-10-0-141-222 kubenswrapper[2577]: W0918 14:13:01.189146    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod373fdc83_6fb7_45b4_aff8_907be2384680.slice/crio-4329b1882041694cdb3a289121a3fc3b34491a9f2e94a9939145d12b1fe5e7a4 WatchSource:0}: Error finding container 4329b1882041694cdb3a289121a3fc3b34491a9f2e94a9939145d12b1fe5e7a4: Status 404 returned error can't find the container with id 4329b1882041694cdb3a289121a3fc3b34491a9f2e94a9939145d12b1fe5e7a4
Sep 18 14:13:01.192105 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:01.192062    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:13:01.192188 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:01.192156    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:13:01.192229 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:01.192201    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:13:01.266760 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:01.266700    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:01.684515 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:01.684479    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82" event={"ID":"373fdc83-6fb7-45b4-aff8-907be2384680","Type":"ContainerStarted","Data":"3f777be22400bb11c97961366263faed42b8371620b349ecc2c34bd409eb851b"}
Sep 18 14:13:01.684774 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:01.684549    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82" event={"ID":"373fdc83-6fb7-45b4-aff8-907be2384680","Type":"ContainerStarted","Data":"4329b1882041694cdb3a289121a3fc3b34491a9f2e94a9939145d12b1fe5e7a4"}
Sep 18 14:13:01.706783 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:01.706727    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82" podStartSLOduration=1.706707916 podStartE2EDuration="1.706707916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:13:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:13:01.704568044 +0000 UTC m=+1730.757227371" watchObservedRunningTime="2026-09-18 14:13:01.706707916 +0000 UTC m=+1730.759367244"
Sep 18 14:13:01.903094 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:01.903065    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82"
Sep 18 14:13:02.054883 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:02.054782    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:02.905462 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:02.905419    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:02.908099 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:02.908078    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82"
Sep 18 14:13:03.542215 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:03.542180    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:03.649017 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:03.648978    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:03.695202 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:03.695163    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82"
Sep 18 14:13:03.696273 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:03.696252    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82"
Sep 18 14:13:03.759207 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:03.759167    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:04.546865 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:04.546833    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:05.327156 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:05.327129    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:06.110206 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:06.110171    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:06.211507 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:06.211475    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:06.351205 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:06.351167    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:06.674439 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:06.674396    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:06.680875 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:06.680842    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9d6301-6b548647c4-mc4tj"]
Sep 18 14:13:06.684227 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:06.684209    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d6301-6b548647c4-mc4tj"
Sep 18 14:13:06.686450 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:06.686426    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-9d6301\""
Sep 18 14:13:06.695708 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:06.695684    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9d6301-6b548647c4-mc4tj"]
Sep 18 14:13:06.765366 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:06.765333    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:06.801322 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:06.801286    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3221e20d-d8d1-466b-91e2-6d5eec88d521-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9d6301-6b548647c4-mc4tj\" (UID: \"3221e20d-d8d1-466b-91e2-6d5eec88d521\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d6301-6b548647c4-mc4tj"
Sep 18 14:13:06.801537 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:06.801472    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5xjt\" (UniqueName: \"kubernetes.io/projected/3221e20d-d8d1-466b-91e2-6d5eec88d521-kube-api-access-l5xjt\") pod \"payload-pre-processing-e2e-ipp-b-9d6301-6b548647c4-mc4tj\" (UID: \"3221e20d-d8d1-466b-91e2-6d5eec88d521\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d6301-6b548647c4-mc4tj"
Sep 18 14:13:06.902623 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:06.902571    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5xjt\" (UniqueName: \"kubernetes.io/projected/3221e20d-d8d1-466b-91e2-6d5eec88d521-kube-api-access-l5xjt\") pod \"payload-pre-processing-e2e-ipp-b-9d6301-6b548647c4-mc4tj\" (UID: \"3221e20d-d8d1-466b-91e2-6d5eec88d521\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d6301-6b548647c4-mc4tj"
Sep 18 14:13:06.902868 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:06.902633    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3221e20d-d8d1-466b-91e2-6d5eec88d521-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9d6301-6b548647c4-mc4tj\" (UID: \"3221e20d-d8d1-466b-91e2-6d5eec88d521\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d6301-6b548647c4-mc4tj"
Sep 18 14:13:06.903417 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:06.903397    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3221e20d-d8d1-466b-91e2-6d5eec88d521-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9d6301-6b548647c4-mc4tj\" (UID: \"3221e20d-d8d1-466b-91e2-6d5eec88d521\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d6301-6b548647c4-mc4tj"
Sep 18 14:13:06.911195 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:06.911166    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5xjt\" (UniqueName: \"kubernetes.io/projected/3221e20d-d8d1-466b-91e2-6d5eec88d521-kube-api-access-l5xjt\") pod \"payload-pre-processing-e2e-ipp-b-9d6301-6b548647c4-mc4tj\" (UID: \"3221e20d-d8d1-466b-91e2-6d5eec88d521\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d6301-6b548647c4-mc4tj"
Sep 18 14:13:06.995321 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:06.995216    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d6301-6b548647c4-mc4tj"
Sep 18 14:13:07.271050 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:07.270953    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9d6301-6b548647c4-mc4tj"]
Sep 18 14:13:07.274480 ip-10-0-141-222 kubenswrapper[2577]: W0918 14:13:07.274441    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3221e20d_d8d1_466b_91e2_6d5eec88d521.slice/crio-5f879f63d47e045d19bbeb950d4123f340577bc539ecec7f4ebf49acc6b897a8 WatchSource:0}: Error finding container 5f879f63d47e045d19bbeb950d4123f340577bc539ecec7f4ebf49acc6b897a8: Status 404 returned error can't find the container with id 5f879f63d47e045d19bbeb950d4123f340577bc539ecec7f4ebf49acc6b897a8
Sep 18 14:13:07.719416 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:07.719378    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d6301-6b548647c4-mc4tj" event={"ID":"3221e20d-d8d1-466b-91e2-6d5eec88d521","Type":"ContainerStarted","Data":"8ab325768820fd1ad7a1fd880c93ca8b15893d96f0f6a13b109097d6a5cdc5bf"}
Sep 18 14:13:07.719638 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:07.719436    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d6301-6b548647c4-mc4tj" event={"ID":"3221e20d-d8d1-466b-91e2-6d5eec88d521","Type":"ContainerStarted","Data":"5f879f63d47e045d19bbeb950d4123f340577bc539ecec7f4ebf49acc6b897a8"}
Sep 18 14:13:07.719638 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:07.719579    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d6301-6b548647c4-mc4tj"
Sep 18 14:13:07.736080 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:07.736023    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d6301-6b548647c4-mc4tj" podStartSLOduration=1.736005939 podStartE2EDuration="1.736005939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:13:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:13:07.734929495 +0000 UTC m=+1736.787588825" watchObservedRunningTime="2026-09-18 14:13:07.736005939 +0000 UTC m=+1736.788665266"
Sep 18 14:13:11.008736 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:11.008700    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 14:13:18.727360 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:18.727318    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d6301-6b548647c4-mc4tj"
Sep 18 14:13:19.737012 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:19.736965    2577 patch_prober.go:28] interesting pod/e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 18 14:13:19.737416 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:19.737038    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82" podUID="373fdc83-6fb7-45b4-aff8-907be2384680" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 18 14:13:21.804135 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:21.804033    2577 generic.go:356] "Generic (PLEG): container finished" podID="373fdc83-6fb7-45b4-aff8-907be2384680" containerID="3f777be22400bb11c97961366263faed42b8371620b349ecc2c34bd409eb851b" exitCode=0
Sep 18 14:13:21.804135 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:21.804110    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82" event={"ID":"373fdc83-6fb7-45b4-aff8-907be2384680","Type":"ContainerDied","Data":"3f777be22400bb11c97961366263faed42b8371620b349ecc2c34bd409eb851b"}
Sep 18 14:13:21.804575 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:21.804557    2577 scope.go:117] "RemoveContainer" containerID="3f777be22400bb11c97961366263faed42b8371620b349ecc2c34bd409eb851b"
Sep 18 14:13:21.805668 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:21.805635    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:13:21.805778 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:21.805704    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:13:21.805778 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:21.805737    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:13:22.811263 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:22.811228    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82" event={"ID":"373fdc83-6fb7-45b4-aff8-907be2384680","Type":"ContainerStarted","Data":"644b60cfd8891d2329d4d37740aa9868fc8ccd0bc5458e21d3ae21ffc6b5fb11"}
Sep 18 14:13:22.902402 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:22.902357    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82"
Sep 18 14:13:22.902661 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:22.902632    2577 patch_prober.go:28] interesting pod/e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Sep 18 14:13:22.902725 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:22.902699    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82" podUID="373fdc83-6fb7-45b4-aff8-907be2384680" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Sep 18 14:13:23.903366 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:23.903320    2577 patch_prober.go:28] interesting pod/e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Sep 18 14:13:23.904045 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:23.904005    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82" podUID="373fdc83-6fb7-45b4-aff8-907be2384680" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Sep 18 14:13:24.903006 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:24.902972    2577 patch_prober.go:28] interesting pod/e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Sep 18 14:13:24.903295 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:24.903251    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82" podUID="373fdc83-6fb7-45b4-aff8-907be2384680" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Sep 18 14:13:25.902681 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:25.902646    2577 patch_prober.go:28] interesting pod/e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Sep 18 14:13:25.903117 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:25.902707    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82" podUID="373fdc83-6fb7-45b4-aff8-907be2384680" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Sep 18 14:13:26.903141 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:26.903098    2577 patch_prober.go:28] interesting pod/e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Sep 18 14:13:26.903548 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:26.903170    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82" podUID="373fdc83-6fb7-45b4-aff8-907be2384680" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Sep 18 14:13:27.903298 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:27.903261    2577 patch_prober.go:28] interesting pod/e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Sep 18 14:13:27.903719 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:27.903325    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82" podUID="373fdc83-6fb7-45b4-aff8-907be2384680" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Sep 18 14:13:28.236524 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:28.236427    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:28.595002 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:28.594971    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:28.893805 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:28.893720    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:28.906233 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:28.906209    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82"
Sep 18 14:13:28.906582 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:28.906525    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82"
Sep 18 14:13:28.907194 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:28.907176    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82"
Sep 18 14:13:29.725637 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:29.725609    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:30.454527 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:30.454458    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:31.149569 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:31.149534    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:31.849470 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:31.849428    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:40.898785 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:40.898747    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-b6289c/ipp-route-b6289c-kserve-7749784cff-bkpzl"]
Sep 18 14:13:40.901237 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:40.901221    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-b6289c/ipp-route-b6289c-kserve-7749784cff-bkpzl"
Sep 18 14:13:40.904325 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:40.904303    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-b6289c\"/\"openshift-service-ca.crt\""
Sep 18 14:13:40.904325 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:40.904313    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-b6289c\"/\"ipp-route-b6289c-kserve-self-signed-certs\""
Sep 18 14:13:40.904516 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:40.904313    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-b6289c\"/\"default-dockercfg-rvk5q\""
Sep 18 14:13:40.904516 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:40.904365    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-b6289c\"/\"kube-root-ca.crt\""
Sep 18 14:13:40.911666 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:40.910745    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-b6289c/ipp-route-b6289c-kserve-7749784cff-bkpzl"]
Sep 18 14:13:41.002112 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:41.002078    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:41.019779 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:41.019751    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhgpr\" (UniqueName: \"kubernetes.io/projected/e9372335-1cf7-4105-9aac-e104f61ca3ac-kube-api-access-lhgpr\") pod \"ipp-route-b6289c-kserve-7749784cff-bkpzl\" (UID: \"e9372335-1cf7-4105-9aac-e104f61ca3ac\") " pod="ai-tenant-e2e-ipp-a-b6289c/ipp-route-b6289c-kserve-7749784cff-bkpzl"
Sep 18 14:13:41.019900 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:41.019801    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9372335-1cf7-4105-9aac-e104f61ca3ac-tmp-dir\") pod \"ipp-route-b6289c-kserve-7749784cff-bkpzl\" (UID: \"e9372335-1cf7-4105-9aac-e104f61ca3ac\") " pod="ai-tenant-e2e-ipp-a-b6289c/ipp-route-b6289c-kserve-7749784cff-bkpzl"
Sep 18 14:13:41.019900 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:41.019854    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9372335-1cf7-4105-9aac-e104f61ca3ac-model-cache\") pod \"ipp-route-b6289c-kserve-7749784cff-bkpzl\" (UID: \"e9372335-1cf7-4105-9aac-e104f61ca3ac\") " pod="ai-tenant-e2e-ipp-a-b6289c/ipp-route-b6289c-kserve-7749784cff-bkpzl"
Sep 18 14:13:41.019900 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:41.019878    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9372335-1cf7-4105-9aac-e104f61ca3ac-tls-certs\") pod \"ipp-route-b6289c-kserve-7749784cff-bkpzl\" (UID: \"e9372335-1cf7-4105-9aac-e104f61ca3ac\") " pod="ai-tenant-e2e-ipp-a-b6289c/ipp-route-b6289c-kserve-7749784cff-bkpzl"
Sep 18 14:13:41.020024 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:41.019905    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9372335-1cf7-4105-9aac-e104f61ca3ac-dshm\") pod \"ipp-route-b6289c-kserve-7749784cff-bkpzl\" (UID: \"e9372335-1cf7-4105-9aac-e104f61ca3ac\") " pod="ai-tenant-e2e-ipp-a-b6289c/ipp-route-b6289c-kserve-7749784cff-bkpzl"
Sep 18 14:13:41.020024 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:41.019964    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9372335-1cf7-4105-9aac-e104f61ca3ac-home\") pod \"ipp-route-b6289c-kserve-7749784cff-bkpzl\" (UID: \"e9372335-1cf7-4105-9aac-e104f61ca3ac\") " pod="ai-tenant-e2e-ipp-a-b6289c/ipp-route-b6289c-kserve-7749784cff-bkpzl"
Sep 18 14:13:41.121091 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:41.121061    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9372335-1cf7-4105-9aac-e104f61ca3ac-tmp-dir\") pod \"ipp-route-b6289c-kserve-7749784cff-bkpzl\" (UID: \"e9372335-1cf7-4105-9aac-e104f61ca3ac\") " pod="ai-tenant-e2e-ipp-a-b6289c/ipp-route-b6289c-kserve-7749784cff-bkpzl"
Sep 18 14:13:41.121091 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:41.121093    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9372335-1cf7-4105-9aac-e104f61ca3ac-model-cache\") pod \"ipp-route-b6289c-kserve-7749784cff-bkpzl\" (UID: \"e9372335-1cf7-4105-9aac-e104f61ca3ac\") " pod="ai-tenant-e2e-ipp-a-b6289c/ipp-route-b6289c-kserve-7749784cff-bkpzl"
Sep 18 14:13:41.121301 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:41.121119    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9372335-1cf7-4105-9aac-e104f61ca3ac-tls-certs\") pod \"ipp-route-b6289c-kserve-7749784cff-bkpzl\" (UID: \"e9372335-1cf7-4105-9aac-e104f61ca3ac\") " pod="ai-tenant-e2e-ipp-a-b6289c/ipp-route-b6289c-kserve-7749784cff-bkpzl"
Sep 18 14:13:41.121301 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:41.121155    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9372335-1cf7-4105-9aac-e104f61ca3ac-dshm\") pod \"ipp-route-b6289c-kserve-7749784cff-bkpzl\" (UID: \"e9372335-1cf7-4105-9aac-e104f61ca3ac\") " pod="ai-tenant-e2e-ipp-a-b6289c/ipp-route-b6289c-kserve-7749784cff-bkpzl"
Sep 18 14:13:41.121301 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:41.121186    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9372335-1cf7-4105-9aac-e104f61ca3ac-home\") pod \"ipp-route-b6289c-kserve-7749784cff-bkpzl\" (UID: \"e9372335-1cf7-4105-9aac-e104f61ca3ac\") " pod="ai-tenant-e2e-ipp-a-b6289c/ipp-route-b6289c-kserve-7749784cff-bkpzl"
Sep 18 14:13:41.121301 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:41.121273    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhgpr\" (UniqueName: \"kubernetes.io/projected/e9372335-1cf7-4105-9aac-e104f61ca3ac-kube-api-access-lhgpr\") pod \"ipp-route-b6289c-kserve-7749784cff-bkpzl\" (UID: \"e9372335-1cf7-4105-9aac-e104f61ca3ac\") " pod="ai-tenant-e2e-ipp-a-b6289c/ipp-route-b6289c-kserve-7749784cff-bkpzl"
Sep 18 14:13:41.121550 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:41.121523    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9372335-1cf7-4105-9aac-e104f61ca3ac-model-cache\") pod \"ipp-route-b6289c-kserve-7749784cff-bkpzl\" (UID: \"e9372335-1cf7-4105-9aac-e104f61ca3ac\") " pod="ai-tenant-e2e-ipp-a-b6289c/ipp-route-b6289c-kserve-7749784cff-bkpzl"
Sep 18 14:13:41.121684 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:41.121551    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9372335-1cf7-4105-9aac-e104f61ca3ac-tmp-dir\") pod \"ipp-route-b6289c-kserve-7749784cff-bkpzl\" (UID: \"e9372335-1cf7-4105-9aac-e104f61ca3ac\") " pod="ai-tenant-e2e-ipp-a-b6289c/ipp-route-b6289c-kserve-7749784cff-bkpzl"
Sep 18 14:13:41.121684 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:41.121605    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9372335-1cf7-4105-9aac-e104f61ca3ac-home\") pod \"ipp-route-b6289c-kserve-7749784cff-bkpzl\" (UID: \"e9372335-1cf7-4105-9aac-e104f61ca3ac\") " pod="ai-tenant-e2e-ipp-a-b6289c/ipp-route-b6289c-kserve-7749784cff-bkpzl"
Sep 18 14:13:41.123453 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:41.123431    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9372335-1cf7-4105-9aac-e104f61ca3ac-dshm\") pod \"ipp-route-b6289c-kserve-7749784cff-bkpzl\" (UID: \"e9372335-1cf7-4105-9aac-e104f61ca3ac\") " pod="ai-tenant-e2e-ipp-a-b6289c/ipp-route-b6289c-kserve-7749784cff-bkpzl"
Sep 18 14:13:41.123891 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:41.123877    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9372335-1cf7-4105-9aac-e104f61ca3ac-tls-certs\") pod \"ipp-route-b6289c-kserve-7749784cff-bkpzl\" (UID: \"e9372335-1cf7-4105-9aac-e104f61ca3ac\") " pod="ai-tenant-e2e-ipp-a-b6289c/ipp-route-b6289c-kserve-7749784cff-bkpzl"
Sep 18 14:13:41.129207 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:41.129189    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhgpr\" (UniqueName: \"kubernetes.io/projected/e9372335-1cf7-4105-9aac-e104f61ca3ac-kube-api-access-lhgpr\") pod \"ipp-route-b6289c-kserve-7749784cff-bkpzl\" (UID: \"e9372335-1cf7-4105-9aac-e104f61ca3ac\") " pod="ai-tenant-e2e-ipp-a-b6289c/ipp-route-b6289c-kserve-7749784cff-bkpzl"
Sep 18 14:13:41.215074 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:41.214986    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-b6289c/ipp-route-b6289c-kserve-7749784cff-bkpzl"
Sep 18 14:13:41.487497 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:41.487453    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-b6289c/ipp-route-b6289c-kserve-7749784cff-bkpzl"]
Sep 18 14:13:41.488692 ip-10-0-141-222 kubenswrapper[2577]: W0918 14:13:41.488662    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode9372335_1cf7_4105_9aac_e104f61ca3ac.slice/crio-26639a1e91b47073107a2b800c7f7a6fb52f3dac17768f5ced40a6d73c56a04c WatchSource:0}: Error finding container 26639a1e91b47073107a2b800c7f7a6fb52f3dac17768f5ced40a6d73c56a04c: Status 404 returned error can't find the container with id 26639a1e91b47073107a2b800c7f7a6fb52f3dac17768f5ced40a6d73c56a04c
Sep 18 14:13:41.921939 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:41.921904    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-b6289c/ipp-route-b6289c-kserve-7749784cff-bkpzl" event={"ID":"e9372335-1cf7-4105-9aac-e104f61ca3ac","Type":"ContainerStarted","Data":"e10190ec9fe599314a0b0f8fc097a3fa6103e1b1e847f901d0c997a29d4a6476"}
Sep 18 14:13:41.921939 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:41.921945    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-b6289c/ipp-route-b6289c-kserve-7749784cff-bkpzl" event={"ID":"e9372335-1cf7-4105-9aac-e104f61ca3ac","Type":"ContainerStarted","Data":"26639a1e91b47073107a2b800c7f7a6fb52f3dac17768f5ced40a6d73c56a04c"}
Sep 18 14:13:41.941050 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:41.941004    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-b6289c/ipp-route-b6289c-kserve-7749784cff-bkpzl" podStartSLOduration=1.940987753 podStartE2EDuration="1.940987753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:13:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:13:41.939929495 +0000 UTC m=+1770.992588823" watchObservedRunningTime="2026-09-18 14:13:41.940987753 +0000 UTC m=+1770.993647079"
Sep 18 14:13:44.989323 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:44.989288    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 14:13:51.215244 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:51.215210    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-b6289c/ipp-route-b6289c-kserve-7749784cff-bkpzl"
Sep 18 14:13:51.215244 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:51.215255    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-b6289c/ipp-route-b6289c-kserve-7749784cff-bkpzl"
Sep 18 14:13:51.227814 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:51.227787    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-b6289c/ipp-route-b6289c-kserve-7749784cff-bkpzl"
Sep 18 14:13:51.991734 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:13:51.991706    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-b6289c/ipp-route-b6289c-kserve-7749784cff-bkpzl"
Sep 18 14:14:00.904258 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:00.904217    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:08.543253 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:08.543218    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:08.605080 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:08.605046    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 14:14:13.254943 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:13.254915    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-8tnxl_69d3178a-5769-4b5f-a243-30a1bedc5bd6/console-operator/1.log"
Sep 18 14:14:13.258470 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:13.258445    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-8tnxl_69d3178a-5769-4b5f-a243-30a1bedc5bd6/console-operator/1.log"
Sep 18 14:14:15.508138 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:15.508101    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:15.772159 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:15.772079    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:15.871727 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:15.871694    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:19.158791 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:19.158751    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:19.489875 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:19.489781    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:19.608753 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:19.608713    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:20.103840 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:20.103781    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:21.007364 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:21.007328    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:21.769669 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:21.769639    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:22.399395 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:22.399366    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cafbd6-8b9445564-f9kww"]
Sep 18 14:14:22.401800 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:22.401780    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cafbd6-8b9445564-f9kww"
Sep 18 14:14:22.403984 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:22.403965    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-cafbd6-serving-cert\""
Sep 18 14:14:22.415241 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:22.415218    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cafbd6-8b9445564-f9kww"]
Sep 18 14:14:22.433911 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:22.433881    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:22.484065 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:22.484031    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2f5dc4d-52b8-45c8-a048-0c9be26ba9f2-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-cafbd6-8b9445564-f9kww\" (UID: \"e2f5dc4d-52b8-45c8-a048-0c9be26ba9f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cafbd6-8b9445564-f9kww"
Sep 18 14:14:22.484243 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:22.484070    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrrz2\" (UniqueName: \"kubernetes.io/projected/e2f5dc4d-52b8-45c8-a048-0c9be26ba9f2-kube-api-access-mrrz2\") pod \"maas-api-e2e-ipp-cleanup-cafbd6-8b9445564-f9kww\" (UID: \"e2f5dc4d-52b8-45c8-a048-0c9be26ba9f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cafbd6-8b9445564-f9kww"
Sep 18 14:14:22.484243 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:22.484149    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2f5dc4d-52b8-45c8-a048-0c9be26ba9f2-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-cafbd6-8b9445564-f9kww\" (UID: \"e2f5dc4d-52b8-45c8-a048-0c9be26ba9f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cafbd6-8b9445564-f9kww"
Sep 18 14:14:22.484243 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:22.484221    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2f5dc4d-52b8-45c8-a048-0c9be26ba9f2-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-cafbd6-8b9445564-f9kww\" (UID: \"e2f5dc4d-52b8-45c8-a048-0c9be26ba9f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cafbd6-8b9445564-f9kww"
Sep 18 14:14:22.536352 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:22.536319    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:22.585047 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:22.585015    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrrz2\" (UniqueName: \"kubernetes.io/projected/e2f5dc4d-52b8-45c8-a048-0c9be26ba9f2-kube-api-access-mrrz2\") pod \"maas-api-e2e-ipp-cleanup-cafbd6-8b9445564-f9kww\" (UID: \"e2f5dc4d-52b8-45c8-a048-0c9be26ba9f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cafbd6-8b9445564-f9kww"
Sep 18 14:14:22.585201 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:22.585078    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2f5dc4d-52b8-45c8-a048-0c9be26ba9f2-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-cafbd6-8b9445564-f9kww\" (UID: \"e2f5dc4d-52b8-45c8-a048-0c9be26ba9f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cafbd6-8b9445564-f9kww"
Sep 18 14:14:22.585201 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:22.585119    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2f5dc4d-52b8-45c8-a048-0c9be26ba9f2-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-cafbd6-8b9445564-f9kww\" (UID: \"e2f5dc4d-52b8-45c8-a048-0c9be26ba9f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cafbd6-8b9445564-f9kww"
Sep 18 14:14:22.585201 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:22.585178    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2f5dc4d-52b8-45c8-a048-0c9be26ba9f2-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-cafbd6-8b9445564-f9kww\" (UID: \"e2f5dc4d-52b8-45c8-a048-0c9be26ba9f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cafbd6-8b9445564-f9kww"
Sep 18 14:14:22.585382 ip-10-0-141-222 kubenswrapper[2577]: E0918 14:14:22.585227    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cafbd6-serving-cert: secret "maas-api-e2e-ipp-cleanup-cafbd6-serving-cert" not found
Sep 18 14:14:22.585382 ip-10-0-141-222 kubenswrapper[2577]: E0918 14:14:22.585297    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2f5dc4d-52b8-45c8-a048-0c9be26ba9f2-maas-api-tls podName:e2f5dc4d-52b8-45c8-a048-0c9be26ba9f2 nodeName:}" failed. No retries permitted until 2026-09-18 14:14:23.085280018 +0000 UTC m=+1812.137939323 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e2f5dc4d-52b8-45c8-a048-0c9be26ba9f2-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-cafbd6-8b9445564-f9kww" (UID: "e2f5dc4d-52b8-45c8-a048-0c9be26ba9f2") : secret "maas-api-e2e-ipp-cleanup-cafbd6-serving-cert" not found
Sep 18 14:14:22.585985 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:22.585961    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2f5dc4d-52b8-45c8-a048-0c9be26ba9f2-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-cafbd6-8b9445564-f9kww\" (UID: \"e2f5dc4d-52b8-45c8-a048-0c9be26ba9f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cafbd6-8b9445564-f9kww"
Sep 18 14:14:22.587677 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:22.587659    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2f5dc4d-52b8-45c8-a048-0c9be26ba9f2-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-cafbd6-8b9445564-f9kww\" (UID: \"e2f5dc4d-52b8-45c8-a048-0c9be26ba9f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cafbd6-8b9445564-f9kww"
Sep 18 14:14:22.594107 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:22.594086    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrrz2\" (UniqueName: \"kubernetes.io/projected/e2f5dc4d-52b8-45c8-a048-0c9be26ba9f2-kube-api-access-mrrz2\") pod \"maas-api-e2e-ipp-cleanup-cafbd6-8b9445564-f9kww\" (UID: \"e2f5dc4d-52b8-45c8-a048-0c9be26ba9f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cafbd6-8b9445564-f9kww"
Sep 18 14:14:22.893328 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:22.893273    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:22.992444 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:22.992411    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:23.081587 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:23.081535    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:23.089801 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:23.089769    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2f5dc4d-52b8-45c8-a048-0c9be26ba9f2-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-cafbd6-8b9445564-f9kww\" (UID: \"e2f5dc4d-52b8-45c8-a048-0c9be26ba9f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cafbd6-8b9445564-f9kww"
Sep 18 14:14:23.092256 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:23.092230    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2f5dc4d-52b8-45c8-a048-0c9be26ba9f2-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-cafbd6-8b9445564-f9kww\" (UID: \"e2f5dc4d-52b8-45c8-a048-0c9be26ba9f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cafbd6-8b9445564-f9kww"
Sep 18 14:14:23.312852 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:23.312733    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cafbd6-8b9445564-f9kww"
Sep 18 14:14:23.589250 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:23.589224    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cafbd6-8b9445564-f9kww"]
Sep 18 14:14:23.591873 ip-10-0-141-222 kubenswrapper[2577]: W0918 14:14:23.591813    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2f5dc4d_52b8_45c8_a048_0c9be26ba9f2.slice/crio-fabb3b3d89b2971898c231783106268e9df522177eb2747a0c6891d54873b68c WatchSource:0}: Error finding container fabb3b3d89b2971898c231783106268e9df522177eb2747a0c6891d54873b68c: Status 404 returned error can't find the container with id fabb3b3d89b2971898c231783106268e9df522177eb2747a0c6891d54873b68c
Sep 18 14:14:23.593218 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:23.593199    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 14:14:24.153807 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:24.153762    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cafbd6-8b9445564-f9kww" event={"ID":"e2f5dc4d-52b8-45c8-a048-0c9be26ba9f2","Type":"ContainerStarted","Data":"fabb3b3d89b2971898c231783106268e9df522177eb2747a0c6891d54873b68c"}
Sep 18 14:14:25.160943 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:25.160907    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cafbd6-8b9445564-f9kww" event={"ID":"e2f5dc4d-52b8-45c8-a048-0c9be26ba9f2","Type":"ContainerStarted","Data":"4c6d3212f837db8ce9e090ddd59baa583f31fce5e9b437586431d9098f5fdcdf"}
Sep 18 14:14:25.161404 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:25.161039    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cafbd6-8b9445564-f9kww"
Sep 18 14:14:25.181670 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:25.181621    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cafbd6-8b9445564-f9kww" podStartSLOduration=2.350946934 podStartE2EDuration="3.181606501s" totalImagesPullingTime="830.659567ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:14:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:14:25.181007634 +0000 UTC m=+1814.233666961" watchObservedRunningTime="2026-09-18 14:14:25.181606501 +0000 UTC m=+1814.234265837"
Sep 18 14:14:27.294941 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:27.294901    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:27.356871 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:27.356835    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:28.701972 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:28.701938    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 14:14:29.170027 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:29.169991    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:29.271148 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:29.271114    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:31.172131 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:31.172097    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cafbd6-8b9445564-f9kww"
Sep 18 14:14:38.008143 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:38.008103    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:39.227853 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:39.227780    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:44.264600 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:44.264564    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:44.502625 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:44.502582    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:45.437961 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:45.437925    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:46.074513 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:46.074485    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:46.758749 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:46.758687    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:47.715778 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:47.715742    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:48.413496 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:48.413461    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:49.202705 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:49.202672    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:50.714907 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:50.714875    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:50.929005 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:50.928973    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:51.255471 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:51.255439    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cafbd6-8b9445564-f9kww"]
Sep 18 14:14:51.255698 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:51.255668    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cafbd6-8b9445564-f9kww" podUID="e2f5dc4d-52b8-45c8-a048-0c9be26ba9f2" containerName="maas-api" containerID="cri-o://4c6d3212f837db8ce9e090ddd59baa583f31fce5e9b437586431d9098f5fdcdf" gracePeriod=30
Sep 18 14:14:51.282645 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:51.282616    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:51.358932 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:51.358898    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:51.460972 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:51.460917    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:51.507424 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:51.507366    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cafbd6-8b9445564-f9kww"
Sep 18 14:14:51.529565 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:51.529533    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:51.648090 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:51.648057    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2f5dc4d-52b8-45c8-a048-0c9be26ba9f2-openshift-service-ca\") pod \"e2f5dc4d-52b8-45c8-a048-0c9be26ba9f2\" (UID: \"e2f5dc4d-52b8-45c8-a048-0c9be26ba9f2\") "
Sep 18 14:14:51.648090 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:51.648094    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mrrz2\" (UniqueName: \"kubernetes.io/projected/e2f5dc4d-52b8-45c8-a048-0c9be26ba9f2-kube-api-access-mrrz2\") pod \"e2f5dc4d-52b8-45c8-a048-0c9be26ba9f2\" (UID: \"e2f5dc4d-52b8-45c8-a048-0c9be26ba9f2\") "
Sep 18 14:14:51.648309 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:51.648129    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2f5dc4d-52b8-45c8-a048-0c9be26ba9f2-maas-api-tls\") pod \"e2f5dc4d-52b8-45c8-a048-0c9be26ba9f2\" (UID: \"e2f5dc4d-52b8-45c8-a048-0c9be26ba9f2\") "
Sep 18 14:14:51.648309 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:51.648197    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2f5dc4d-52b8-45c8-a048-0c9be26ba9f2-metrics-certs\") pod \"e2f5dc4d-52b8-45c8-a048-0c9be26ba9f2\" (UID: \"e2f5dc4d-52b8-45c8-a048-0c9be26ba9f2\") "
Sep 18 14:14:51.648595 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:51.648571    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2f5dc4d-52b8-45c8-a048-0c9be26ba9f2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e2f5dc4d-52b8-45c8-a048-0c9be26ba9f2" (UID: "e2f5dc4d-52b8-45c8-a048-0c9be26ba9f2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:14:51.650201 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:51.650175    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2f5dc4d-52b8-45c8-a048-0c9be26ba9f2-kube-api-access-mrrz2" (OuterVolumeSpecName: "kube-api-access-mrrz2") pod "e2f5dc4d-52b8-45c8-a048-0c9be26ba9f2" (UID: "e2f5dc4d-52b8-45c8-a048-0c9be26ba9f2"). InnerVolumeSpecName "kube-api-access-mrrz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:14:51.650321 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:51.650286    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2f5dc4d-52b8-45c8-a048-0c9be26ba9f2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e2f5dc4d-52b8-45c8-a048-0c9be26ba9f2" (UID: "e2f5dc4d-52b8-45c8-a048-0c9be26ba9f2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:14:51.650370 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:51.650334    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2f5dc4d-52b8-45c8-a048-0c9be26ba9f2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e2f5dc4d-52b8-45c8-a048-0c9be26ba9f2" (UID: "e2f5dc4d-52b8-45c8-a048-0c9be26ba9f2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:14:51.748981 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:51.748944    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2f5dc4d-52b8-45c8-a048-0c9be26ba9f2-openshift-service-ca\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:14:51.748981 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:51.748975    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrrz2\" (UniqueName: \"kubernetes.io/projected/e2f5dc4d-52b8-45c8-a048-0c9be26ba9f2-kube-api-access-mrrz2\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:14:51.748981 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:51.748985    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2f5dc4d-52b8-45c8-a048-0c9be26ba9f2-maas-api-tls\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:14:51.749409 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:51.748997    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2f5dc4d-52b8-45c8-a048-0c9be26ba9f2-metrics-certs\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:14:51.823082 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:51.823006    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:51.923608 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:51.923577    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:52.317016 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:52.316983    2577 generic.go:356] "Generic (PLEG): container finished" podID="e2f5dc4d-52b8-45c8-a048-0c9be26ba9f2" containerID="4c6d3212f837db8ce9e090ddd59baa583f31fce5e9b437586431d9098f5fdcdf" exitCode=0
Sep 18 14:14:52.317192 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:52.317025    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cafbd6-8b9445564-f9kww"
Sep 18 14:14:52.317192 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:52.317044    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cafbd6-8b9445564-f9kww" event={"ID":"e2f5dc4d-52b8-45c8-a048-0c9be26ba9f2","Type":"ContainerDied","Data":"4c6d3212f837db8ce9e090ddd59baa583f31fce5e9b437586431d9098f5fdcdf"}
Sep 18 14:14:52.317192 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:52.317089    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cafbd6-8b9445564-f9kww" event={"ID":"e2f5dc4d-52b8-45c8-a048-0c9be26ba9f2","Type":"ContainerDied","Data":"fabb3b3d89b2971898c231783106268e9df522177eb2747a0c6891d54873b68c"}
Sep 18 14:14:52.317192 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:52.317111    2577 scope.go:117] "RemoveContainer" containerID="4c6d3212f837db8ce9e090ddd59baa583f31fce5e9b437586431d9098f5fdcdf"
Sep 18 14:14:52.331081 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:52.331061    2577 scope.go:117] "RemoveContainer" containerID="4c6d3212f837db8ce9e090ddd59baa583f31fce5e9b437586431d9098f5fdcdf"
Sep 18 14:14:52.331357 ip-10-0-141-222 kubenswrapper[2577]: E0918 14:14:52.331333    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c6d3212f837db8ce9e090ddd59baa583f31fce5e9b437586431d9098f5fdcdf\": container with ID starting with 4c6d3212f837db8ce9e090ddd59baa583f31fce5e9b437586431d9098f5fdcdf not found: ID does not exist" containerID="4c6d3212f837db8ce9e090ddd59baa583f31fce5e9b437586431d9098f5fdcdf"
Sep 18 14:14:52.331404 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:52.331366    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c6d3212f837db8ce9e090ddd59baa583f31fce5e9b437586431d9098f5fdcdf"} err="failed to get container status \"4c6d3212f837db8ce9e090ddd59baa583f31fce5e9b437586431d9098f5fdcdf\": rpc error: code = NotFound desc = could not find container \"4c6d3212f837db8ce9e090ddd59baa583f31fce5e9b437586431d9098f5fdcdf\": container with ID starting with 4c6d3212f837db8ce9e090ddd59baa583f31fce5e9b437586431d9098f5fdcdf not found: ID does not exist"
Sep 18 14:14:52.344650 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:52.344620    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cafbd6-8b9445564-f9kww"]
Sep 18 14:14:52.353009 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:52.352986    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cafbd6-8b9445564-f9kww"]
Sep 18 14:14:52.731487 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:52.731450    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:53.107362 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:53.107327    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2f5dc4d-52b8-45c8-a048-0c9be26ba9f2" path="/var/lib/kubelet/pods/e2f5dc4d-52b8-45c8-a048-0c9be26ba9f2/volumes"
Sep 18 14:14:53.467691 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:53.467607    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:54.102013 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:54.101973    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:54.907331 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:54.907297    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:55.894274 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:55.894238    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 14:14:58.512713 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:58.512675    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:58.587630 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:58.587594    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:58.788052 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:14:58.787960    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:15:00.154807 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:00.154770    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-9d6301-29829015-ml77r"]
Sep 18 14:15:00.155212 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:00.155145    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2f5dc4d-52b8-45c8-a048-0c9be26ba9f2" containerName="maas-api"
Sep 18 14:15:00.155212 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:00.155159    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2f5dc4d-52b8-45c8-a048-0c9be26ba9f2" containerName="maas-api"
Sep 18 14:15:00.155290 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:00.155232    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2f5dc4d-52b8-45c8-a048-0c9be26ba9f2" containerName="maas-api"
Sep 18 14:15:00.159476 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:00.159440    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b6289c-29829015-glhtd"]
Sep 18 14:15:00.159618 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:00.159581    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-9d6301-29829015-ml77r"
Sep 18 14:15:00.162288 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:00.162265    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cxtb2\""
Sep 18 14:15:00.163304 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:00.163286    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b6289c-29829015-glhtd"
Sep 18 14:15:00.169615 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:00.169585    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-9d6301-29829015-ml77r"]
Sep 18 14:15:00.173185 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:00.173166    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-9d6301-29829015-ml77r"
Sep 18 14:15:00.180102 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:00.180080    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b6289c-29829015-glhtd"
Sep 18 14:15:00.189127 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:00.189057    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b6289c-29829015-glhtd"]
Sep 18 14:15:00.458229 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:00.458146    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-9d6301-29829015-ml77r"]
Sep 18 14:15:00.461260 ip-10-0-141-222 kubenswrapper[2577]: W0918 14:15:00.461228    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0ac76f7_3bab_44c0_99ef_ba6831ff5a79.slice/crio-35c1ffaa48c13e0dfc151e6c04b83a7257b642fd6aebd93929fd646676865b5f WatchSource:0}: Error finding container 35c1ffaa48c13e0dfc151e6c04b83a7257b642fd6aebd93929fd646676865b5f: Status 404 returned error can't find the container with id 35c1ffaa48c13e0dfc151e6c04b83a7257b642fd6aebd93929fd646676865b5f
Sep 18 14:15:00.472466 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:00.472446    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b6289c-29829015-glhtd"]
Sep 18 14:15:00.474939 ip-10-0-141-222 kubenswrapper[2577]: W0918 14:15:00.474912    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dc7713b_8de3_4ee6_9ea5_bc96e410e3ca.slice/crio-6c390fafd0246d0ea87bc32f8e5a1067deaa7eef88800cf45c872e7068d1042d WatchSource:0}: Error finding container 6c390fafd0246d0ea87bc32f8e5a1067deaa7eef88800cf45c872e7068d1042d: Status 404 returned error can't find the container with id 6c390fafd0246d0ea87bc32f8e5a1067deaa7eef88800cf45c872e7068d1042d
Sep 18 14:15:01.368039 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:01.367997    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-9d6301-29829015-ml77r" event={"ID":"a0ac76f7-3bab-44c0-99ef-ba6831ff5a79","Type":"ContainerStarted","Data":"125d78a24fc8b041f12d0b83570c94f4371f829d43c2392d83d982cc2076b990"}
Sep 18 14:15:01.368529 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:01.368068    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-9d6301-29829015-ml77r" event={"ID":"a0ac76f7-3bab-44c0-99ef-ba6831ff5a79","Type":"ContainerStarted","Data":"35c1ffaa48c13e0dfc151e6c04b83a7257b642fd6aebd93929fd646676865b5f"}
Sep 18 14:15:01.369691 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:01.369666    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b6289c-29829015-glhtd" event={"ID":"0dc7713b-8de3-4ee6-9ea5-bc96e410e3ca","Type":"ContainerStarted","Data":"24a32b82b904a2ad4f1faa583302ae5015a9d47c4451a43539bd8a47aca4b024"}
Sep 18 14:15:01.369846 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:01.369714    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b6289c-29829015-glhtd" event={"ID":"0dc7713b-8de3-4ee6-9ea5-bc96e410e3ca","Type":"ContainerStarted","Data":"6c390fafd0246d0ea87bc32f8e5a1067deaa7eef88800cf45c872e7068d1042d"}
Sep 18 14:15:01.383924 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:01.383870    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-9d6301-29829015-ml77r" podStartSLOduration=1.383850219 podStartE2EDuration="1.383850219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:15:01.381953454 +0000 UTC m=+1850.434612783" watchObservedRunningTime="2026-09-18 14:15:01.383850219 +0000 UTC m=+1850.436509545"
Sep 18 14:15:01.397527 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:01.397478    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b6289c-29829015-glhtd" podStartSLOduration=1.397461847 podStartE2EDuration="1.397461847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:15:01.396589418 +0000 UTC m=+1850.449248745" watchObservedRunningTime="2026-09-18 14:15:01.397461847 +0000 UTC m=+1850.450121172"
Sep 18 14:15:02.379744 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:02.379705    2577 generic.go:356] "Generic (PLEG): container finished" podID="a0ac76f7-3bab-44c0-99ef-ba6831ff5a79" containerID="125d78a24fc8b041f12d0b83570c94f4371f829d43c2392d83d982cc2076b990" exitCode=0
Sep 18 14:15:02.380219 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:02.379798    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-9d6301-29829015-ml77r" event={"ID":"a0ac76f7-3bab-44c0-99ef-ba6831ff5a79","Type":"ContainerDied","Data":"125d78a24fc8b041f12d0b83570c94f4371f829d43c2392d83d982cc2076b990"}
Sep 18 14:15:02.381464 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:02.381438    2577 generic.go:356] "Generic (PLEG): container finished" podID="0dc7713b-8de3-4ee6-9ea5-bc96e410e3ca" containerID="24a32b82b904a2ad4f1faa583302ae5015a9d47c4451a43539bd8a47aca4b024" exitCode=0
Sep 18 14:15:02.381593 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:02.381524    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b6289c-29829015-glhtd" event={"ID":"0dc7713b-8de3-4ee6-9ea5-bc96e410e3ca","Type":"ContainerDied","Data":"24a32b82b904a2ad4f1faa583302ae5015a9d47c4451a43539bd8a47aca4b024"}
Sep 18 14:15:02.621774 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:02.621742    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:15:03.516191 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:03.516168    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b6289c-29829015-glhtd"
Sep 18 14:15:03.547093 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:03.547059    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:15:03.551138 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:03.551117    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-9d6301-29829015-ml77r"
Sep 18 14:15:04.322513 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:04.322481    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:15:04.393657 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:04.393626    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-9d6301-29829015-ml77r"
Sep 18 14:15:04.393865 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:04.393665    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-9d6301-29829015-ml77r" event={"ID":"a0ac76f7-3bab-44c0-99ef-ba6831ff5a79","Type":"ContainerDied","Data":"35c1ffaa48c13e0dfc151e6c04b83a7257b642fd6aebd93929fd646676865b5f"}
Sep 18 14:15:04.393865 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:04.393720    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35c1ffaa48c13e0dfc151e6c04b83a7257b642fd6aebd93929fd646676865b5f"
Sep 18 14:15:04.395012 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:04.394986    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b6289c-29829015-glhtd"
Sep 18 14:15:04.395126 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:04.395021    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b6289c-29829015-glhtd" event={"ID":"0dc7713b-8de3-4ee6-9ea5-bc96e410e3ca","Type":"ContainerDied","Data":"6c390fafd0246d0ea87bc32f8e5a1067deaa7eef88800cf45c872e7068d1042d"}
Sep 18 14:15:04.395126 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:04.395064    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c390fafd0246d0ea87bc32f8e5a1067deaa7eef88800cf45c872e7068d1042d"
Sep 18 14:15:05.021369 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:05.021336    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:15:05.695208 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:05.695173    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:15:06.370207 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:06.370172    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:15:15.889125 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:15.889091    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 14:15:17.863301 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:17.863258    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:15:17.863874 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:17.863441    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b6289c-29829015-glhtd"]
Sep 18 14:15:17.865843 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:17.865807    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b6289c-29829015-glhtd"]
Sep 18 14:15:17.964564 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:17.964530    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:15:18.064417 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:18.064381    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:15:18.146162 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:18.146074    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:15:18.460949 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:18.460860    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:15:18.566419 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:18.566385    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:15:19.107461 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:19.107420    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0dc7713b-8de3-4ee6-9ea5-bc96e410e3ca" path="/var/lib/kubelet/pods/0dc7713b-8de3-4ee6-9ea5-bc96e410e3ca/volumes"
Sep 18 14:15:19.404004 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:19.403922    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:15:20.256916 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:20.256880    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:15:21.001208 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:21.001170    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:15:23.438003 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:23.437970    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-b6289c/ipp-route-b6289c-kserve-7749784cff-bkpzl"]
Sep 18 14:15:23.534265 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:23.534230    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:15:29.286756 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:29.286666    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 14:15:38.457537 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:38.457472    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-b6289c/ipp-route-b6289c-kserve-7749784cff-bkpzl" podUID="e9372335-1cf7-4105-9aac-e104f61ca3ac" containerName="main" containerID="cri-o://e10190ec9fe599314a0b0f8fc097a3fa6103e1b1e847f901d0c997a29d4a6476" gracePeriod=45
Sep 18 14:15:38.592097 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:38.592064    2577 generic.go:356] "Generic (PLEG): container finished" podID="e9372335-1cf7-4105-9aac-e104f61ca3ac" containerID="e10190ec9fe599314a0b0f8fc097a3fa6103e1b1e847f901d0c997a29d4a6476" exitCode=0
Sep 18 14:15:38.592249 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:38.592131    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-b6289c/ipp-route-b6289c-kserve-7749784cff-bkpzl" event={"ID":"e9372335-1cf7-4105-9aac-e104f61ca3ac","Type":"ContainerDied","Data":"e10190ec9fe599314a0b0f8fc097a3fa6103e1b1e847f901d0c997a29d4a6476"}
Sep 18 14:15:38.709374 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:38.709316    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-b6289c/ipp-route-b6289c-kserve-7749784cff-bkpzl"
Sep 18 14:15:38.772763 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:38.772731    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9372335-1cf7-4105-9aac-e104f61ca3ac-model-cache\") pod \"e9372335-1cf7-4105-9aac-e104f61ca3ac\" (UID: \"e9372335-1cf7-4105-9aac-e104f61ca3ac\") "
Sep 18 14:15:38.772763 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:38.772771    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9372335-1cf7-4105-9aac-e104f61ca3ac-tls-certs\") pod \"e9372335-1cf7-4105-9aac-e104f61ca3ac\" (UID: \"e9372335-1cf7-4105-9aac-e104f61ca3ac\") "
Sep 18 14:15:38.773025 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:38.772797    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9372335-1cf7-4105-9aac-e104f61ca3ac-home\") pod \"e9372335-1cf7-4105-9aac-e104f61ca3ac\" (UID: \"e9372335-1cf7-4105-9aac-e104f61ca3ac\") "
Sep 18 14:15:38.773025 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:38.772845    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9372335-1cf7-4105-9aac-e104f61ca3ac-dshm\") pod \"e9372335-1cf7-4105-9aac-e104f61ca3ac\" (UID: \"e9372335-1cf7-4105-9aac-e104f61ca3ac\") "
Sep 18 14:15:38.773025 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:38.772872    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lhgpr\" (UniqueName: \"kubernetes.io/projected/e9372335-1cf7-4105-9aac-e104f61ca3ac-kube-api-access-lhgpr\") pod \"e9372335-1cf7-4105-9aac-e104f61ca3ac\" (UID: \"e9372335-1cf7-4105-9aac-e104f61ca3ac\") "
Sep 18 14:15:38.773025 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:38.772931    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9372335-1cf7-4105-9aac-e104f61ca3ac-tmp-dir\") pod \"e9372335-1cf7-4105-9aac-e104f61ca3ac\" (UID: \"e9372335-1cf7-4105-9aac-e104f61ca3ac\") "
Sep 18 14:15:38.773251 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:38.773143    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9372335-1cf7-4105-9aac-e104f61ca3ac-model-cache" (OuterVolumeSpecName: "model-cache") pod "e9372335-1cf7-4105-9aac-e104f61ca3ac" (UID: "e9372335-1cf7-4105-9aac-e104f61ca3ac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:15:38.773251 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:38.773155    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9372335-1cf7-4105-9aac-e104f61ca3ac-home" (OuterVolumeSpecName: "home") pod "e9372335-1cf7-4105-9aac-e104f61ca3ac" (UID: "e9372335-1cf7-4105-9aac-e104f61ca3ac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:15:38.773363 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:38.773263    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9372335-1cf7-4105-9aac-e104f61ca3ac-model-cache\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:15:38.773363 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:38.773280    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9372335-1cf7-4105-9aac-e104f61ca3ac-home\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:15:38.773471 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:38.773355    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9372335-1cf7-4105-9aac-e104f61ca3ac-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e9372335-1cf7-4105-9aac-e104f61ca3ac" (UID: "e9372335-1cf7-4105-9aac-e104f61ca3ac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:15:38.775129 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:38.775097    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9372335-1cf7-4105-9aac-e104f61ca3ac-dshm" (OuterVolumeSpecName: "dshm") pod "e9372335-1cf7-4105-9aac-e104f61ca3ac" (UID: "e9372335-1cf7-4105-9aac-e104f61ca3ac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:15:38.775253 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:38.775145    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9372335-1cf7-4105-9aac-e104f61ca3ac-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e9372335-1cf7-4105-9aac-e104f61ca3ac" (UID: "e9372335-1cf7-4105-9aac-e104f61ca3ac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:15:38.775253 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:38.775154    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9372335-1cf7-4105-9aac-e104f61ca3ac-kube-api-access-lhgpr" (OuterVolumeSpecName: "kube-api-access-lhgpr") pod "e9372335-1cf7-4105-9aac-e104f61ca3ac" (UID: "e9372335-1cf7-4105-9aac-e104f61ca3ac"). InnerVolumeSpecName "kube-api-access-lhgpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:15:38.874671 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:38.874635    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9372335-1cf7-4105-9aac-e104f61ca3ac-tls-certs\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:15:38.874671 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:38.874664    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9372335-1cf7-4105-9aac-e104f61ca3ac-dshm\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:15:38.874671 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:38.874672    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhgpr\" (UniqueName: \"kubernetes.io/projected/e9372335-1cf7-4105-9aac-e104f61ca3ac-kube-api-access-lhgpr\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:15:38.874671 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:38.874682    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9372335-1cf7-4105-9aac-e104f61ca3ac-tmp-dir\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:15:39.598936 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:39.598906    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-b6289c/ipp-route-b6289c-kserve-7749784cff-bkpzl" event={"ID":"e9372335-1cf7-4105-9aac-e104f61ca3ac","Type":"ContainerDied","Data":"26639a1e91b47073107a2b800c7f7a6fb52f3dac17768f5ced40a6d73c56a04c"}
Sep 18 14:15:39.599332 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:39.598958    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-b6289c/ipp-route-b6289c-kserve-7749784cff-bkpzl"
Sep 18 14:15:39.599332 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:39.598954    2577 scope.go:117] "RemoveContainer" containerID="e10190ec9fe599314a0b0f8fc097a3fa6103e1b1e847f901d0c997a29d4a6476"
Sep 18 14:15:39.617338 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:39.617310    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-b6289c/ipp-route-b6289c-kserve-7749784cff-bkpzl"]
Sep 18 14:15:39.620926 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:39.620905    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-b6289c/ipp-route-b6289c-kserve-7749784cff-bkpzl"]
Sep 18 14:15:41.107712 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:15:41.107676    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9372335-1cf7-4105-9aac-e104f61ca3ac" path="/var/lib/kubelet/pods/e9372335-1cf7-4105-9aac-e104f61ca3ac/volumes"
Sep 18 14:16:06.353053 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:06.353018    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:06.729046 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:06.728956    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:12.862624 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:12.862587    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:27.406242 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:27.406204    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:27.486721 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:27.486684    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:27.719654 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:27.719570    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:28.256834 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:28.256794    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:29.047817 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:29.047779    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:29.949880 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:29.949846    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:30.919221 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:30.919187    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:31.695265 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:31.695231    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:38.523810 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:38.523779    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:38.528248 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:38.528219    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-9d6301-29829015-ml77r"]
Sep 18 14:16:38.530910 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:38.530886    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-9d6301-29829015-ml77r"]
Sep 18 14:16:38.566695 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:38.566657    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9d6301-6b548647c4-mc4tj"]
Sep 18 14:16:38.566989 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:38.566958    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d6301-6b548647c4-mc4tj" podUID="3221e20d-d8d1-466b-91e2-6d5eec88d521" containerName="payload-pre-processing" containerID="cri-o://8ab325768820fd1ad7a1fd880c93ca8b15893d96f0f6a13b109097d6a5cdc5bf" gracePeriod=30
Sep 18 14:16:38.624979 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:38.624949    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:38.725389 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:38.725353    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:38.725568 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:38.725456    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d6301-6b548647c4-mc4tj" podUID="3221e20d-d8d1-466b-91e2-6d5eec88d521" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.89:9004: connect: connection refused"
Sep 18 14:16:38.824532 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:38.824501    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:38.829070 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:38.829046    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d6301-6b548647c4-mc4tj"
Sep 18 14:16:38.903307 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:38.903268    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l5xjt\" (UniqueName: \"kubernetes.io/projected/3221e20d-d8d1-466b-91e2-6d5eec88d521-kube-api-access-l5xjt\") pod \"3221e20d-d8d1-466b-91e2-6d5eec88d521\" (UID: \"3221e20d-d8d1-466b-91e2-6d5eec88d521\") "
Sep 18 14:16:38.903507 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:38.903325    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3221e20d-d8d1-466b-91e2-6d5eec88d521-plugins-config-volume\") pod \"3221e20d-d8d1-466b-91e2-6d5eec88d521\" (UID: \"3221e20d-d8d1-466b-91e2-6d5eec88d521\") "
Sep 18 14:16:38.903889 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:38.903837    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3221e20d-d8d1-466b-91e2-6d5eec88d521-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3221e20d-d8d1-466b-91e2-6d5eec88d521" (UID: "3221e20d-d8d1-466b-91e2-6d5eec88d521"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:16:38.905656 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:38.905626    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3221e20d-d8d1-466b-91e2-6d5eec88d521-kube-api-access-l5xjt" (OuterVolumeSpecName: "kube-api-access-l5xjt") pod "3221e20d-d8d1-466b-91e2-6d5eec88d521" (UID: "3221e20d-d8d1-466b-91e2-6d5eec88d521"). InnerVolumeSpecName "kube-api-access-l5xjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:16:38.932431 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:38.932399    2577 generic.go:356] "Generic (PLEG): container finished" podID="3221e20d-d8d1-466b-91e2-6d5eec88d521" containerID="8ab325768820fd1ad7a1fd880c93ca8b15893d96f0f6a13b109097d6a5cdc5bf" exitCode=0
Sep 18 14:16:38.932602 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:38.932447    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d6301-6b548647c4-mc4tj"
Sep 18 14:16:38.932602 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:38.932488    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d6301-6b548647c4-mc4tj" event={"ID":"3221e20d-d8d1-466b-91e2-6d5eec88d521","Type":"ContainerDied","Data":"8ab325768820fd1ad7a1fd880c93ca8b15893d96f0f6a13b109097d6a5cdc5bf"}
Sep 18 14:16:38.932602 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:38.932545    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d6301-6b548647c4-mc4tj" event={"ID":"3221e20d-d8d1-466b-91e2-6d5eec88d521","Type":"ContainerDied","Data":"5f879f63d47e045d19bbeb950d4123f340577bc539ecec7f4ebf49acc6b897a8"}
Sep 18 14:16:38.932602 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:38.932569    2577 scope.go:117] "RemoveContainer" containerID="8ab325768820fd1ad7a1fd880c93ca8b15893d96f0f6a13b109097d6a5cdc5bf"
Sep 18 14:16:38.942535 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:38.942514    2577 scope.go:117] "RemoveContainer" containerID="8ab325768820fd1ad7a1fd880c93ca8b15893d96f0f6a13b109097d6a5cdc5bf"
Sep 18 14:16:38.942817 ip-10-0-141-222 kubenswrapper[2577]: E0918 14:16:38.942796    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ab325768820fd1ad7a1fd880c93ca8b15893d96f0f6a13b109097d6a5cdc5bf\": container with ID starting with 8ab325768820fd1ad7a1fd880c93ca8b15893d96f0f6a13b109097d6a5cdc5bf not found: ID does not exist" containerID="8ab325768820fd1ad7a1fd880c93ca8b15893d96f0f6a13b109097d6a5cdc5bf"
Sep 18 14:16:38.942914 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:38.942839    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ab325768820fd1ad7a1fd880c93ca8b15893d96f0f6a13b109097d6a5cdc5bf"} err="failed to get container status \"8ab325768820fd1ad7a1fd880c93ca8b15893d96f0f6a13b109097d6a5cdc5bf\": rpc error: code = NotFound desc = could not find container \"8ab325768820fd1ad7a1fd880c93ca8b15893d96f0f6a13b109097d6a5cdc5bf\": container with ID starting with 8ab325768820fd1ad7a1fd880c93ca8b15893d96f0f6a13b109097d6a5cdc5bf not found: ID does not exist"
Sep 18 14:16:38.955405 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:38.955375    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9d6301-6b548647c4-mc4tj"]
Sep 18 14:16:38.961336 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:38.961309    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9d6301-6b548647c4-mc4tj"]
Sep 18 14:16:39.004476 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:39.004438    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5xjt\" (UniqueName: \"kubernetes.io/projected/3221e20d-d8d1-466b-91e2-6d5eec88d521-kube-api-access-l5xjt\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:16:39.004476 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:39.004473    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3221e20d-d8d1-466b-91e2-6d5eec88d521-plugins-config-volume\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:16:39.108978 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:39.108947    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3221e20d-d8d1-466b-91e2-6d5eec88d521" path="/var/lib/kubelet/pods/3221e20d-d8d1-466b-91e2-6d5eec88d521/volumes"
Sep 18 14:16:39.109272 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:39.109258    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0ac76f7-3bab-44c0-99ef-ba6831ff5a79" path="/var/lib/kubelet/pods/a0ac76f7-3bab-44c0-99ef-ba6831ff5a79/volumes"
Sep 18 14:16:39.133156 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:39.133121    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:39.196698 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:39.196669    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:39.809100 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:39.809062    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:40.644098 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:40.644061    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:42.289500 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:42.289461    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 14:16:45.388725 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:45.388686    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:45.488314 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:45.488271    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:45.688624 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:45.688545    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82"]
Sep 18 14:16:45.688773 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:45.688745    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82" podUID="373fdc83-6fb7-45b4-aff8-907be2384680" containerName="istio-proxy" containerID="cri-o://644b60cfd8891d2329d4d37740aa9868fc8ccd0bc5458e21d3ae21ffc6b5fb11" gracePeriod=30
Sep 18 14:16:45.731465 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:45.731427    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:46.152885 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:46.152854    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:46.956707 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:46.956673    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:47.735173 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:47.735133    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:48.394630 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:48.394590    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:50.943363 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:50.943338    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82"
Sep 18 14:16:51.009445 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:51.009352    2577 generic.go:356] "Generic (PLEG): container finished" podID="373fdc83-6fb7-45b4-aff8-907be2384680" containerID="644b60cfd8891d2329d4d37740aa9868fc8ccd0bc5458e21d3ae21ffc6b5fb11" exitCode=0
Sep 18 14:16:51.009445 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:51.009402    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82"
Sep 18 14:16:51.009645 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:51.009442    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82" event={"ID":"373fdc83-6fb7-45b4-aff8-907be2384680","Type":"ContainerDied","Data":"644b60cfd8891d2329d4d37740aa9868fc8ccd0bc5458e21d3ae21ffc6b5fb11"}
Sep 18 14:16:51.009645 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:51.009496    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82" event={"ID":"373fdc83-6fb7-45b4-aff8-907be2384680","Type":"ContainerDied","Data":"4329b1882041694cdb3a289121a3fc3b34491a9f2e94a9939145d12b1fe5e7a4"}
Sep 18 14:16:51.009645 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:51.009517    2577 scope.go:117] "RemoveContainer" containerID="644b60cfd8891d2329d4d37740aa9868fc8ccd0bc5458e21d3ae21ffc6b5fb11"
Sep 18 14:16:51.010671 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:51.010562    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/373fdc83-6fb7-45b4-aff8-907be2384680-istio-envoy\") pod \"373fdc83-6fb7-45b4-aff8-907be2384680\" (UID: \"373fdc83-6fb7-45b4-aff8-907be2384680\") "
Sep 18 14:16:51.010671 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:51.010588    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/373fdc83-6fb7-45b4-aff8-907be2384680-istio-podinfo\") pod \"373fdc83-6fb7-45b4-aff8-907be2384680\" (UID: \"373fdc83-6fb7-45b4-aff8-907be2384680\") "
Sep 18 14:16:51.010671 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:51.010633    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/373fdc83-6fb7-45b4-aff8-907be2384680-istio-token\") pod \"373fdc83-6fb7-45b4-aff8-907be2384680\" (UID: \"373fdc83-6fb7-45b4-aff8-907be2384680\") "
Sep 18 14:16:51.010859 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:51.010688    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kbrt4\" (UniqueName: \"kubernetes.io/projected/373fdc83-6fb7-45b4-aff8-907be2384680-kube-api-access-kbrt4\") pod \"373fdc83-6fb7-45b4-aff8-907be2384680\" (UID: \"373fdc83-6fb7-45b4-aff8-907be2384680\") "
Sep 18 14:16:51.010859 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:51.010721    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/373fdc83-6fb7-45b4-aff8-907be2384680-istiod-ca-cert\") pod \"373fdc83-6fb7-45b4-aff8-907be2384680\" (UID: \"373fdc83-6fb7-45b4-aff8-907be2384680\") "
Sep 18 14:16:51.010859 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:51.010747    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/373fdc83-6fb7-45b4-aff8-907be2384680-istio-data\") pod \"373fdc83-6fb7-45b4-aff8-907be2384680\" (UID: \"373fdc83-6fb7-45b4-aff8-907be2384680\") "
Sep 18 14:16:51.010859 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:51.010792    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/373fdc83-6fb7-45b4-aff8-907be2384680-workload-certs\") pod \"373fdc83-6fb7-45b4-aff8-907be2384680\" (UID: \"373fdc83-6fb7-45b4-aff8-907be2384680\") "
Sep 18 14:16:51.010859 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:51.010858    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/373fdc83-6fb7-45b4-aff8-907be2384680-credential-socket\") pod \"373fdc83-6fb7-45b4-aff8-907be2384680\" (UID: \"373fdc83-6fb7-45b4-aff8-907be2384680\") "
Sep 18 14:16:51.011168 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:51.010909    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/373fdc83-6fb7-45b4-aff8-907be2384680-workload-socket\") pod \"373fdc83-6fb7-45b4-aff8-907be2384680\" (UID: \"373fdc83-6fb7-45b4-aff8-907be2384680\") "
Sep 18 14:16:51.011168 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:51.011139    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/373fdc83-6fb7-45b4-aff8-907be2384680-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "373fdc83-6fb7-45b4-aff8-907be2384680" (UID: "373fdc83-6fb7-45b4-aff8-907be2384680"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:16:51.011286 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:51.011241    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/373fdc83-6fb7-45b4-aff8-907be2384680-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "373fdc83-6fb7-45b4-aff8-907be2384680" (UID: "373fdc83-6fb7-45b4-aff8-907be2384680"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:16:51.011286 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:51.011268    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/373fdc83-6fb7-45b4-aff8-907be2384680-workload-certs\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:16:51.011397 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:51.011357    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/373fdc83-6fb7-45b4-aff8-907be2384680-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "373fdc83-6fb7-45b4-aff8-907be2384680" (UID: "373fdc83-6fb7-45b4-aff8-907be2384680"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:16:51.011599 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:51.011574    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/373fdc83-6fb7-45b4-aff8-907be2384680-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "373fdc83-6fb7-45b4-aff8-907be2384680" (UID: "373fdc83-6fb7-45b4-aff8-907be2384680"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:16:51.011712 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:51.011638    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/373fdc83-6fb7-45b4-aff8-907be2384680-istio-data" (OuterVolumeSpecName: "istio-data") pod "373fdc83-6fb7-45b4-aff8-907be2384680" (UID: "373fdc83-6fb7-45b4-aff8-907be2384680"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:16:51.013282 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:51.013239    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/373fdc83-6fb7-45b4-aff8-907be2384680-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "373fdc83-6fb7-45b4-aff8-907be2384680" (UID: "373fdc83-6fb7-45b4-aff8-907be2384680"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 14:16:51.013282 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:51.013265    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/373fdc83-6fb7-45b4-aff8-907be2384680-kube-api-access-kbrt4" (OuterVolumeSpecName: "kube-api-access-kbrt4") pod "373fdc83-6fb7-45b4-aff8-907be2384680" (UID: "373fdc83-6fb7-45b4-aff8-907be2384680"). InnerVolumeSpecName "kube-api-access-kbrt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:16:51.013430 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:51.013328    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/373fdc83-6fb7-45b4-aff8-907be2384680-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "373fdc83-6fb7-45b4-aff8-907be2384680" (UID: "373fdc83-6fb7-45b4-aff8-907be2384680"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:16:51.013430 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:51.013351    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/373fdc83-6fb7-45b4-aff8-907be2384680-istio-token" (OuterVolumeSpecName: "istio-token") pod "373fdc83-6fb7-45b4-aff8-907be2384680" (UID: "373fdc83-6fb7-45b4-aff8-907be2384680"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:16:51.032603 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:51.032581    2577 scope.go:117] "RemoveContainer" containerID="3f777be22400bb11c97961366263faed42b8371620b349ecc2c34bd409eb851b"
Sep 18 14:16:51.041031 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:51.041014    2577 scope.go:117] "RemoveContainer" containerID="644b60cfd8891d2329d4d37740aa9868fc8ccd0bc5458e21d3ae21ffc6b5fb11"
Sep 18 14:16:51.041301 ip-10-0-141-222 kubenswrapper[2577]: E0918 14:16:51.041280    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"644b60cfd8891d2329d4d37740aa9868fc8ccd0bc5458e21d3ae21ffc6b5fb11\": container with ID starting with 644b60cfd8891d2329d4d37740aa9868fc8ccd0bc5458e21d3ae21ffc6b5fb11 not found: ID does not exist" containerID="644b60cfd8891d2329d4d37740aa9868fc8ccd0bc5458e21d3ae21ffc6b5fb11"
Sep 18 14:16:51.041369 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:51.041309    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"644b60cfd8891d2329d4d37740aa9868fc8ccd0bc5458e21d3ae21ffc6b5fb11"} err="failed to get container status \"644b60cfd8891d2329d4d37740aa9868fc8ccd0bc5458e21d3ae21ffc6b5fb11\": rpc error: code = NotFound desc = could not find container \"644b60cfd8891d2329d4d37740aa9868fc8ccd0bc5458e21d3ae21ffc6b5fb11\": container with ID starting with 644b60cfd8891d2329d4d37740aa9868fc8ccd0bc5458e21d3ae21ffc6b5fb11 not found: ID does not exist"
Sep 18 14:16:51.041369 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:51.041327    2577 scope.go:117] "RemoveContainer" containerID="3f777be22400bb11c97961366263faed42b8371620b349ecc2c34bd409eb851b"
Sep 18 14:16:51.041626 ip-10-0-141-222 kubenswrapper[2577]: E0918 14:16:51.041606    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f777be22400bb11c97961366263faed42b8371620b349ecc2c34bd409eb851b\": container with ID starting with 3f777be22400bb11c97961366263faed42b8371620b349ecc2c34bd409eb851b not found: ID does not exist" containerID="3f777be22400bb11c97961366263faed42b8371620b349ecc2c34bd409eb851b"
Sep 18 14:16:51.041678 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:51.041633    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f777be22400bb11c97961366263faed42b8371620b349ecc2c34bd409eb851b"} err="failed to get container status \"3f777be22400bb11c97961366263faed42b8371620b349ecc2c34bd409eb851b\": rpc error: code = NotFound desc = could not find container \"3f777be22400bb11c97961366263faed42b8371620b349ecc2c34bd409eb851b\": container with ID starting with 3f777be22400bb11c97961366263faed42b8371620b349ecc2c34bd409eb851b not found: ID does not exist"
Sep 18 14:16:51.111987 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:51.111951    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/373fdc83-6fb7-45b4-aff8-907be2384680-credential-socket\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:16:51.111987 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:51.111980    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/373fdc83-6fb7-45b4-aff8-907be2384680-workload-socket\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:16:51.111987 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:51.111993    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/373fdc83-6fb7-45b4-aff8-907be2384680-istio-envoy\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:16:51.112239 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:51.112006    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/373fdc83-6fb7-45b4-aff8-907be2384680-istio-podinfo\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:16:51.112239 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:51.112016    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/373fdc83-6fb7-45b4-aff8-907be2384680-istio-token\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:16:51.112239 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:51.112027    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbrt4\" (UniqueName: \"kubernetes.io/projected/373fdc83-6fb7-45b4-aff8-907be2384680-kube-api-access-kbrt4\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:16:51.112239 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:51.112037    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/373fdc83-6fb7-45b4-aff8-907be2384680-istiod-ca-cert\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:16:51.112239 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:51.112048    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/373fdc83-6fb7-45b4-aff8-907be2384680-istio-data\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:16:51.326557 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:51.326475    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82"]
Sep 18 14:16:51.331233 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:51.331203    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-9d6301-openshift-default-76bc4f88b-9bn82"]
Sep 18 14:16:53.110323 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:53.110289    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="373fdc83-6fb7-45b4-aff8-907be2384680" path="/var/lib/kubelet/pods/373fdc83-6fb7-45b4-aff8-907be2384680/volumes"
Sep 18 14:16:56.645618 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:56.645542    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:56.674018 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:56.673985    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-783872-668dc565db-cktw6"]
Sep 18 14:16:56.674288 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:56.674247    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-783872-668dc565db-cktw6" podUID="987fc83f-9e16-4765-894d-66787ce477ef" containerName="payload-pre-processing" containerID="cri-o://1e795a3f996f73e4e2a72773d631e23c43b19ac4db757534d3cca4e34f9bfdb0" gracePeriod=30
Sep 18 14:16:56.717655 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:56.717622    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:56.816965 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:56.816930    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:56.916883 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:56.916853    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:56.921095 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:56.921074    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-783872-668dc565db-cktw6"
Sep 18 14:16:57.044055 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:57.044018    2577 generic.go:356] "Generic (PLEG): container finished" podID="987fc83f-9e16-4765-894d-66787ce477ef" containerID="1e795a3f996f73e4e2a72773d631e23c43b19ac4db757534d3cca4e34f9bfdb0" exitCode=0
Sep 18 14:16:57.044296 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:57.044085    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-783872-668dc565db-cktw6"
Sep 18 14:16:57.044296 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:57.044131    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-783872-668dc565db-cktw6" event={"ID":"987fc83f-9e16-4765-894d-66787ce477ef","Type":"ContainerDied","Data":"1e795a3f996f73e4e2a72773d631e23c43b19ac4db757534d3cca4e34f9bfdb0"}
Sep 18 14:16:57.044296 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:57.044234    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-783872-668dc565db-cktw6" event={"ID":"987fc83f-9e16-4765-894d-66787ce477ef","Type":"ContainerDied","Data":"c07e40130824b51ba3478f5402268de849326e82433556f0a4758a4aa43dc8c1"}
Sep 18 14:16:57.044296 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:57.044273    2577 scope.go:117] "RemoveContainer" containerID="1e795a3f996f73e4e2a72773d631e23c43b19ac4db757534d3cca4e34f9bfdb0"
Sep 18 14:16:57.053141 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:57.053122    2577 scope.go:117] "RemoveContainer" containerID="1e795a3f996f73e4e2a72773d631e23c43b19ac4db757534d3cca4e34f9bfdb0"
Sep 18 14:16:57.053423 ip-10-0-141-222 kubenswrapper[2577]: E0918 14:16:57.053403    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e795a3f996f73e4e2a72773d631e23c43b19ac4db757534d3cca4e34f9bfdb0\": container with ID starting with 1e795a3f996f73e4e2a72773d631e23c43b19ac4db757534d3cca4e34f9bfdb0 not found: ID does not exist" containerID="1e795a3f996f73e4e2a72773d631e23c43b19ac4db757534d3cca4e34f9bfdb0"
Sep 18 14:16:57.053480 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:57.053434    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e795a3f996f73e4e2a72773d631e23c43b19ac4db757534d3cca4e34f9bfdb0"} err="failed to get container status \"1e795a3f996f73e4e2a72773d631e23c43b19ac4db757534d3cca4e34f9bfdb0\": rpc error: code = NotFound desc = could not find container \"1e795a3f996f73e4e2a72773d631e23c43b19ac4db757534d3cca4e34f9bfdb0\": container with ID starting with 1e795a3f996f73e4e2a72773d631e23c43b19ac4db757534d3cca4e34f9bfdb0 not found: ID does not exist"
Sep 18 14:16:57.067753 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:57.067730    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/987fc83f-9e16-4765-894d-66787ce477ef-plugins-config-volume\") pod \"987fc83f-9e16-4765-894d-66787ce477ef\" (UID: \"987fc83f-9e16-4765-894d-66787ce477ef\") "
Sep 18 14:16:57.067857 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:57.067761    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q9kqs\" (UniqueName: \"kubernetes.io/projected/987fc83f-9e16-4765-894d-66787ce477ef-kube-api-access-q9kqs\") pod \"987fc83f-9e16-4765-894d-66787ce477ef\" (UID: \"987fc83f-9e16-4765-894d-66787ce477ef\") "
Sep 18 14:16:57.068108 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:57.068087    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/987fc83f-9e16-4765-894d-66787ce477ef-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "987fc83f-9e16-4765-894d-66787ce477ef" (UID: "987fc83f-9e16-4765-894d-66787ce477ef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:16:57.069783 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:57.069760    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/987fc83f-9e16-4765-894d-66787ce477ef-kube-api-access-q9kqs" (OuterVolumeSpecName: "kube-api-access-q9kqs") pod "987fc83f-9e16-4765-894d-66787ce477ef" (UID: "987fc83f-9e16-4765-894d-66787ce477ef"). InnerVolumeSpecName "kube-api-access-q9kqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:16:57.169278 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:57.169214    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/987fc83f-9e16-4765-894d-66787ce477ef-plugins-config-volume\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:16:57.169278 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:57.169240    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9kqs\" (UniqueName: \"kubernetes.io/projected/987fc83f-9e16-4765-894d-66787ce477ef-kube-api-access-q9kqs\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:16:57.217743 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:57.217711    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:57.318369 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:57.318335    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:57.359840 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:57.359795    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-783872-668dc565db-cktw6"]
Sep 18 14:16:57.365026 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:57.365001    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-783872-668dc565db-cktw6"]
Sep 18 14:16:58.060621 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:58.060586    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:59.107185 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:59.107151    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="987fc83f-9e16-4765-894d-66787ce477ef" path="/var/lib/kubelet/pods/987fc83f-9e16-4765-894d-66787ce477ef/volumes"
Sep 18 14:16:59.893802 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:16:59.893766    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 14:17:03.407280 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:03.407228    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:03.508394 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:03.508360    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:03.676217 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:03.676127    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h"]
Sep 18 14:17:03.676396 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:03.676362    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h" podUID="4e5f9a01-999f-41d8-bb46-422417d62a33" containerName="istio-proxy" containerID="cri-o://11b0dbb74aa212753063903a81f87cbd7157f12e0904e986997a92645ce8d07b" gracePeriod=30
Sep 18 14:17:03.706416 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:03.706375    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:04.661539 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:04.661497    2577 patch_prober.go:28] interesting pod/e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 18 14:17:04.661949 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:04.661557    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h" podUID="4e5f9a01-999f-41d8-bb46-422417d62a33" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 14:17:04.922024 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:04.921929    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:05.568721 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:05.568682    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:08.929802 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:08.929776    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h"
Sep 18 14:17:09.069367 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:09.069255    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e5f9a01-999f-41d8-bb46-422417d62a33-workload-certs\") pod \"4e5f9a01-999f-41d8-bb46-422417d62a33\" (UID: \"4e5f9a01-999f-41d8-bb46-422417d62a33\") "
Sep 18 14:17:09.069367 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:09.069313    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t2szv\" (UniqueName: \"kubernetes.io/projected/4e5f9a01-999f-41d8-bb46-422417d62a33-kube-api-access-t2szv\") pod \"4e5f9a01-999f-41d8-bb46-422417d62a33\" (UID: \"4e5f9a01-999f-41d8-bb46-422417d62a33\") "
Sep 18 14:17:09.069367 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:09.069351    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e5f9a01-999f-41d8-bb46-422417d62a33-istio-data\") pod \"4e5f9a01-999f-41d8-bb46-422417d62a33\" (UID: \"4e5f9a01-999f-41d8-bb46-422417d62a33\") "
Sep 18 14:17:09.069678 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:09.069381    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e5f9a01-999f-41d8-bb46-422417d62a33-workload-socket\") pod \"4e5f9a01-999f-41d8-bb46-422417d62a33\" (UID: \"4e5f9a01-999f-41d8-bb46-422417d62a33\") "
Sep 18 14:17:09.069678 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:09.069408    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e5f9a01-999f-41d8-bb46-422417d62a33-credential-socket\") pod \"4e5f9a01-999f-41d8-bb46-422417d62a33\" (UID: \"4e5f9a01-999f-41d8-bb46-422417d62a33\") "
Sep 18 14:17:09.069678 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:09.069449    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e5f9a01-999f-41d8-bb46-422417d62a33-istio-token\") pod \"4e5f9a01-999f-41d8-bb46-422417d62a33\" (UID: \"4e5f9a01-999f-41d8-bb46-422417d62a33\") "
Sep 18 14:17:09.069678 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:09.069475    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e5f9a01-999f-41d8-bb46-422417d62a33-istio-podinfo\") pod \"4e5f9a01-999f-41d8-bb46-422417d62a33\" (UID: \"4e5f9a01-999f-41d8-bb46-422417d62a33\") "
Sep 18 14:17:09.069678 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:09.069503    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e5f9a01-999f-41d8-bb46-422417d62a33-istiod-ca-cert\") pod \"4e5f9a01-999f-41d8-bb46-422417d62a33\" (UID: \"4e5f9a01-999f-41d8-bb46-422417d62a33\") "
Sep 18 14:17:09.069678 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:09.069568    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e5f9a01-999f-41d8-bb46-422417d62a33-istio-envoy\") pod \"4e5f9a01-999f-41d8-bb46-422417d62a33\" (UID: \"4e5f9a01-999f-41d8-bb46-422417d62a33\") "
Sep 18 14:17:09.069678 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:09.069607    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e5f9a01-999f-41d8-bb46-422417d62a33-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4e5f9a01-999f-41d8-bb46-422417d62a33" (UID: "4e5f9a01-999f-41d8-bb46-422417d62a33"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:17:09.070106 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:09.069745    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e5f9a01-999f-41d8-bb46-422417d62a33-istio-data" (OuterVolumeSpecName: "istio-data") pod "4e5f9a01-999f-41d8-bb46-422417d62a33" (UID: "4e5f9a01-999f-41d8-bb46-422417d62a33"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:17:09.070106 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:09.069794    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e5f9a01-999f-41d8-bb46-422417d62a33-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4e5f9a01-999f-41d8-bb46-422417d62a33" (UID: "4e5f9a01-999f-41d8-bb46-422417d62a33"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:17:09.070106 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:09.069814    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e5f9a01-999f-41d8-bb46-422417d62a33-workload-certs\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:17:09.070106 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:09.069807    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e5f9a01-999f-41d8-bb46-422417d62a33-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4e5f9a01-999f-41d8-bb46-422417d62a33" (UID: "4e5f9a01-999f-41d8-bb46-422417d62a33"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:17:09.070106 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:09.070031    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e5f9a01-999f-41d8-bb46-422417d62a33-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4e5f9a01-999f-41d8-bb46-422417d62a33" (UID: "4e5f9a01-999f-41d8-bb46-422417d62a33"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:17:09.071622 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:09.071589    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e5f9a01-999f-41d8-bb46-422417d62a33-kube-api-access-t2szv" (OuterVolumeSpecName: "kube-api-access-t2szv") pod "4e5f9a01-999f-41d8-bb46-422417d62a33" (UID: "4e5f9a01-999f-41d8-bb46-422417d62a33"). InnerVolumeSpecName "kube-api-access-t2szv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:17:09.071733 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:09.071706    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e5f9a01-999f-41d8-bb46-422417d62a33-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4e5f9a01-999f-41d8-bb46-422417d62a33" (UID: "4e5f9a01-999f-41d8-bb46-422417d62a33"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 14:17:09.072180 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:09.072160    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e5f9a01-999f-41d8-bb46-422417d62a33-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4e5f9a01-999f-41d8-bb46-422417d62a33" (UID: "4e5f9a01-999f-41d8-bb46-422417d62a33"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:17:09.072234 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:09.072166    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e5f9a01-999f-41d8-bb46-422417d62a33-istio-token" (OuterVolumeSpecName: "istio-token") pod "4e5f9a01-999f-41d8-bb46-422417d62a33" (UID: "4e5f9a01-999f-41d8-bb46-422417d62a33"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:17:09.110261 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:09.110230    2577 generic.go:356] "Generic (PLEG): container finished" podID="4e5f9a01-999f-41d8-bb46-422417d62a33" containerID="11b0dbb74aa212753063903a81f87cbd7157f12e0904e986997a92645ce8d07b" exitCode=0
Sep 18 14:17:09.110425 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:09.110283    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h"
Sep 18 14:17:09.110425 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:09.110311    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h" event={"ID":"4e5f9a01-999f-41d8-bb46-422417d62a33","Type":"ContainerDied","Data":"11b0dbb74aa212753063903a81f87cbd7157f12e0904e986997a92645ce8d07b"}
Sep 18 14:17:09.110425 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:09.110361    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h" event={"ID":"4e5f9a01-999f-41d8-bb46-422417d62a33","Type":"ContainerDied","Data":"2ce62b385207fe3ff453ae0484c90ba0d122b9ba9f4763bfe56da9b540a86b61"}
Sep 18 14:17:09.110425 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:09.110385    2577 scope.go:117] "RemoveContainer" containerID="11b0dbb74aa212753063903a81f87cbd7157f12e0904e986997a92645ce8d07b"
Sep 18 14:17:09.120049 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:09.120031    2577 scope.go:117] "RemoveContainer" containerID="2cf8b9527178b27969acd289ec8b7b37e12513119787a02e0cb68db5b8f3952c"
Sep 18 14:17:09.128003 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:09.127985    2577 scope.go:117] "RemoveContainer" containerID="11b0dbb74aa212753063903a81f87cbd7157f12e0904e986997a92645ce8d07b"
Sep 18 14:17:09.128277 ip-10-0-141-222 kubenswrapper[2577]: E0918 14:17:09.128259    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11b0dbb74aa212753063903a81f87cbd7157f12e0904e986997a92645ce8d07b\": container with ID starting with 11b0dbb74aa212753063903a81f87cbd7157f12e0904e986997a92645ce8d07b not found: ID does not exist" containerID="11b0dbb74aa212753063903a81f87cbd7157f12e0904e986997a92645ce8d07b"
Sep 18 14:17:09.128327 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:09.128286    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11b0dbb74aa212753063903a81f87cbd7157f12e0904e986997a92645ce8d07b"} err="failed to get container status \"11b0dbb74aa212753063903a81f87cbd7157f12e0904e986997a92645ce8d07b\": rpc error: code = NotFound desc = could not find container \"11b0dbb74aa212753063903a81f87cbd7157f12e0904e986997a92645ce8d07b\": container with ID starting with 11b0dbb74aa212753063903a81f87cbd7157f12e0904e986997a92645ce8d07b not found: ID does not exist"
Sep 18 14:17:09.128327 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:09.128306    2577 scope.go:117] "RemoveContainer" containerID="2cf8b9527178b27969acd289ec8b7b37e12513119787a02e0cb68db5b8f3952c"
Sep 18 14:17:09.128544 ip-10-0-141-222 kubenswrapper[2577]: E0918 14:17:09.128524    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cf8b9527178b27969acd289ec8b7b37e12513119787a02e0cb68db5b8f3952c\": container with ID starting with 2cf8b9527178b27969acd289ec8b7b37e12513119787a02e0cb68db5b8f3952c not found: ID does not exist" containerID="2cf8b9527178b27969acd289ec8b7b37e12513119787a02e0cb68db5b8f3952c"
Sep 18 14:17:09.128612 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:09.128553    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cf8b9527178b27969acd289ec8b7b37e12513119787a02e0cb68db5b8f3952c"} err="failed to get container status \"2cf8b9527178b27969acd289ec8b7b37e12513119787a02e0cb68db5b8f3952c\": rpc error: code = NotFound desc = could not find container \"2cf8b9527178b27969acd289ec8b7b37e12513119787a02e0cb68db5b8f3952c\": container with ID starting with 2cf8b9527178b27969acd289ec8b7b37e12513119787a02e0cb68db5b8f3952c not found: ID does not exist"
Sep 18 14:17:09.133965 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:09.133941    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h"]
Sep 18 14:17:09.140210 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:09.140184    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-783872-openshift-default-7f455f4dc4-cn85h"]
Sep 18 14:17:09.170508 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:09.170479    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e5f9a01-999f-41d8-bb46-422417d62a33-istio-data\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:17:09.170508 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:09.170504    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e5f9a01-999f-41d8-bb46-422417d62a33-workload-socket\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:17:09.170652 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:09.170514    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e5f9a01-999f-41d8-bb46-422417d62a33-credential-socket\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:17:09.170652 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:09.170524    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e5f9a01-999f-41d8-bb46-422417d62a33-istio-token\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:17:09.170652 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:09.170535    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e5f9a01-999f-41d8-bb46-422417d62a33-istio-podinfo\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:17:09.170652 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:09.170542    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e5f9a01-999f-41d8-bb46-422417d62a33-istiod-ca-cert\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:17:09.170652 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:09.170551    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e5f9a01-999f-41d8-bb46-422417d62a33-istio-envoy\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:17:09.170652 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:09.170559    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2szv\" (UniqueName: \"kubernetes.io/projected/4e5f9a01-999f-41d8-bb46-422417d62a33-kube-api-access-t2szv\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:17:11.107581 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:11.107546    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e5f9a01-999f-41d8-bb46-422417d62a33" path="/var/lib/kubelet/pods/4e5f9a01-999f-41d8-bb46-422417d62a33/volumes"
Sep 18 14:17:14.634693 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:14.634655    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1a4b6d-764599fb9c-xxckl"]
Sep 18 14:17:14.635115 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:14.634935    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1a4b6d-764599fb9c-xxckl" podUID="39003008-ec43-427c-8642-784fcb7ea73b" containerName="maas-api" containerID="cri-o://45776957fa82e091194308f91f1a49b369def9a710682ee711dceda59fba54c9" gracePeriod=30
Sep 18 14:17:14.672524 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:14.672489    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:14.773377 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:14.773344    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:14.870658 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:14.870600    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:14.886558 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:14.886491    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1a4b6d-764599fb9c-xxckl"
Sep 18 14:17:14.917287 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:14.917253    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39003008-ec43-427c-8642-784fcb7ea73b-openshift-service-ca\") pod \"39003008-ec43-427c-8642-784fcb7ea73b\" (UID: \"39003008-ec43-427c-8642-784fcb7ea73b\") "
Sep 18 14:17:14.917287 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:14.917286    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wljnf\" (UniqueName: \"kubernetes.io/projected/39003008-ec43-427c-8642-784fcb7ea73b-kube-api-access-wljnf\") pod \"39003008-ec43-427c-8642-784fcb7ea73b\" (UID: \"39003008-ec43-427c-8642-784fcb7ea73b\") "
Sep 18 14:17:14.917549 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:14.917353    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39003008-ec43-427c-8642-784fcb7ea73b-metrics-certs\") pod \"39003008-ec43-427c-8642-784fcb7ea73b\" (UID: \"39003008-ec43-427c-8642-784fcb7ea73b\") "
Sep 18 14:17:14.917549 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:14.917375    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39003008-ec43-427c-8642-784fcb7ea73b-maas-api-tls\") pod \"39003008-ec43-427c-8642-784fcb7ea73b\" (UID: \"39003008-ec43-427c-8642-784fcb7ea73b\") "
Sep 18 14:17:14.917669 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:14.917643    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39003008-ec43-427c-8642-784fcb7ea73b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "39003008-ec43-427c-8642-784fcb7ea73b" (UID: "39003008-ec43-427c-8642-784fcb7ea73b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:17:14.919552 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:14.919521    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39003008-ec43-427c-8642-784fcb7ea73b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "39003008-ec43-427c-8642-784fcb7ea73b" (UID: "39003008-ec43-427c-8642-784fcb7ea73b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:17:14.919661 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:14.919576    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39003008-ec43-427c-8642-784fcb7ea73b-kube-api-access-wljnf" (OuterVolumeSpecName: "kube-api-access-wljnf") pod "39003008-ec43-427c-8642-784fcb7ea73b" (UID: "39003008-ec43-427c-8642-784fcb7ea73b"). InnerVolumeSpecName "kube-api-access-wljnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:17:14.919661 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:14.919591    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39003008-ec43-427c-8642-784fcb7ea73b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "39003008-ec43-427c-8642-784fcb7ea73b" (UID: "39003008-ec43-427c-8642-784fcb7ea73b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:17:14.973296 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:14.973260    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:15.018268 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:15.018227    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39003008-ec43-427c-8642-784fcb7ea73b-metrics-certs\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:17:15.018268 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:15.018257    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39003008-ec43-427c-8642-784fcb7ea73b-maas-api-tls\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:17:15.018268 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:15.018272    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39003008-ec43-427c-8642-784fcb7ea73b-openshift-service-ca\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:17:15.018515 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:15.018283    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wljnf\" (UniqueName: \"kubernetes.io/projected/39003008-ec43-427c-8642-784fcb7ea73b-kube-api-access-wljnf\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:17:15.143615 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:15.143524    2577 generic.go:356] "Generic (PLEG): container finished" podID="39003008-ec43-427c-8642-784fcb7ea73b" containerID="45776957fa82e091194308f91f1a49b369def9a710682ee711dceda59fba54c9" exitCode=0
Sep 18 14:17:15.143615 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:15.143574    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1a4b6d-764599fb9c-xxckl"
Sep 18 14:17:15.143615 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:15.143604    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1a4b6d-764599fb9c-xxckl" event={"ID":"39003008-ec43-427c-8642-784fcb7ea73b","Type":"ContainerDied","Data":"45776957fa82e091194308f91f1a49b369def9a710682ee711dceda59fba54c9"}
Sep 18 14:17:15.143920 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:15.143655    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1a4b6d-764599fb9c-xxckl" event={"ID":"39003008-ec43-427c-8642-784fcb7ea73b","Type":"ContainerDied","Data":"0a748c68e3707e6b9e7eaa9cdc40f6f9b0f9f0f9a78842a4b286bd19678cbabf"}
Sep 18 14:17:15.143920 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:15.143676    2577 scope.go:117] "RemoveContainer" containerID="45776957fa82e091194308f91f1a49b369def9a710682ee711dceda59fba54c9"
Sep 18 14:17:15.153227 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:15.153205    2577 scope.go:117] "RemoveContainer" containerID="45776957fa82e091194308f91f1a49b369def9a710682ee711dceda59fba54c9"
Sep 18 14:17:15.153513 ip-10-0-141-222 kubenswrapper[2577]: E0918 14:17:15.153497    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45776957fa82e091194308f91f1a49b369def9a710682ee711dceda59fba54c9\": container with ID starting with 45776957fa82e091194308f91f1a49b369def9a710682ee711dceda59fba54c9 not found: ID does not exist" containerID="45776957fa82e091194308f91f1a49b369def9a710682ee711dceda59fba54c9"
Sep 18 14:17:15.153561 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:15.153521    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45776957fa82e091194308f91f1a49b369def9a710682ee711dceda59fba54c9"} err="failed to get container status \"45776957fa82e091194308f91f1a49b369def9a710682ee711dceda59fba54c9\": rpc error: code = NotFound desc = could not find container \"45776957fa82e091194308f91f1a49b369def9a710682ee711dceda59fba54c9\": container with ID starting with 45776957fa82e091194308f91f1a49b369def9a710682ee711dceda59fba54c9 not found: ID does not exist"
Sep 18 14:17:15.159579 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:15.159555    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1a4b6d-764599fb9c-xxckl"]
Sep 18 14:17:15.165188 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:15.165159    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1a4b6d-764599fb9c-xxckl"]
Sep 18 14:17:15.248745 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:15.248709    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:15.349460 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:15.349420    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:17.107457 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:17.107420    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39003008-ec43-427c-8642-784fcb7ea73b" path="/var/lib/kubelet/pods/39003008-ec43-427c-8642-784fcb7ea73b/volumes"
Sep 18 14:17:17.599946 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:17.599915    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 14:17:21.992157 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:21.992125    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:22.073975 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:22.073945    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:22.268051 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:22.267970    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:27.702870 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:27.702836    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 14:17:52.473060 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:52.473021    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:52.531999 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:17:52.531960    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:18:48.604273 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:18:48.604186    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 14:18:49.688259 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:18:49.688226    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:18:51.207652 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:18:51.207622    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:19:13.293308 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:19:13.293278    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-8tnxl_69d3178a-5769-4b5f-a243-30a1bedc5bd6/console-operator/1.log"
Sep 18 14:19:13.301309 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:19:13.301280    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-8tnxl_69d3178a-5769-4b5f-a243-30a1bedc5bd6/console-operator/1.log"
Sep 18 14:19:34.310149 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:19:34.310111    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 14:19:42.487358 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:19:42.487320    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 14:19:47.292626 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:19:47.292588    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:19:50.194096 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:19:50.194057    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 14:20:07.999912 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:20:07.999877    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 14:20:13.999502 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:20:13.999462    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 14:20:47.342145 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:20:47.342110    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:21:13.597548 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:21:13.597515    2577 scope.go:117] "RemoveContainer" containerID="24a32b82b904a2ad4f1faa583302ae5015a9d47c4451a43539bd8a47aca4b024"
Sep 18 14:21:13.606568 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:21:13.606547    2577 scope.go:117] "RemoveContainer" containerID="125d78a24fc8b041f12d0b83570c94f4371f829d43c2392d83d982cc2076b990"
Sep 18 14:21:32.420217 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:21:32.420137    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:21:39.404954 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:21:39.404919    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 14:21:47.800417 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:21:47.800382    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 14:21:48.581965 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:21:48.581928    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:22:02.795307 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:02.795268    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-16aa6981-kserve-7647678447-d89n8"]
Sep 18 14:22:02.795727 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:02.795709    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="373fdc83-6fb7-45b4-aff8-907be2384680" containerName="istio-proxy"
Sep 18 14:22:02.795773 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:02.795730    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="373fdc83-6fb7-45b4-aff8-907be2384680" containerName="istio-proxy"
Sep 18 14:22:02.795773 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:02.795745    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e5f9a01-999f-41d8-bb46-422417d62a33" containerName="istio-proxy"
Sep 18 14:22:02.795773 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:02.795755    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e5f9a01-999f-41d8-bb46-422417d62a33" containerName="istio-proxy"
Sep 18 14:22:02.795773 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:02.795769    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="373fdc83-6fb7-45b4-aff8-907be2384680" containerName="istio-proxy"
Sep 18 14:22:02.795927 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:02.795777    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="373fdc83-6fb7-45b4-aff8-907be2384680" containerName="istio-proxy"
Sep 18 14:22:02.795927 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:02.795798    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3221e20d-d8d1-466b-91e2-6d5eec88d521" containerName="payload-pre-processing"
Sep 18 14:22:02.795927 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:02.795806    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3221e20d-d8d1-466b-91e2-6d5eec88d521" containerName="payload-pre-processing"
Sep 18 14:22:02.795927 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:02.795817    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0ac76f7-3bab-44c0-99ef-ba6831ff5a79" containerName="cleanup"
Sep 18 14:22:02.795927 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:02.795840    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0ac76f7-3bab-44c0-99ef-ba6831ff5a79" containerName="cleanup"
Sep 18 14:22:02.795927 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:02.795851    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="987fc83f-9e16-4765-894d-66787ce477ef" containerName="payload-pre-processing"
Sep 18 14:22:02.795927 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:02.795859    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="987fc83f-9e16-4765-894d-66787ce477ef" containerName="payload-pre-processing"
Sep 18 14:22:02.795927 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:02.795871    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e5f9a01-999f-41d8-bb46-422417d62a33" containerName="istio-proxy"
Sep 18 14:22:02.795927 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:02.795881    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e5f9a01-999f-41d8-bb46-422417d62a33" containerName="istio-proxy"
Sep 18 14:22:02.795927 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:02.795895    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dc7713b-8de3-4ee6-9ea5-bc96e410e3ca" containerName="cleanup"
Sep 18 14:22:02.795927 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:02.795902    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dc7713b-8de3-4ee6-9ea5-bc96e410e3ca" containerName="cleanup"
Sep 18 14:22:02.795927 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:02.795914    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9372335-1cf7-4105-9aac-e104f61ca3ac" containerName="main"
Sep 18 14:22:02.795927 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:02.795922    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9372335-1cf7-4105-9aac-e104f61ca3ac" containerName="main"
Sep 18 14:22:02.796315 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:02.795935    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39003008-ec43-427c-8642-784fcb7ea73b" containerName="maas-api"
Sep 18 14:22:02.796315 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:02.795943    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="39003008-ec43-427c-8642-784fcb7ea73b" containerName="maas-api"
Sep 18 14:22:02.796315 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:02.796043    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="987fc83f-9e16-4765-894d-66787ce477ef" containerName="payload-pre-processing"
Sep 18 14:22:02.796315 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:02.796053    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9372335-1cf7-4105-9aac-e104f61ca3ac" containerName="main"
Sep 18 14:22:02.796315 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:02.796062    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0ac76f7-3bab-44c0-99ef-ba6831ff5a79" containerName="cleanup"
Sep 18 14:22:02.796315 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:02.796069    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="373fdc83-6fb7-45b4-aff8-907be2384680" containerName="istio-proxy"
Sep 18 14:22:02.796315 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:02.796076    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="373fdc83-6fb7-45b4-aff8-907be2384680" containerName="istio-proxy"
Sep 18 14:22:02.796315 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:02.796084    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e5f9a01-999f-41d8-bb46-422417d62a33" containerName="istio-proxy"
Sep 18 14:22:02.796315 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:02.796091    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3221e20d-d8d1-466b-91e2-6d5eec88d521" containerName="payload-pre-processing"
Sep 18 14:22:02.796315 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:02.796099    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="39003008-ec43-427c-8642-784fcb7ea73b" containerName="maas-api"
Sep 18 14:22:02.796315 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:02.796109    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e5f9a01-999f-41d8-bb46-422417d62a33" containerName="istio-proxy"
Sep 18 14:22:02.796315 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:02.796133    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0dc7713b-8de3-4ee6-9ea5-bc96e410e3ca" containerName="cleanup"
Sep 18 14:22:02.799501 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:02.799483    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-16aa6981-kserve-7647678447-d89n8"
Sep 18 14:22:02.801371 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:02.801347    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-16aa6981-kserve-self-signed-certs\""
Sep 18 14:22:02.806023 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:02.805960    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-16aa6981-kserve-7647678447-d89n8"]
Sep 18 14:22:02.901815 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:02.901784    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:22:02.914186 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:02.914160    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d0d928a-1fdc-4753-92f7-45584c86ebce-dshm\") pod \"e2e-same-id-a-16aa6981-kserve-7647678447-d89n8\" (UID: \"0d0d928a-1fdc-4753-92f7-45584c86ebce\") " pod="llm/e2e-same-id-a-16aa6981-kserve-7647678447-d89n8"
Sep 18 14:22:02.914280 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:02.914196    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d0d928a-1fdc-4753-92f7-45584c86ebce-tmp-dir\") pod \"e2e-same-id-a-16aa6981-kserve-7647678447-d89n8\" (UID: \"0d0d928a-1fdc-4753-92f7-45584c86ebce\") " pod="llm/e2e-same-id-a-16aa6981-kserve-7647678447-d89n8"
Sep 18 14:22:02.914280 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:02.914227    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmq5h\" (UniqueName: \"kubernetes.io/projected/0d0d928a-1fdc-4753-92f7-45584c86ebce-kube-api-access-vmq5h\") pod \"e2e-same-id-a-16aa6981-kserve-7647678447-d89n8\" (UID: \"0d0d928a-1fdc-4753-92f7-45584c86ebce\") " pod="llm/e2e-same-id-a-16aa6981-kserve-7647678447-d89n8"
Sep 18 14:22:02.914356 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:02.914281    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d0d928a-1fdc-4753-92f7-45584c86ebce-tls-certs\") pod \"e2e-same-id-a-16aa6981-kserve-7647678447-d89n8\" (UID: \"0d0d928a-1fdc-4753-92f7-45584c86ebce\") " pod="llm/e2e-same-id-a-16aa6981-kserve-7647678447-d89n8"
Sep 18 14:22:02.914356 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:02.914306    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d0d928a-1fdc-4753-92f7-45584c86ebce-home\") pod \"e2e-same-id-a-16aa6981-kserve-7647678447-d89n8\" (UID: \"0d0d928a-1fdc-4753-92f7-45584c86ebce\") " pod="llm/e2e-same-id-a-16aa6981-kserve-7647678447-d89n8"
Sep 18 14:22:02.914356 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:02.914330    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d0d928a-1fdc-4753-92f7-45584c86ebce-model-cache\") pod \"e2e-same-id-a-16aa6981-kserve-7647678447-d89n8\" (UID: \"0d0d928a-1fdc-4753-92f7-45584c86ebce\") " pod="llm/e2e-same-id-a-16aa6981-kserve-7647678447-d89n8"
Sep 18 14:22:03.015592 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:03.015549    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmq5h\" (UniqueName: \"kubernetes.io/projected/0d0d928a-1fdc-4753-92f7-45584c86ebce-kube-api-access-vmq5h\") pod \"e2e-same-id-a-16aa6981-kserve-7647678447-d89n8\" (UID: \"0d0d928a-1fdc-4753-92f7-45584c86ebce\") " pod="llm/e2e-same-id-a-16aa6981-kserve-7647678447-d89n8"
Sep 18 14:22:03.015803 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:03.015611    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d0d928a-1fdc-4753-92f7-45584c86ebce-tls-certs\") pod \"e2e-same-id-a-16aa6981-kserve-7647678447-d89n8\" (UID: \"0d0d928a-1fdc-4753-92f7-45584c86ebce\") " pod="llm/e2e-same-id-a-16aa6981-kserve-7647678447-d89n8"
Sep 18 14:22:03.015803 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:03.015653    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d0d928a-1fdc-4753-92f7-45584c86ebce-home\") pod \"e2e-same-id-a-16aa6981-kserve-7647678447-d89n8\" (UID: \"0d0d928a-1fdc-4753-92f7-45584c86ebce\") " pod="llm/e2e-same-id-a-16aa6981-kserve-7647678447-d89n8"
Sep 18 14:22:03.015803 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:03.015695    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d0d928a-1fdc-4753-92f7-45584c86ebce-model-cache\") pod \"e2e-same-id-a-16aa6981-kserve-7647678447-d89n8\" (UID: \"0d0d928a-1fdc-4753-92f7-45584c86ebce\") " pod="llm/e2e-same-id-a-16aa6981-kserve-7647678447-d89n8"
Sep 18 14:22:03.015803 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:03.015735    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d0d928a-1fdc-4753-92f7-45584c86ebce-dshm\") pod \"e2e-same-id-a-16aa6981-kserve-7647678447-d89n8\" (UID: \"0d0d928a-1fdc-4753-92f7-45584c86ebce\") " pod="llm/e2e-same-id-a-16aa6981-kserve-7647678447-d89n8"
Sep 18 14:22:03.015803 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:03.015768    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d0d928a-1fdc-4753-92f7-45584c86ebce-tmp-dir\") pod \"e2e-same-id-a-16aa6981-kserve-7647678447-d89n8\" (UID: \"0d0d928a-1fdc-4753-92f7-45584c86ebce\") " pod="llm/e2e-same-id-a-16aa6981-kserve-7647678447-d89n8"
Sep 18 14:22:03.016153 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:03.016131    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d0d928a-1fdc-4753-92f7-45584c86ebce-home\") pod \"e2e-same-id-a-16aa6981-kserve-7647678447-d89n8\" (UID: \"0d0d928a-1fdc-4753-92f7-45584c86ebce\") " pod="llm/e2e-same-id-a-16aa6981-kserve-7647678447-d89n8"
Sep 18 14:22:03.016219 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:03.016201    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d0d928a-1fdc-4753-92f7-45584c86ebce-model-cache\") pod \"e2e-same-id-a-16aa6981-kserve-7647678447-d89n8\" (UID: \"0d0d928a-1fdc-4753-92f7-45584c86ebce\") " pod="llm/e2e-same-id-a-16aa6981-kserve-7647678447-d89n8"
Sep 18 14:22:03.016282 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:03.016258    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d0d928a-1fdc-4753-92f7-45584c86ebce-tmp-dir\") pod \"e2e-same-id-a-16aa6981-kserve-7647678447-d89n8\" (UID: \"0d0d928a-1fdc-4753-92f7-45584c86ebce\") " pod="llm/e2e-same-id-a-16aa6981-kserve-7647678447-d89n8"
Sep 18 14:22:03.018108 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:03.018084    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d0d928a-1fdc-4753-92f7-45584c86ebce-dshm\") pod \"e2e-same-id-a-16aa6981-kserve-7647678447-d89n8\" (UID: \"0d0d928a-1fdc-4753-92f7-45584c86ebce\") " pod="llm/e2e-same-id-a-16aa6981-kserve-7647678447-d89n8"
Sep 18 14:22:03.018310 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:03.018293    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d0d928a-1fdc-4753-92f7-45584c86ebce-tls-certs\") pod \"e2e-same-id-a-16aa6981-kserve-7647678447-d89n8\" (UID: \"0d0d928a-1fdc-4753-92f7-45584c86ebce\") " pod="llm/e2e-same-id-a-16aa6981-kserve-7647678447-d89n8"
Sep 18 14:22:03.022920 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:03.022896    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmq5h\" (UniqueName: \"kubernetes.io/projected/0d0d928a-1fdc-4753-92f7-45584c86ebce-kube-api-access-vmq5h\") pod \"e2e-same-id-a-16aa6981-kserve-7647678447-d89n8\" (UID: \"0d0d928a-1fdc-4753-92f7-45584c86ebce\") " pod="llm/e2e-same-id-a-16aa6981-kserve-7647678447-d89n8"
Sep 18 14:22:03.110019 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:03.109980    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-16aa6981-kserve-7647678447-d89n8"
Sep 18 14:22:03.382562 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:03.382536    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-16aa6981-kserve-7647678447-d89n8"]
Sep 18 14:22:03.384902 ip-10-0-141-222 kubenswrapper[2577]: W0918 14:22:03.384870    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d0d928a_1fdc_4753_92f7_45584c86ebce.slice/crio-3185f49fa33ec6d7f5390e3e92a5bfeb992d369d44a05ae6430ab8acffee17f9 WatchSource:0}: Error finding container 3185f49fa33ec6d7f5390e3e92a5bfeb992d369d44a05ae6430ab8acffee17f9: Status 404 returned error can't find the container with id 3185f49fa33ec6d7f5390e3e92a5bfeb992d369d44a05ae6430ab8acffee17f9
Sep 18 14:22:03.597008 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:03.596967    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-16aa6981-kserve-7647678447-d89n8" event={"ID":"0d0d928a-1fdc-4753-92f7-45584c86ebce","Type":"ContainerStarted","Data":"d0f60e23fe3d03c667176723ebad92d19ecd7a94905fd9b1282de6571b7df731"}
Sep 18 14:22:03.597176 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:03.597016    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-16aa6981-kserve-7647678447-d89n8" event={"ID":"0d0d928a-1fdc-4753-92f7-45584c86ebce","Type":"ContainerStarted","Data":"3185f49fa33ec6d7f5390e3e92a5bfeb992d369d44a05ae6430ab8acffee17f9"}
Sep 18 14:22:03.620783 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:03.617090    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-16aa6981-kserve-7647678447-d89n8" podStartSLOduration=1.6170659189999999 podStartE2EDuration="1.617065919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:22:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:22:03.614099182 +0000 UTC m=+2272.666758500" watchObservedRunningTime="2026-09-18 14:22:03.617065919 +0000 UTC m=+2272.669725247"
Sep 18 14:22:05.399187 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:05.399148    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 14:22:09.448582 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:09.448541    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:22:13.110622 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:13.110588    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-16aa6981-kserve-7647678447-d89n8"
Sep 18 14:22:13.110997 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:13.110641    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-16aa6981-kserve-7647678447-d89n8"
Sep 18 14:22:13.122977 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:13.122947    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-16aa6981-kserve-7647678447-d89n8"
Sep 18 14:22:13.659492 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:13.659458    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-16aa6981-kserve-7647678447-d89n8"
Sep 18 14:22:17.991544 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:17.991506    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 14:22:41.220400 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:41.220361    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-16aa6981-kserve-7647678447-d89n8"]
Sep 18 14:22:41.245575 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:41.245510    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:22:43.122116 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:43.122031    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:22:43.594638 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:43.594601    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 14:22:53.091880 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:53.091846    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 14:22:56.239451 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:56.239330    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-16aa6981-kserve-7647678447-d89n8" podUID="0d0d928a-1fdc-4753-92f7-45584c86ebce" containerName="main" containerID="cri-o://d0f60e23fe3d03c667176723ebad92d19ecd7a94905fd9b1282de6571b7df731" gracePeriod=45
Sep 18 14:22:56.255759 ip-10-0-141-222 kubenswrapper[2577]: E0918 14:22:56.255734    2577 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d0d928a_1fdc_4753_92f7_45584c86ebce.slice/crio-d0f60e23fe3d03c667176723ebad92d19ecd7a94905fd9b1282de6571b7df731.scope\": RecentStats: unable to find data in memory cache]"
Sep 18 14:22:56.478712 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:56.478687    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-16aa6981-kserve-7647678447-d89n8"
Sep 18 14:22:56.579129 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:56.579044    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d0d928a-1fdc-4753-92f7-45584c86ebce-dshm\") pod \"0d0d928a-1fdc-4753-92f7-45584c86ebce\" (UID: \"0d0d928a-1fdc-4753-92f7-45584c86ebce\") "
Sep 18 14:22:56.579129 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:56.579085    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d0d928a-1fdc-4753-92f7-45584c86ebce-model-cache\") pod \"0d0d928a-1fdc-4753-92f7-45584c86ebce\" (UID: \"0d0d928a-1fdc-4753-92f7-45584c86ebce\") "
Sep 18 14:22:56.579129 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:56.579134    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d0d928a-1fdc-4753-92f7-45584c86ebce-tls-certs\") pod \"0d0d928a-1fdc-4753-92f7-45584c86ebce\" (UID: \"0d0d928a-1fdc-4753-92f7-45584c86ebce\") "
Sep 18 14:22:56.579436 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:56.579161    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d0d928a-1fdc-4753-92f7-45584c86ebce-tmp-dir\") pod \"0d0d928a-1fdc-4753-92f7-45584c86ebce\" (UID: \"0d0d928a-1fdc-4753-92f7-45584c86ebce\") "
Sep 18 14:22:56.579436 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:56.579180    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d0d928a-1fdc-4753-92f7-45584c86ebce-home\") pod \"0d0d928a-1fdc-4753-92f7-45584c86ebce\" (UID: \"0d0d928a-1fdc-4753-92f7-45584c86ebce\") "
Sep 18 14:22:56.579436 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:56.579203    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vmq5h\" (UniqueName: \"kubernetes.io/projected/0d0d928a-1fdc-4753-92f7-45584c86ebce-kube-api-access-vmq5h\") pod \"0d0d928a-1fdc-4753-92f7-45584c86ebce\" (UID: \"0d0d928a-1fdc-4753-92f7-45584c86ebce\") "
Sep 18 14:22:56.579576 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:56.579445    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d0d928a-1fdc-4753-92f7-45584c86ebce-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0d0d928a-1fdc-4753-92f7-45584c86ebce" (UID: "0d0d928a-1fdc-4753-92f7-45584c86ebce"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:22:56.579576 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:56.579467    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d0d928a-1fdc-4753-92f7-45584c86ebce-home" (OuterVolumeSpecName: "home") pod "0d0d928a-1fdc-4753-92f7-45584c86ebce" (UID: "0d0d928a-1fdc-4753-92f7-45584c86ebce"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:22:56.579576 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:56.579484    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d0d928a-1fdc-4753-92f7-45584c86ebce-model-cache" (OuterVolumeSpecName: "model-cache") pod "0d0d928a-1fdc-4753-92f7-45584c86ebce" (UID: "0d0d928a-1fdc-4753-92f7-45584c86ebce"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:22:56.581433 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:56.581404    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d0d928a-1fdc-4753-92f7-45584c86ebce-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0d0d928a-1fdc-4753-92f7-45584c86ebce" (UID: "0d0d928a-1fdc-4753-92f7-45584c86ebce"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:22:56.581540 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:56.581445    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d0d928a-1fdc-4753-92f7-45584c86ebce-kube-api-access-vmq5h" (OuterVolumeSpecName: "kube-api-access-vmq5h") pod "0d0d928a-1fdc-4753-92f7-45584c86ebce" (UID: "0d0d928a-1fdc-4753-92f7-45584c86ebce"). InnerVolumeSpecName "kube-api-access-vmq5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:22:56.581540 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:56.581450    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d0d928a-1fdc-4753-92f7-45584c86ebce-dshm" (OuterVolumeSpecName: "dshm") pod "0d0d928a-1fdc-4753-92f7-45584c86ebce" (UID: "0d0d928a-1fdc-4753-92f7-45584c86ebce"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:22:56.680691 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:56.680644    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d0d928a-1fdc-4753-92f7-45584c86ebce-tmp-dir\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:22:56.680691 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:56.680679    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d0d928a-1fdc-4753-92f7-45584c86ebce-home\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:22:56.680691 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:56.680687    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmq5h\" (UniqueName: \"kubernetes.io/projected/0d0d928a-1fdc-4753-92f7-45584c86ebce-kube-api-access-vmq5h\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:22:56.680691 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:56.680695    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d0d928a-1fdc-4753-92f7-45584c86ebce-dshm\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:22:56.680691 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:56.680704    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d0d928a-1fdc-4753-92f7-45584c86ebce-model-cache\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:22:56.681136 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:56.680711    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d0d928a-1fdc-4753-92f7-45584c86ebce-tls-certs\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:22:56.873407 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:56.873369    2577 generic.go:356] "Generic (PLEG): container finished" podID="0d0d928a-1fdc-4753-92f7-45584c86ebce" containerID="d0f60e23fe3d03c667176723ebad92d19ecd7a94905fd9b1282de6571b7df731" exitCode=0
Sep 18 14:22:56.873576 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:56.873417    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-16aa6981-kserve-7647678447-d89n8"
Sep 18 14:22:56.873576 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:56.873458    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-16aa6981-kserve-7647678447-d89n8" event={"ID":"0d0d928a-1fdc-4753-92f7-45584c86ebce","Type":"ContainerDied","Data":"d0f60e23fe3d03c667176723ebad92d19ecd7a94905fd9b1282de6571b7df731"}
Sep 18 14:22:56.873576 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:56.873503    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-16aa6981-kserve-7647678447-d89n8" event={"ID":"0d0d928a-1fdc-4753-92f7-45584c86ebce","Type":"ContainerDied","Data":"3185f49fa33ec6d7f5390e3e92a5bfeb992d369d44a05ae6430ab8acffee17f9"}
Sep 18 14:22:56.873576 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:56.873523    2577 scope.go:117] "RemoveContainer" containerID="d0f60e23fe3d03c667176723ebad92d19ecd7a94905fd9b1282de6571b7df731"
Sep 18 14:22:56.882516 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:56.882497    2577 scope.go:117] "RemoveContainer" containerID="d0f60e23fe3d03c667176723ebad92d19ecd7a94905fd9b1282de6571b7df731"
Sep 18 14:22:56.882816 ip-10-0-141-222 kubenswrapper[2577]: E0918 14:22:56.882797    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0f60e23fe3d03c667176723ebad92d19ecd7a94905fd9b1282de6571b7df731\": container with ID starting with d0f60e23fe3d03c667176723ebad92d19ecd7a94905fd9b1282de6571b7df731 not found: ID does not exist" containerID="d0f60e23fe3d03c667176723ebad92d19ecd7a94905fd9b1282de6571b7df731"
Sep 18 14:22:56.882892 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:56.882845    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0f60e23fe3d03c667176723ebad92d19ecd7a94905fd9b1282de6571b7df731"} err="failed to get container status \"d0f60e23fe3d03c667176723ebad92d19ecd7a94905fd9b1282de6571b7df731\": rpc error: code = NotFound desc = could not find container \"d0f60e23fe3d03c667176723ebad92d19ecd7a94905fd9b1282de6571b7df731\": container with ID starting with d0f60e23fe3d03c667176723ebad92d19ecd7a94905fd9b1282de6571b7df731 not found: ID does not exist"
Sep 18 14:22:56.898772 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:56.898746    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-16aa6981-kserve-7647678447-d89n8"]
Sep 18 14:22:56.903233 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:56.903213    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-16aa6981-kserve-7647678447-d89n8"]
Sep 18 14:22:57.107608 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:22:57.107571    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d0d928a-1fdc-4753-92f7-45584c86ebce" path="/var/lib/kubelet/pods/0d0d928a-1fdc-4753-92f7-45584c86ebce/volumes"
Sep 18 14:23:01.216802 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:01.216764    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-859bfcbfc5-7swqf"]
Sep 18 14:23:01.217213 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:01.217119    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d0d928a-1fdc-4753-92f7-45584c86ebce" containerName="main"
Sep 18 14:23:01.217213 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:01.217130    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d0d928a-1fdc-4753-92f7-45584c86ebce" containerName="main"
Sep 18 14:23:01.217213 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:01.217193    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d0d928a-1fdc-4753-92f7-45584c86ebce" containerName="main"
Sep 18 14:23:01.221394 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:01.221375    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-859bfcbfc5-7swqf"
Sep 18 14:23:01.224438 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:01.224417    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vx9xd\""
Sep 18 14:23:01.224545 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:01.224460    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 18 14:23:01.224712 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:01.224695    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 14:23:01.224847 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:01.224740    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 18 14:23:01.224923 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:01.224849    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 18 14:23:01.231375 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:01.231354    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-859bfcbfc5-7swqf"]
Sep 18 14:23:01.299797 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:01.299763    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 14:23:01.321513 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:01.321480    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18b8985c-ea40-46f8-bf31-2bb78b6fe088-metrics-certs\") pod \"maas-api-859bfcbfc5-7swqf\" (UID: \"18b8985c-ea40-46f8-bf31-2bb78b6fe088\") " pod="odh-ai-gateway-infra/maas-api-859bfcbfc5-7swqf"
Sep 18 14:23:01.321645 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:01.321522    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18b8985c-ea40-46f8-bf31-2bb78b6fe088-maas-api-tls\") pod \"maas-api-859bfcbfc5-7swqf\" (UID: \"18b8985c-ea40-46f8-bf31-2bb78b6fe088\") " pod="odh-ai-gateway-infra/maas-api-859bfcbfc5-7swqf"
Sep 18 14:23:01.321645 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:01.321573    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18b8985c-ea40-46f8-bf31-2bb78b6fe088-openshift-service-ca\") pod \"maas-api-859bfcbfc5-7swqf\" (UID: \"18b8985c-ea40-46f8-bf31-2bb78b6fe088\") " pod="odh-ai-gateway-infra/maas-api-859bfcbfc5-7swqf"
Sep 18 14:23:01.321645 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:01.321634    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcfn8\" (UniqueName: \"kubernetes.io/projected/18b8985c-ea40-46f8-bf31-2bb78b6fe088-kube-api-access-gcfn8\") pod \"maas-api-859bfcbfc5-7swqf\" (UID: \"18b8985c-ea40-46f8-bf31-2bb78b6fe088\") " pod="odh-ai-gateway-infra/maas-api-859bfcbfc5-7swqf"
Sep 18 14:23:01.422389 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:01.422351    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18b8985c-ea40-46f8-bf31-2bb78b6fe088-openshift-service-ca\") pod \"maas-api-859bfcbfc5-7swqf\" (UID: \"18b8985c-ea40-46f8-bf31-2bb78b6fe088\") " pod="odh-ai-gateway-infra/maas-api-859bfcbfc5-7swqf"
Sep 18 14:23:01.422583 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:01.422409    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcfn8\" (UniqueName: \"kubernetes.io/projected/18b8985c-ea40-46f8-bf31-2bb78b6fe088-kube-api-access-gcfn8\") pod \"maas-api-859bfcbfc5-7swqf\" (UID: \"18b8985c-ea40-46f8-bf31-2bb78b6fe088\") " pod="odh-ai-gateway-infra/maas-api-859bfcbfc5-7swqf"
Sep 18 14:23:01.422583 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:01.422470    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18b8985c-ea40-46f8-bf31-2bb78b6fe088-metrics-certs\") pod \"maas-api-859bfcbfc5-7swqf\" (UID: \"18b8985c-ea40-46f8-bf31-2bb78b6fe088\") " pod="odh-ai-gateway-infra/maas-api-859bfcbfc5-7swqf"
Sep 18 14:23:01.422583 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:01.422514    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18b8985c-ea40-46f8-bf31-2bb78b6fe088-maas-api-tls\") pod \"maas-api-859bfcbfc5-7swqf\" (UID: \"18b8985c-ea40-46f8-bf31-2bb78b6fe088\") " pod="odh-ai-gateway-infra/maas-api-859bfcbfc5-7swqf"
Sep 18 14:23:01.423232 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:01.423204    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18b8985c-ea40-46f8-bf31-2bb78b6fe088-openshift-service-ca\") pod \"maas-api-859bfcbfc5-7swqf\" (UID: \"18b8985c-ea40-46f8-bf31-2bb78b6fe088\") " pod="odh-ai-gateway-infra/maas-api-859bfcbfc5-7swqf"
Sep 18 14:23:01.425048 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:01.425027    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18b8985c-ea40-46f8-bf31-2bb78b6fe088-maas-api-tls\") pod \"maas-api-859bfcbfc5-7swqf\" (UID: \"18b8985c-ea40-46f8-bf31-2bb78b6fe088\") " pod="odh-ai-gateway-infra/maas-api-859bfcbfc5-7swqf"
Sep 18 14:23:01.425153 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:01.425064    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18b8985c-ea40-46f8-bf31-2bb78b6fe088-metrics-certs\") pod \"maas-api-859bfcbfc5-7swqf\" (UID: \"18b8985c-ea40-46f8-bf31-2bb78b6fe088\") " pod="odh-ai-gateway-infra/maas-api-859bfcbfc5-7swqf"
Sep 18 14:23:01.430324 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:01.430303    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcfn8\" (UniqueName: \"kubernetes.io/projected/18b8985c-ea40-46f8-bf31-2bb78b6fe088-kube-api-access-gcfn8\") pod \"maas-api-859bfcbfc5-7swqf\" (UID: \"18b8985c-ea40-46f8-bf31-2bb78b6fe088\") " pod="odh-ai-gateway-infra/maas-api-859bfcbfc5-7swqf"
Sep 18 14:23:01.532898 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:01.532799    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-859bfcbfc5-7swqf"
Sep 18 14:23:01.810366 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:01.810337    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-859bfcbfc5-7swqf"]
Sep 18 14:23:01.811844 ip-10-0-141-222 kubenswrapper[2577]: W0918 14:23:01.811799    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18b8985c_ea40_46f8_bf31_2bb78b6fe088.slice/crio-5759bd96f7d7761782b89c5b963c087748d7b86d4e0496c719caae06102bc34a WatchSource:0}: Error finding container 5759bd96f7d7761782b89c5b963c087748d7b86d4e0496c719caae06102bc34a: Status 404 returned error can't find the container with id 5759bd96f7d7761782b89c5b963c087748d7b86d4e0496c719caae06102bc34a
Sep 18 14:23:01.813264 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:01.813242    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 14:23:01.900169 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:01.900136    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-859bfcbfc5-7swqf" event={"ID":"18b8985c-ea40-46f8-bf31-2bb78b6fe088","Type":"ContainerStarted","Data":"5759bd96f7d7761782b89c5b963c087748d7b86d4e0496c719caae06102bc34a"}
Sep 18 14:23:02.906516 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:02.906478    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-859bfcbfc5-7swqf" event={"ID":"18b8985c-ea40-46f8-bf31-2bb78b6fe088","Type":"ContainerStarted","Data":"b33d977e3a2b6954435484d74489283be054e5d66bbcfb98dabdc8e808ec4c49"}
Sep 18 14:23:02.906944 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:02.906539    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-859bfcbfc5-7swqf"
Sep 18 14:23:02.924837 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:02.924782    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-859bfcbfc5-7swqf" podStartSLOduration=1.423973643 podStartE2EDuration="1.92476601s" totalImagesPullingTime="500.792367ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:23:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:23:02.922973017 +0000 UTC m=+2331.975632344" watchObservedRunningTime="2026-09-18 14:23:02.92476601 +0000 UTC m=+2331.977425338"
Sep 18 14:23:08.698288 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:08.698241    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 14:23:08.713446 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:08.713418    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-859bfcbfc5-7swqf"]
Sep 18 14:23:08.713600 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:08.713579    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-859bfcbfc5-7swqf" podUID="18b8985c-ea40-46f8-bf31-2bb78b6fe088" containerName="maas-api" containerID="cri-o://b33d977e3a2b6954435484d74489283be054e5d66bbcfb98dabdc8e808ec4c49" gracePeriod=30
Sep 18 14:23:08.718838 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:08.718800    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-859bfcbfc5-7swqf"
Sep 18 14:23:08.747487 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:08.747456    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-78fd7758f9-9lnx7"]
Sep 18 14:23:08.750572 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:08.750555    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-78fd7758f9-9lnx7"
Sep 18 14:23:08.763534 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:08.763509    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-78fd7758f9-9lnx7"]
Sep 18 14:23:08.889503 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:08.889359    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h5k7\" (UniqueName: \"kubernetes.io/projected/60c1c143-edf0-47e2-9d92-524d813ce34a-kube-api-access-7h5k7\") pod \"maas-api-78fd7758f9-9lnx7\" (UID: \"60c1c143-edf0-47e2-9d92-524d813ce34a\") " pod="odh-ai-gateway-infra/maas-api-78fd7758f9-9lnx7"
Sep 18 14:23:08.889503 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:08.889416    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60c1c143-edf0-47e2-9d92-524d813ce34a-openshift-service-ca\") pod \"maas-api-78fd7758f9-9lnx7\" (UID: \"60c1c143-edf0-47e2-9d92-524d813ce34a\") " pod="odh-ai-gateway-infra/maas-api-78fd7758f9-9lnx7"
Sep 18 14:23:08.889503 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:08.889478    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60c1c143-edf0-47e2-9d92-524d813ce34a-maas-api-tls\") pod \"maas-api-78fd7758f9-9lnx7\" (UID: \"60c1c143-edf0-47e2-9d92-524d813ce34a\") " pod="odh-ai-gateway-infra/maas-api-78fd7758f9-9lnx7"
Sep 18 14:23:08.889782 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:08.889546    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60c1c143-edf0-47e2-9d92-524d813ce34a-metrics-certs\") pod \"maas-api-78fd7758f9-9lnx7\" (UID: \"60c1c143-edf0-47e2-9d92-524d813ce34a\") " pod="odh-ai-gateway-infra/maas-api-78fd7758f9-9lnx7"
Sep 18 14:23:08.939025 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:08.938998    2577 generic.go:356] "Generic (PLEG): container finished" podID="18b8985c-ea40-46f8-bf31-2bb78b6fe088" containerID="b33d977e3a2b6954435484d74489283be054e5d66bbcfb98dabdc8e808ec4c49" exitCode=0
Sep 18 14:23:08.939160 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:08.939042    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-859bfcbfc5-7swqf" event={"ID":"18b8985c-ea40-46f8-bf31-2bb78b6fe088","Type":"ContainerDied","Data":"b33d977e3a2b6954435484d74489283be054e5d66bbcfb98dabdc8e808ec4c49"}
Sep 18 14:23:08.957653 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:08.957597    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-859bfcbfc5-7swqf"
Sep 18 14:23:08.990596 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:08.990564    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h5k7\" (UniqueName: \"kubernetes.io/projected/60c1c143-edf0-47e2-9d92-524d813ce34a-kube-api-access-7h5k7\") pod \"maas-api-78fd7758f9-9lnx7\" (UID: \"60c1c143-edf0-47e2-9d92-524d813ce34a\") " pod="odh-ai-gateway-infra/maas-api-78fd7758f9-9lnx7"
Sep 18 14:23:08.990596 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:08.990602    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60c1c143-edf0-47e2-9d92-524d813ce34a-openshift-service-ca\") pod \"maas-api-78fd7758f9-9lnx7\" (UID: \"60c1c143-edf0-47e2-9d92-524d813ce34a\") " pod="odh-ai-gateway-infra/maas-api-78fd7758f9-9lnx7"
Sep 18 14:23:08.990811 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:08.990737    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60c1c143-edf0-47e2-9d92-524d813ce34a-maas-api-tls\") pod \"maas-api-78fd7758f9-9lnx7\" (UID: \"60c1c143-edf0-47e2-9d92-524d813ce34a\") " pod="odh-ai-gateway-infra/maas-api-78fd7758f9-9lnx7"
Sep 18 14:23:08.990879 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:08.990862    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60c1c143-edf0-47e2-9d92-524d813ce34a-metrics-certs\") pod \"maas-api-78fd7758f9-9lnx7\" (UID: \"60c1c143-edf0-47e2-9d92-524d813ce34a\") " pod="odh-ai-gateway-infra/maas-api-78fd7758f9-9lnx7"
Sep 18 14:23:08.991250 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:08.991224    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60c1c143-edf0-47e2-9d92-524d813ce34a-openshift-service-ca\") pod \"maas-api-78fd7758f9-9lnx7\" (UID: \"60c1c143-edf0-47e2-9d92-524d813ce34a\") " pod="odh-ai-gateway-infra/maas-api-78fd7758f9-9lnx7"
Sep 18 14:23:08.993277 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:08.993259    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60c1c143-edf0-47e2-9d92-524d813ce34a-metrics-certs\") pod \"maas-api-78fd7758f9-9lnx7\" (UID: \"60c1c143-edf0-47e2-9d92-524d813ce34a\") " pod="odh-ai-gateway-infra/maas-api-78fd7758f9-9lnx7"
Sep 18 14:23:08.993342 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:08.993316    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60c1c143-edf0-47e2-9d92-524d813ce34a-maas-api-tls\") pod \"maas-api-78fd7758f9-9lnx7\" (UID: \"60c1c143-edf0-47e2-9d92-524d813ce34a\") " pod="odh-ai-gateway-infra/maas-api-78fd7758f9-9lnx7"
Sep 18 14:23:08.999362 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:08.999343    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h5k7\" (UniqueName: \"kubernetes.io/projected/60c1c143-edf0-47e2-9d92-524d813ce34a-kube-api-access-7h5k7\") pod \"maas-api-78fd7758f9-9lnx7\" (UID: \"60c1c143-edf0-47e2-9d92-524d813ce34a\") " pod="odh-ai-gateway-infra/maas-api-78fd7758f9-9lnx7"
Sep 18 14:23:09.092208 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:09.092170    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gcfn8\" (UniqueName: \"kubernetes.io/projected/18b8985c-ea40-46f8-bf31-2bb78b6fe088-kube-api-access-gcfn8\") pod \"18b8985c-ea40-46f8-bf31-2bb78b6fe088\" (UID: \"18b8985c-ea40-46f8-bf31-2bb78b6fe088\") "
Sep 18 14:23:09.092395 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:09.092245    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18b8985c-ea40-46f8-bf31-2bb78b6fe088-metrics-certs\") pod \"18b8985c-ea40-46f8-bf31-2bb78b6fe088\" (UID: \"18b8985c-ea40-46f8-bf31-2bb78b6fe088\") "
Sep 18 14:23:09.092395 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:09.092329    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18b8985c-ea40-46f8-bf31-2bb78b6fe088-openshift-service-ca\") pod \"18b8985c-ea40-46f8-bf31-2bb78b6fe088\" (UID: \"18b8985c-ea40-46f8-bf31-2bb78b6fe088\") "
Sep 18 14:23:09.092395 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:09.092371    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18b8985c-ea40-46f8-bf31-2bb78b6fe088-maas-api-tls\") pod \"18b8985c-ea40-46f8-bf31-2bb78b6fe088\" (UID: \"18b8985c-ea40-46f8-bf31-2bb78b6fe088\") "
Sep 18 14:23:09.092672 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:09.092646    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18b8985c-ea40-46f8-bf31-2bb78b6fe088-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "18b8985c-ea40-46f8-bf31-2bb78b6fe088" (UID: "18b8985c-ea40-46f8-bf31-2bb78b6fe088"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:23:09.094421 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:09.094402    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18b8985c-ea40-46f8-bf31-2bb78b6fe088-kube-api-access-gcfn8" (OuterVolumeSpecName: "kube-api-access-gcfn8") pod "18b8985c-ea40-46f8-bf31-2bb78b6fe088" (UID: "18b8985c-ea40-46f8-bf31-2bb78b6fe088"). InnerVolumeSpecName "kube-api-access-gcfn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:23:09.094421 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:09.094414    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18b8985c-ea40-46f8-bf31-2bb78b6fe088-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "18b8985c-ea40-46f8-bf31-2bb78b6fe088" (UID: "18b8985c-ea40-46f8-bf31-2bb78b6fe088"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:23:09.094531 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:09.094428    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18b8985c-ea40-46f8-bf31-2bb78b6fe088-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "18b8985c-ea40-46f8-bf31-2bb78b6fe088" (UID: "18b8985c-ea40-46f8-bf31-2bb78b6fe088"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:23:09.094571 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:09.094531    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-78fd7758f9-9lnx7"
Sep 18 14:23:09.193389 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:09.193348    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18b8985c-ea40-46f8-bf31-2bb78b6fe088-metrics-certs\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:23:09.193389 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:09.193384    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18b8985c-ea40-46f8-bf31-2bb78b6fe088-openshift-service-ca\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:23:09.193589 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:09.193400    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18b8985c-ea40-46f8-bf31-2bb78b6fe088-maas-api-tls\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:23:09.193589 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:09.193413    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcfn8\" (UniqueName: \"kubernetes.io/projected/18b8985c-ea40-46f8-bf31-2bb78b6fe088-kube-api-access-gcfn8\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
Sep 18 14:23:09.372872 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:09.372848    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-78fd7758f9-9lnx7"]
Sep 18 14:23:09.374906 ip-10-0-141-222 kubenswrapper[2577]: W0918 14:23:09.374878    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60c1c143_edf0_47e2_9d92_524d813ce34a.slice/crio-862a36a047fb3f244ecffa3ed8af943ce1b15c9d1972a5c4aa85e36f6c6434c3 WatchSource:0}: Error finding container 862a36a047fb3f244ecffa3ed8af943ce1b15c9d1972a5c4aa85e36f6c6434c3: Status 404 returned error can't find the container with id 862a36a047fb3f244ecffa3ed8af943ce1b15c9d1972a5c4aa85e36f6c6434c3
Sep 18 14:23:09.952207 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:09.952102    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-78fd7758f9-9lnx7" event={"ID":"60c1c143-edf0-47e2-9d92-524d813ce34a","Type":"ContainerStarted","Data":"8017ef03ceb4b717b40847c9bd35f77306474de8d739a8acad9507d4080cd5b7"}
Sep 18 14:23:09.952207 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:09.952202    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-78fd7758f9-9lnx7" event={"ID":"60c1c143-edf0-47e2-9d92-524d813ce34a","Type":"ContainerStarted","Data":"862a36a047fb3f244ecffa3ed8af943ce1b15c9d1972a5c4aa85e36f6c6434c3"}
Sep 18 14:23:09.952670 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:09.952320    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-78fd7758f9-9lnx7"
Sep 18 14:23:09.953614 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:09.953592    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-859bfcbfc5-7swqf"
Sep 18 14:23:09.953614 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:09.953599    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-859bfcbfc5-7swqf" event={"ID":"18b8985c-ea40-46f8-bf31-2bb78b6fe088","Type":"ContainerDied","Data":"5759bd96f7d7761782b89c5b963c087748d7b86d4e0496c719caae06102bc34a"}
Sep 18 14:23:09.953788 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:09.953650    2577 scope.go:117] "RemoveContainer" containerID="b33d977e3a2b6954435484d74489283be054e5d66bbcfb98dabdc8e808ec4c49"
Sep 18 14:23:09.969500 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:09.969460    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-78fd7758f9-9lnx7" podStartSLOduration=1.647999234 podStartE2EDuration="1.969446625s" totalImagesPullingTime="321.447391ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:23:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:23:09.966880944 +0000 UTC m=+2339.019540270" watchObservedRunningTime="2026-09-18 14:23:09.969446625 +0000 UTC m=+2339.022105956"
Sep 18 14:23:09.981495 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:09.981470    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-859bfcbfc5-7swqf"]
Sep 18 14:23:09.985296 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:09.985269    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-859bfcbfc5-7swqf"]
Sep 18 14:23:11.107158 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:11.107124    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18b8985c-ea40-46f8-bf31-2bb78b6fe088" path="/var/lib/kubelet/pods/18b8985c-ea40-46f8-bf31-2bb78b6fe088/volumes"
Sep 18 14:23:15.797792 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:15.797755    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 14:23:15.965377 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:15.965346    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-78fd7758f9-9lnx7"
Sep 18 14:23:16.491108 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:16.491075    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:24.091928 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:24.091892    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 14:23:29.590220 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:29.590179    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ljk2l"]
Sep 18 14:23:32.021992 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:32.021954    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6cd9cc5c75-j2z8r_4f42e29e-9888-4eab-847f-159c841c49ed/authorino/0.log"
Sep 18 14:23:38.070605 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:38.070576    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-978ccd68d-2l9f7_278aa551-51a0-4564-af58-0d20184ab87a/copy-manifests/0.log"
Sep 18 14:23:38.077850 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:38.077813    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-978ccd68d-2l9f7_278aa551-51a0-4564-af58-0d20184ab87a/manager/0.log"
Sep 18 14:23:38.687556 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:38.687527    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-78fd7758f9-9lnx7_60c1c143-edf0-47e2-9d92-524d813ce34a/maas-api/0.log"
Sep 18 14:23:39.925308 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:39.925273    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6cd9cc5c75-j2z8r_4f42e29e-9888-4eab-847f-159c841c49ed/authorino/0.log"
Sep 18 14:23:40.285718 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:40.285638    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-2ll27_109ada03-9032-44a0-b216-e6852ce3abb9/kuadrant-console-plugin/0.log"
Sep 18 14:23:40.556049 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:40.555977    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-ljk2l_2272b7c5-5194-414b-84ea-5380c4a73d2e/limitador/0.log"
Sep 18 14:23:41.269112 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:41.269080    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d5c57f774-6wnxx_5e49a69a-0b86-4a68-b0fa-09d7c271e040/kube-auth-proxy/0.log"
Sep 18 14:23:42.624091 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:42.624062    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2gnbv_3e5aeebd-6155-49a7-883c-0309a304a033/main/0.log"
Sep 18 14:23:42.845274 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:42.845246    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-9vhpj_e41aa385-7758-49b1-8d2f-abfa51d38a99/main/0.log"
Sep 18 14:23:48.805845 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:48.805723    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8mptg/must-gather-9wgfs"]
Sep 18 14:23:48.810752 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:48.806057    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18b8985c-ea40-46f8-bf31-2bb78b6fe088" containerName="maas-api"
Sep 18 14:23:48.810752 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:48.806070    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="18b8985c-ea40-46f8-bf31-2bb78b6fe088" containerName="maas-api"
Sep 18 14:23:48.810752 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:48.806139    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="18b8985c-ea40-46f8-bf31-2bb78b6fe088" containerName="maas-api"
Sep 18 14:23:48.810752 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:48.809925    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8mptg/must-gather-9wgfs"
Sep 18 14:23:48.812933 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:48.812913    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8mptg\"/\"kube-root-ca.crt\""
Sep 18 14:23:48.813080 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:48.812935    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8mptg\"/\"openshift-service-ca.crt\""
Sep 18 14:23:48.813497 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:48.813477    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-8mptg\"/\"default-dockercfg-9mm44\""
Sep 18 14:23:48.821632 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:48.821607    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8mptg/must-gather-9wgfs"]
Sep 18 14:23:48.825122 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:48.825096    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:48.924206 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:48.924171    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:48.926377 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:48.926353    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6e4540df-0416-4426-bcbf-ab2823208e59-must-gather-output\") pod \"must-gather-9wgfs\" (UID: \"6e4540df-0416-4426-bcbf-ab2823208e59\") " pod="openshift-must-gather-8mptg/must-gather-9wgfs"
Sep 18 14:23:48.926442 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:48.926412    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gb9t\" (UniqueName: \"kubernetes.io/projected/6e4540df-0416-4426-bcbf-ab2823208e59-kube-api-access-8gb9t\") pod \"must-gather-9wgfs\" (UID: \"6e4540df-0416-4426-bcbf-ab2823208e59\") " pod="openshift-must-gather-8mptg/must-gather-9wgfs"
Sep 18 14:23:49.027747 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:49.027696    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6e4540df-0416-4426-bcbf-ab2823208e59-must-gather-output\") pod \"must-gather-9wgfs\" (UID: \"6e4540df-0416-4426-bcbf-ab2823208e59\") " pod="openshift-must-gather-8mptg/must-gather-9wgfs"
Sep 18 14:23:49.027984 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:49.027791    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gb9t\" (UniqueName: \"kubernetes.io/projected/6e4540df-0416-4426-bcbf-ab2823208e59-kube-api-access-8gb9t\") pod \"must-gather-9wgfs\" (UID: \"6e4540df-0416-4426-bcbf-ab2823208e59\") " pod="openshift-must-gather-8mptg/must-gather-9wgfs"
Sep 18 14:23:49.028144 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:49.028118    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6e4540df-0416-4426-bcbf-ab2823208e59-must-gather-output\") pod \"must-gather-9wgfs\" (UID: \"6e4540df-0416-4426-bcbf-ab2823208e59\") " pod="openshift-must-gather-8mptg/must-gather-9wgfs"
Sep 18 14:23:49.036345 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:49.036319    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gb9t\" (UniqueName: \"kubernetes.io/projected/6e4540df-0416-4426-bcbf-ab2823208e59-kube-api-access-8gb9t\") pod \"must-gather-9wgfs\" (UID: \"6e4540df-0416-4426-bcbf-ab2823208e59\") " pod="openshift-must-gather-8mptg/must-gather-9wgfs"
Sep 18 14:23:49.119809 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:49.119777    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8mptg/must-gather-9wgfs"
Sep 18 14:23:49.396019 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:49.395985    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8mptg/must-gather-9wgfs"]
Sep 18 14:23:49.397979 ip-10-0-141-222 kubenswrapper[2577]: W0918 14:23:49.397945    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6e4540df_0416_4426_bcbf_ab2823208e59.slice/crio-beb1ba40b1578a98e956a13b919b19c44affb31e9e24230f26b8879028732751 WatchSource:0}: Error finding container beb1ba40b1578a98e956a13b919b19c44affb31e9e24230f26b8879028732751: Status 404 returned error can't find the container with id beb1ba40b1578a98e956a13b919b19c44affb31e9e24230f26b8879028732751
Sep 18 14:23:50.169145 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:50.169110    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8mptg/must-gather-9wgfs" event={"ID":"6e4540df-0416-4426-bcbf-ab2823208e59","Type":"ContainerStarted","Data":"beb1ba40b1578a98e956a13b919b19c44affb31e9e24230f26b8879028732751"}
Sep 18 14:23:51.176900 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:51.176857    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8mptg/must-gather-9wgfs" event={"ID":"6e4540df-0416-4426-bcbf-ab2823208e59","Type":"ContainerStarted","Data":"4923765357e0a42fd5b6b26ef276223ba34a2bd7c7a4504cd63cbbbcdf175675"}
Sep 18 14:23:51.177466 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:51.177435    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8mptg/must-gather-9wgfs" event={"ID":"6e4540df-0416-4426-bcbf-ab2823208e59","Type":"ContainerStarted","Data":"f07073b6a21948d6d6c2fef58518016f467dfd0301705057cca43834633f4c38"}
Sep 18 14:23:51.194912 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:51.194849    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-8mptg/must-gather-9wgfs" podStartSLOduration=2.396085529 podStartE2EDuration="3.194811712s" totalImagesPullingTime="798.726183ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:23:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:23:51.192412282 +0000 UTC m=+2380.245071624" watchObservedRunningTime="2026-09-18 14:23:51.194811712 +0000 UTC m=+2380.247471040"
Sep 18 14:23:51.891324 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:51.891289    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-bdwmn_802e0a84-ff96-44d6-9aa2-db34555f46eb/global-pull-secret-syncer/0.log"
Sep 18 14:23:51.993093 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:51.993062    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-7cm48_8902efdf-cf1d-4513-bfbb-cdf2b9a2b922/konnectivity-agent/0.log"
Sep 18 14:23:52.108156 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:52.108124    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-222.ec2.internal_4a4b0c9f3c3ac36b289317453f80d9d6/haproxy/0.log"
Sep 18 14:23:56.970665 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:56.968246    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6cd9cc5c75-j2z8r_4f42e29e-9888-4eab-847f-159c841c49ed/authorino/0.log"
Sep 18 14:23:57.069318 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:57.069282    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-2ll27_109ada03-9032-44a0-b216-e6852ce3abb9/kuadrant-console-plugin/0.log"
Sep 18 14:23:57.142552 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:57.142512    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-ljk2l_2272b7c5-5194-414b-84ea-5380c4a73d2e/limitador/0.log"
Sep 18 14:23:58.731873 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:58.731811    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-847b8bd8d7-7g9jh_38fce007-dfca-4566-bea9-3e8c2ea32974/cluster-monitoring-operator/0.log"
Sep 18 14:23:58.889110 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:58.889073    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jpl99_f8742812-3cdb-4df7-801b-443e5178cb24/node-exporter/0.log"
Sep 18 14:23:58.914843 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:58.914791    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jpl99_f8742812-3cdb-4df7-801b-443e5178cb24/kube-rbac-proxy/0.log"
Sep 18 14:23:58.940527 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:23:58.940496    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jpl99_f8742812-3cdb-4df7-801b-443e5178cb24/init-textfile/0.log"
Sep 18 14:24:01.742394 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:24:01.742345    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-8tnxl_69d3178a-5769-4b5f-a243-30a1bedc5bd6/console-operator/1.log"
Sep 18 14:24:01.749181 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:24:01.749106    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-8tnxl_69d3178a-5769-4b5f-a243-30a1bedc5bd6/console-operator/2.log"
Sep 18 14:24:01.817757 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:24:01.817724    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8mptg/perf-node-gather-daemonset-b52hm"]
Sep 18 14:24:01.824079 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:24:01.824047    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8mptg/perf-node-gather-daemonset-b52hm"
Sep 18 14:24:01.828374 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:24:01.828344    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8mptg/perf-node-gather-daemonset-b52hm"]
Sep 18 14:24:01.965158 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:24:01.965111    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0b18e63d-0013-4058-969e-1d06b1044dbb-lib-modules\") pod \"perf-node-gather-daemonset-b52hm\" (UID: \"0b18e63d-0013-4058-969e-1d06b1044dbb\") " pod="openshift-must-gather-8mptg/perf-node-gather-daemonset-b52hm"
Sep 18 14:24:01.965158 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:24:01.965159    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0b18e63d-0013-4058-969e-1d06b1044dbb-podres\") pod \"perf-node-gather-daemonset-b52hm\" (UID: \"0b18e63d-0013-4058-969e-1d06b1044dbb\") " pod="openshift-must-gather-8mptg/perf-node-gather-daemonset-b52hm"
Sep 18 14:24:01.965432 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:24:01.965258    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx9pr\" (UniqueName: \"kubernetes.io/projected/0b18e63d-0013-4058-969e-1d06b1044dbb-kube-api-access-vx9pr\") pod \"perf-node-gather-daemonset-b52hm\" (UID: \"0b18e63d-0013-4058-969e-1d06b1044dbb\") " pod="openshift-must-gather-8mptg/perf-node-gather-daemonset-b52hm"
Sep 18 14:24:01.965432 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:24:01.965318    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0b18e63d-0013-4058-969e-1d06b1044dbb-sys\") pod \"perf-node-gather-daemonset-b52hm\" (UID: \"0b18e63d-0013-4058-969e-1d06b1044dbb\") " pod="openshift-must-gather-8mptg/perf-node-gather-daemonset-b52hm"
Sep 18 14:24:01.965432 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:24:01.965389    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0b18e63d-0013-4058-969e-1d06b1044dbb-proc\") pod \"perf-node-gather-daemonset-b52hm\" (UID: \"0b18e63d-0013-4058-969e-1d06b1044dbb\") " pod="openshift-must-gather-8mptg/perf-node-gather-daemonset-b52hm"
Sep 18 14:24:02.066858 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:24:02.066733    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0b18e63d-0013-4058-969e-1d06b1044dbb-proc\") pod \"perf-node-gather-daemonset-b52hm\" (UID: \"0b18e63d-0013-4058-969e-1d06b1044dbb\") " pod="openshift-must-gather-8mptg/perf-node-gather-daemonset-b52hm"
Sep 18 14:24:02.067040 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:24:02.066859    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0b18e63d-0013-4058-969e-1d06b1044dbb-lib-modules\") pod \"perf-node-gather-daemonset-b52hm\" (UID: \"0b18e63d-0013-4058-969e-1d06b1044dbb\") " pod="openshift-must-gather-8mptg/perf-node-gather-daemonset-b52hm"
Sep 18 14:24:02.067040 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:24:02.066892    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0b18e63d-0013-4058-969e-1d06b1044dbb-podres\") pod \"perf-node-gather-daemonset-b52hm\" (UID: \"0b18e63d-0013-4058-969e-1d06b1044dbb\") " pod="openshift-must-gather-8mptg/perf-node-gather-daemonset-b52hm"
Sep 18 14:24:02.067040 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:24:02.066896    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0b18e63d-0013-4058-969e-1d06b1044dbb-proc\") pod \"perf-node-gather-daemonset-b52hm\" (UID: \"0b18e63d-0013-4058-969e-1d06b1044dbb\") " pod="openshift-must-gather-8mptg/perf-node-gather-daemonset-b52hm"
Sep 18 14:24:02.067040 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:24:02.066936    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx9pr\" (UniqueName: \"kubernetes.io/projected/0b18e63d-0013-4058-969e-1d06b1044dbb-kube-api-access-vx9pr\") pod \"perf-node-gather-daemonset-b52hm\" (UID: \"0b18e63d-0013-4058-969e-1d06b1044dbb\") " pod="openshift-must-gather-8mptg/perf-node-gather-daemonset-b52hm"
Sep 18 14:24:02.067040 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:24:02.066981    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0b18e63d-0013-4058-969e-1d06b1044dbb-sys\") pod \"perf-node-gather-daemonset-b52hm\" (UID: \"0b18e63d-0013-4058-969e-1d06b1044dbb\") " pod="openshift-must-gather-8mptg/perf-node-gather-daemonset-b52hm"
Sep 18 14:24:02.067300 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:24:02.067057    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0b18e63d-0013-4058-969e-1d06b1044dbb-lib-modules\") pod \"perf-node-gather-daemonset-b52hm\" (UID: \"0b18e63d-0013-4058-969e-1d06b1044dbb\") " pod="openshift-must-gather-8mptg/perf-node-gather-daemonset-b52hm"
Sep 18 14:24:02.067300 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:24:02.067068    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0b18e63d-0013-4058-969e-1d06b1044dbb-sys\") pod \"perf-node-gather-daemonset-b52hm\" (UID: \"0b18e63d-0013-4058-969e-1d06b1044dbb\") " pod="openshift-must-gather-8mptg/perf-node-gather-daemonset-b52hm"
Sep 18 14:24:02.067300 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:24:02.067144    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0b18e63d-0013-4058-969e-1d06b1044dbb-podres\") pod \"perf-node-gather-daemonset-b52hm\" (UID: \"0b18e63d-0013-4058-969e-1d06b1044dbb\") " pod="openshift-must-gather-8mptg/perf-node-gather-daemonset-b52hm"
Sep 18 14:24:02.075903 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:24:02.075816    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx9pr\" (UniqueName: \"kubernetes.io/projected/0b18e63d-0013-4058-969e-1d06b1044dbb-kube-api-access-vx9pr\") pod \"perf-node-gather-daemonset-b52hm\" (UID: \"0b18e63d-0013-4058-969e-1d06b1044dbb\") " pod="openshift-must-gather-8mptg/perf-node-gather-daemonset-b52hm"
Sep 18 14:24:02.139279 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:24:02.139236    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8mptg/perf-node-gather-daemonset-b52hm"
Sep 18 14:24:02.367584 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:24:02.367548    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6594f4c75-xhzgm_e549fe9c-3866-403f-bbd4-f72d0ec6bd70/download-server/0.log"
Sep 18 14:24:02.451908 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:24:02.451865    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8mptg/perf-node-gather-daemonset-b52hm"]
Sep 18 14:24:03.272364 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:24:03.272326    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8mptg/perf-node-gather-daemonset-b52hm" event={"ID":"0b18e63d-0013-4058-969e-1d06b1044dbb","Type":"ContainerStarted","Data":"3599eedcd0c58e32f1e45dd03611d8b704017b3cf08d536fddba15119f7f8979"}
Sep 18 14:24:03.272877 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:24:03.272387    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8mptg/perf-node-gather-daemonset-b52hm" event={"ID":"0b18e63d-0013-4058-969e-1d06b1044dbb","Type":"ContainerStarted","Data":"fa72f68c640f6eb3c61015ce226ed33528624864956187e7d4bc90c1732cbd41"}
Sep 18 14:24:03.272877 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:24:03.272416    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-8mptg/perf-node-gather-daemonset-b52hm"
Sep 18 14:24:03.289440 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:24:03.289370    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-8mptg/perf-node-gather-daemonset-b52hm" podStartSLOduration=2.289348058 podStartE2EDuration="2.289348058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 14:24:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:24:03.286269748 +0000 UTC m=+2392.338929077" watchObservedRunningTime="2026-09-18 14:24:03.289348058 +0000 UTC m=+2392.342007385"
Sep 18 14:24:04.249654 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:24:04.249616    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fcbr5_ce0ee0f8-b52b-4ddc-86f9-41f650a590b2/dns/0.log"
Sep 18 14:24:04.276605 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:24:04.276577    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fcbr5_ce0ee0f8-b52b-4ddc-86f9-41f650a590b2/kube-rbac-proxy/0.log"
Sep 18 14:24:04.376297 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:24:04.376264    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vjv4c_dbd84791-daa9-43f0-9f32-8f4415eceeca/dns-node-resolver/0.log"
Sep 18 14:24:05.033571 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:24:05.033540    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wczf6_ac788d27-88a8-4ed2-9363-fe4f0aa51013/node-ca/0.log"
Sep 18 14:24:06.421954 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:24:06.421925    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d5c57f774-6wnxx_5e49a69a-0b86-4a68-b0fa-09d7c271e040/kube-auth-proxy/0.log"
Sep 18 14:24:07.506035 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:24:07.505992    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hlqkp_e5510d2c-e7d9-4512-934a-a2044c0ba084/serve-healthcheck-canary/0.log"
Sep 18 14:24:08.188404 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:24:08.188377    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6lm8v_9d9c9436-8b6c-4966-95f5-2cf258e95cc4/kube-rbac-proxy/0.log"
Sep 18 14:24:08.214388 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:24:08.214362    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6lm8v_9d9c9436-8b6c-4966-95f5-2cf258e95cc4/exporter/0.log"
Sep 18 14:24:08.236880 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:24:08.236846    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6lm8v_9d9c9436-8b6c-4966-95f5-2cf258e95cc4/extractor/0.log"
Sep 18 14:24:09.292289 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:24:09.292261    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-8mptg/perf-node-gather-daemonset-b52hm"
Sep 18 14:24:10.876314 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:24:10.876276    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-978ccd68d-2l9f7_278aa551-51a0-4564-af58-0d20184ab87a/manager/0.log"
Sep 18 14:24:10.899004 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:24:10.898973    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-978ccd68d-2l9f7_278aa551-51a0-4564-af58-0d20184ab87a/copy-manifests/0.log"
Sep 18 14:24:12.456855 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:24:12.456807    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6cdc5b77c9-hm24r_8a2f0da4-3361-4b70-985b-c7ecadb8c4a8/manager/0.log"
Sep 18 14:24:12.481996 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:24:12.481970    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-5m9bf_9082649d-4dfb-4491-b8f6-854fb3ef7485/openshift-lws-operator/0.log"
Sep 18 14:24:13.200441 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:24:13.200411    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d5c57f774-6wnxx_5e49a69a-0b86-4a68-b0fa-09d7c271e040/kube-auth-proxy/0.log"
Sep 18 14:24:13.353110 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:24:13.353081    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-8tnxl_69d3178a-5769-4b5f-a243-30a1bedc5bd6/console-operator/1.log"
Sep 18 14:24:13.363168 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:24:13.363141    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-8tnxl_69d3178a-5769-4b5f-a243-30a1bedc5bd6/console-operator/1.log"
Sep 18 14:24:15.399450 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:24:15.399416    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d5c57f774-6wnxx_5e49a69a-0b86-4a68-b0fa-09d7c271e040/kube-auth-proxy/0.log"
Sep 18 14:24:17.850714 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:24:17.850680    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-b77cp_0c3e7539-d533-47e4-ad17-dbe6fccdd6ba/migrator/0.log"
Sep 18 14:24:17.872374 ip-10-0-141-222 kubenswrapper[2577]: I0918 14:24:17.872346    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-b77cp_0c3e7539-d533-47e4-ad17-dbe6fccdd6ba/graceful-termination/0.log"