Sep 14 02:50:35.420038 ip-10-0-141-4 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 14 02:50:35.420048 ip-10-0-141-4 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 14 02:50:35.420055 ip-10-0-141-4 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 14 02:50:35.420321 ip-10-0-141-4 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 14 02:50:45.545457 ip-10-0-141-4 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 14 02:50:45.545472 ip-10-0-141-4 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 98487bd2d5514313bedde45d75bca470 --
Sep 14 02:52:33.089247 ip-10-0-141-4 systemd[1]: Starting Kubernetes Kubelet...
Sep 14 02:52:33.509753 ip-10-0-141-4 kubenswrapper[2564]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 14 02:52:33.509753 ip-10-0-141-4 kubenswrapper[2564]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 14 02:52:33.509753 ip-10-0-141-4 kubenswrapper[2564]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 14 02:52:33.509753 ip-10-0-141-4 kubenswrapper[2564]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 14 02:52:33.509753 ip-10-0-141-4 kubenswrapper[2564]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 14 02:52:33.510374 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.509744    2564 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 14 02:52:33.512645 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512629    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 14 02:52:33.512645 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512644    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 14 02:52:33.512706 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512647    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 14 02:52:33.512706 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512651    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 14 02:52:33.512706 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512654    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 14 02:52:33.512706 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512658    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 14 02:52:33.512706 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512660    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 14 02:52:33.512706 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512663    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 14 02:52:33.512706 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512666    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 14 02:52:33.512706 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512669    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 14 02:52:33.512706 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512671    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 14 02:52:33.512706 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512674    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 14 02:52:33.512706 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512676    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 14 02:52:33.512706 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512679    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 14 02:52:33.512706 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512683    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 14 02:52:33.512706 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512685    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 14 02:52:33.512706 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512688    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 14 02:52:33.512706 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512691    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 14 02:52:33.512706 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512694    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 14 02:52:33.512706 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512696    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 14 02:52:33.512706 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512699    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 14 02:52:33.513150 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512702    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 14 02:52:33.513150 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512705    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 14 02:52:33.513150 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512708    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 14 02:52:33.513150 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512711    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 14 02:52:33.513150 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512714    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 14 02:52:33.513150 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512717    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 14 02:52:33.513150 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512720    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 14 02:52:33.513150 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512722    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 14 02:52:33.513150 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512725    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 14 02:52:33.513150 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512728    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 14 02:52:33.513150 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512730    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 14 02:52:33.513150 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512733    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 14 02:52:33.513150 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512735    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 14 02:52:33.513150 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512739    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 14 02:52:33.513150 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512742    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 14 02:52:33.513150 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512745    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 14 02:52:33.513150 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512747    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 14 02:52:33.513150 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512750    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 14 02:52:33.513150 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512753    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 14 02:52:33.513150 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512755    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 14 02:52:33.513624 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512758    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 14 02:52:33.513624 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512760    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 14 02:52:33.513624 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512762    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 14 02:52:33.513624 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512765    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 14 02:52:33.513624 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512767    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 14 02:52:33.513624 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512770    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 14 02:52:33.513624 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512772    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 14 02:52:33.513624 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512775    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 14 02:52:33.513624 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512777    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 14 02:52:33.513624 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512780    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 14 02:52:33.513624 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512783    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 14 02:52:33.513624 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512785    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 14 02:52:33.513624 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512787    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 14 02:52:33.513624 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512792    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 14 02:52:33.513624 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512794    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 14 02:52:33.513624 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512799    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 14 02:52:33.513624 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512803    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 14 02:52:33.513624 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512806    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 14 02:52:33.513624 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512809    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 14 02:52:33.514096 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512812    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 14 02:52:33.514096 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512815    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 14 02:52:33.514096 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512817    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 14 02:52:33.514096 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512820    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 14 02:52:33.514096 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512823    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 14 02:52:33.514096 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512825    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 14 02:52:33.514096 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512828    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 14 02:52:33.514096 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512831    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 14 02:52:33.514096 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512834    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 14 02:52:33.514096 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512836    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 14 02:52:33.514096 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512839    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 14 02:52:33.514096 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512844    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 14 02:52:33.514096 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512848    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 14 02:52:33.514096 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512851    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 14 02:52:33.514096 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512865    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 14 02:52:33.514096 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512868    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 14 02:52:33.514096 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512871    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 14 02:52:33.514096 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512873    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 14 02:52:33.514096 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512878    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 14 02:52:33.514096 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512880    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 14 02:52:33.514565 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512883    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 14 02:52:33.514565 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512887    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 14 02:52:33.514565 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512890    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 14 02:52:33.514565 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512894    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 14 02:52:33.514565 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512896    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 14 02:52:33.514565 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512899    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 14 02:52:33.514565 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512902    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 14 02:52:33.514565 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512904    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 14 02:52:33.514565 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512907    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 14 02:52:33.514565 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512910    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 14 02:52:33.514565 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512912    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 14 02:52:33.514565 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512915    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 14 02:52:33.514565 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512918    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 14 02:52:33.514565 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512920    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 14 02:52:33.514565 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512923    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 14 02:52:33.514565 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512925    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 14 02:52:33.514565 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512928    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 14 02:52:33.514565 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512931    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 14 02:52:33.514565 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.512933    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 14 02:52:33.514565 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513603    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 14 02:52:33.515064 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513609    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 14 02:52:33.515064 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513613    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 14 02:52:33.515064 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513616    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 14 02:52:33.515064 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513620    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 14 02:52:33.515064 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513624    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 14 02:52:33.515064 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513628    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 14 02:52:33.515064 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513631    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 14 02:52:33.515064 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513633    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 14 02:52:33.515064 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513636    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 14 02:52:33.515064 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513639    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 14 02:52:33.515064 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513642    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 14 02:52:33.515064 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513645    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 14 02:52:33.515064 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513648    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 14 02:52:33.515064 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513652    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 14 02:52:33.515064 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513657    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 14 02:52:33.515064 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513660    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 14 02:52:33.515064 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513662    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 14 02:52:33.515064 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513665    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 14 02:52:33.515064 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513668    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 14 02:52:33.515064 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513671    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 14 02:52:33.515557 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513674    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 14 02:52:33.515557 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513676    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 14 02:52:33.515557 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513679    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 14 02:52:33.515557 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513682    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 14 02:52:33.515557 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513684    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 14 02:52:33.515557 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513687    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 14 02:52:33.515557 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513690    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 14 02:52:33.515557 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513692    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 14 02:52:33.515557 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513695    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 14 02:52:33.515557 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513697    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 14 02:52:33.515557 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513700    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 14 02:52:33.515557 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513703    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 14 02:52:33.515557 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513705    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 14 02:52:33.515557 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513708    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 14 02:52:33.515557 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513710    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 14 02:52:33.515557 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513713    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 14 02:52:33.515557 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513716    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 14 02:52:33.515557 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513719    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 14 02:52:33.515557 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513721    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 14 02:52:33.515557 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513724    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 14 02:52:33.516056 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513726    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 14 02:52:33.516056 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513729    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 14 02:52:33.516056 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513732    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 14 02:52:33.516056 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513735    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 14 02:52:33.516056 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513738    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 14 02:52:33.516056 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513741    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 14 02:52:33.516056 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513744    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 14 02:52:33.516056 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513748    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 14 02:52:33.516056 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513750    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 14 02:52:33.516056 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513754    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 14 02:52:33.516056 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513758    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 14 02:52:33.516056 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513761    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 14 02:52:33.516056 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513763    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 14 02:52:33.516056 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513766    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 14 02:52:33.516056 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513769    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 14 02:52:33.516056 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513771    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 14 02:52:33.516056 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513774    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 14 02:52:33.516056 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513776    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 14 02:52:33.516056 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513779    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 14 02:52:33.516056 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513782    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 14 02:52:33.516541 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513784    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 14 02:52:33.516541 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513787    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 14 02:52:33.516541 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513790    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 14 02:52:33.516541 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513792    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 14 02:52:33.516541 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513795    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 14 02:52:33.516541 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513797    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 14 02:52:33.516541 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513800    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 14 02:52:33.516541 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513803    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 14 02:52:33.516541 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513805    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 14 02:52:33.516541 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513808    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 14 02:52:33.516541 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513813    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 14 02:52:33.516541 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513816    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 14 02:52:33.516541 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513819    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 14 02:52:33.516541 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513822    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 14 02:52:33.516541 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513824    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 14 02:52:33.516541 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513827    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 14 02:52:33.516541 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513829    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 14 02:52:33.516541 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513834    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 14 02:52:33.516541 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513837    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 14 02:52:33.516541 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513839    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 14 02:52:33.517039 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513842    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 14 02:52:33.517039 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513846    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 14 02:52:33.517039 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513849    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 14 02:52:33.517039 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513868    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 14 02:52:33.517039 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513875    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 14 02:52:33.517039 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513880    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 14 02:52:33.517039 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513885    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 14 02:52:33.517039 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513888    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 14 02:52:33.517039 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513891    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 14 02:52:33.517039 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513894    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 14 02:52:33.517039 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513896    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 14 02:52:33.517039 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513899    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 14 02:52:33.517039 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513901    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 14 02:52:33.517039 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513904    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 14 02:52:33.517039 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513907    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 14 02:52:33.517039 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513909    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 14 02:52:33.517039 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513912    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 14 02:52:33.517039 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.513914    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 14 02:52:33.517039 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514262    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 14 02:52:33.517039 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514267    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 14 02:52:33.517511 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514269    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 14 02:52:33.517511 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514272    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 14 02:52:33.517511 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514275    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 14 02:52:33.517511 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514278    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 14 02:52:33.517511 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514281    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 14 02:52:33.517511 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514284    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 14 02:52:33.517511 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514286    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 14 02:52:33.517511 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514289    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 14 02:52:33.517511 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514292    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 14 02:52:33.517511 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514296    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 14 02:52:33.517511 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514298    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 14 02:52:33.517511 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514301    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 14 02:52:33.517511 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514304    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 14 02:52:33.517511 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514306    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 14 02:52:33.517511 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514309    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 14 02:52:33.517511 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514312    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 14 02:52:33.517511 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514315    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 14 02:52:33.517511 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514318    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 14 02:52:33.517511 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514320    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 14 02:52:33.517511 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514323    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 14 02:52:33.517998 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514325    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 14 02:52:33.517998 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514328    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 14 02:52:33.517998 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514330    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 14 02:52:33.517998 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514333    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 14 02:52:33.517998 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514336    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 14 02:52:33.517998 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514340    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 14 02:52:33.517998 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514343    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 14 02:52:33.517998 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514346    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 14 02:52:33.517998 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514349    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 14 02:52:33.517998 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514351    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 14 02:52:33.517998 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514354    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 14 02:52:33.517998 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514357    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 14 02:52:33.517998 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514359    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 14 02:52:33.517998 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514362    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 14 02:52:33.517998 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514364    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 14 02:52:33.517998 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514367    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 14 02:52:33.517998 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514370    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 14 02:52:33.517998 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514373    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 14 02:52:33.517998 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514375    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 14 02:52:33.518444 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514378    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 14 02:52:33.518444 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514380    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 14 02:52:33.518444 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514384    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 14 02:52:33.518444 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514386    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 14 02:52:33.518444 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514389    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 14 02:52:33.518444 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514391    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 14 02:52:33.518444 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514395    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 14 02:52:33.518444 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514397    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 14 02:52:33.518444 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514400    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 14 02:52:33.518444 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514402    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 14 02:52:33.518444 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514405    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 14 02:52:33.518444 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514407    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 14 02:52:33.518444 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514410    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 14 02:52:33.518444 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514412    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 14 02:52:33.518444 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514415    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 14 02:52:33.518444 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514418    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 14 02:52:33.518444 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514420    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 14 02:52:33.518444 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514422    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 14 02:52:33.518444 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514425    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 14 02:52:33.518444 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514427    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 14 02:52:33.518923 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514430    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 14 02:52:33.518923 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514432    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 14 02:52:33.518923 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514435    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 14 02:52:33.518923 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514438    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 14 02:52:33.518923 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514440    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 14 02:52:33.518923 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514442    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 14 02:52:33.518923 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514446    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 14 02:52:33.518923 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514450    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 14 02:52:33.518923 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514453    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 14 02:52:33.518923 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514457    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 14 02:52:33.518923 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514460    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 14 02:52:33.518923 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514463    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 14 02:52:33.518923 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514465    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 14 02:52:33.518923 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514468    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 14 02:52:33.518923 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514472    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 14 02:52:33.518923 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514474    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 14 02:52:33.518923 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514477    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 14 02:52:33.518923 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514479    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 14 02:52:33.518923 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514483    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 14 02:52:33.519452 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514485    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 14 02:52:33.519452 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514488    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 14 02:52:33.519452 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514491    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 14 02:52:33.519452 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514493    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 14 02:52:33.519452 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514496    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 14 02:52:33.519452 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514498    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 14 02:52:33.519452 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514501    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 14 02:52:33.519452 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514503    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 14 02:52:33.519452 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514506    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 14 02:52:33.519452 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514508    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 14 02:52:33.519452 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514511    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 14 02:52:33.519452 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514513    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 14 02:52:33.519452 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514516    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 14 02:52:33.519452 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514518    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 14 02:52:33.519452 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514521    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 14 02:52:33.519452 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514523    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 14 02:52:33.519452 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514526    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 14 02:52:33.519452 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514528    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 14 02:52:33.519452 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.514531    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 14 02:52:33.519452 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514598    2564 flags.go:64] FLAG: --address="0.0.0.0"
Sep 14 02:52:33.519988 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514640    2564 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 14 02:52:33.519988 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514647    2564 flags.go:64] FLAG: --anonymous-auth="true"
Sep 14 02:52:33.519988 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514651    2564 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 14 02:52:33.519988 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514657    2564 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 14 02:52:33.519988 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514660    2564 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 14 02:52:33.519988 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514664    2564 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 14 02:52:33.519988 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514668    2564 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 14 02:52:33.519988 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514673    2564 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 14 02:52:33.519988 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514676    2564 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 14 02:52:33.519988 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514680    2564 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 14 02:52:33.519988 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514683    2564 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 14 02:52:33.519988 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514686    2564 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 14 02:52:33.519988 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514689    2564 flags.go:64] FLAG: --cgroup-root=""
Sep 14 02:52:33.519988 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514692    2564 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 14 02:52:33.519988 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514695    2564 flags.go:64] FLAG: --client-ca-file=""
Sep 14 02:52:33.519988 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514698    2564 flags.go:64] FLAG: --cloud-provider="external"
Sep 14 02:52:33.519988 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514700    2564 flags.go:64] FLAG: --cluster-dns="[]"
Sep 14 02:52:33.519988 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514704    2564 flags.go:64] FLAG: --cluster-domain=""
Sep 14 02:52:33.519988 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514707    2564 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 14 02:52:33.519988 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514710    2564 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 14 02:52:33.519988 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514714    2564 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 14 02:52:33.519988 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514717    2564 flags.go:64] FLAG: --container-log-max-files="5"
Sep 14 02:52:33.519988 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514720    2564 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 14 02:52:33.519988 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514723    2564 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 14 02:52:33.520559 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514726    2564 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 14 02:52:33.520559 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514730    2564 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 14 02:52:33.520559 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514733    2564 flags.go:64] FLAG: --contention-profiling="false"
Sep 14 02:52:33.520559 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514736    2564 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 14 02:52:33.520559 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514739    2564 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 14 02:52:33.520559 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514742    2564 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 14 02:52:33.520559 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514745    2564 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 14 02:52:33.520559 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514749    2564 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 14 02:52:33.520559 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514752    2564 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 14 02:52:33.520559 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514755    2564 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 14 02:52:33.520559 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514758    2564 flags.go:64] FLAG: --enable-load-reader="false"
Sep 14 02:52:33.520559 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514761    2564 flags.go:64] FLAG: --enable-server="true"
Sep 14 02:52:33.520559 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514764    2564 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 14 02:52:33.520559 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514768    2564 flags.go:64] FLAG: --event-burst="100"
Sep 14 02:52:33.520559 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514771    2564 flags.go:64] FLAG: --event-qps="50"
Sep 14 02:52:33.520559 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514777    2564 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 14 02:52:33.520559 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514780    2564 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 14 02:52:33.520559 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514782    2564 flags.go:64] FLAG: --eviction-hard=""
Sep 14 02:52:33.520559 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514786    2564 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 14 02:52:33.520559 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514789    2564 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 14 02:52:33.520559 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514792    2564 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 14 02:52:33.520559 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514795    2564 flags.go:64] FLAG: --eviction-soft=""
Sep 14 02:52:33.520559 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514797    2564 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 14 02:52:33.520559 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514800    2564 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 14 02:52:33.520559 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514803    2564 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 14 02:52:33.521183 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514806    2564 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 14 02:52:33.521183 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514809    2564 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 14 02:52:33.521183 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514812    2564 flags.go:64] FLAG: --fail-swap-on="true"
Sep 14 02:52:33.521183 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514815    2564 flags.go:64] FLAG: --feature-gates=""
Sep 14 02:52:33.521183 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514819    2564 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 14 02:52:33.521183 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514822    2564 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 14 02:52:33.521183 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514825    2564 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 14 02:52:33.521183 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514828    2564 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 14 02:52:33.521183 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514831    2564 flags.go:64] FLAG: --healthz-port="10248"
Sep 14 02:52:33.521183 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514834    2564 flags.go:64] FLAG: --help="false"
Sep 14 02:52:33.521183 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514836    2564 flags.go:64] FLAG: --hostname-override="ip-10-0-141-4.ec2.internal"
Sep 14 02:52:33.521183 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514839    2564 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 14 02:52:33.521183 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514842    2564 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 14 02:52:33.521183 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514845    2564 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 14 02:52:33.521183 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514848    2564 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 14 02:52:33.521183 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514867    2564 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 14 02:52:33.521183 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514871    2564 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 14 02:52:33.521183 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514874    2564 flags.go:64] FLAG: --image-service-endpoint=""
Sep 14 02:52:33.521183 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514876    2564 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 14 02:52:33.521183 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514879    2564 flags.go:64] FLAG: --kube-api-burst="100"
Sep 14 02:52:33.521183 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514883    2564 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 14 02:52:33.521183 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514886    2564 flags.go:64] FLAG: --kube-api-qps="50"
Sep 14 02:52:33.521183 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514890    2564 flags.go:64] FLAG: --kube-reserved=""
Sep 14 02:52:33.521183 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514892    2564 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 14 02:52:33.521773 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514895    2564 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 14 02:52:33.521773 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514898    2564 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 14 02:52:33.521773 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514901    2564 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 14 02:52:33.521773 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514904    2564 flags.go:64] FLAG: --lock-file=""
Sep 14 02:52:33.521773 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514906    2564 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 14 02:52:33.521773 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514909    2564 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 14 02:52:33.521773 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514912    2564 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 14 02:52:33.521773 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514917    2564 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 14 02:52:33.521773 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514920    2564 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 14 02:52:33.521773 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514923    2564 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 14 02:52:33.521773 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514925    2564 flags.go:64] FLAG: --logging-format="text"
Sep 14 02:52:33.521773 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514929    2564 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 14 02:52:33.521773 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514932    2564 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 14 02:52:33.521773 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514935    2564 flags.go:64] FLAG: --manifest-url=""
Sep 14 02:52:33.521773 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514938    2564 flags.go:64] FLAG: --manifest-url-header=""
Sep 14 02:52:33.521773 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514942    2564 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 14 02:52:33.521773 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514945    2564 flags.go:64] FLAG: --max-open-files="1000000"
Sep 14 02:52:33.521773 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514948    2564 flags.go:64] FLAG: --max-pods="110"
Sep 14 02:52:33.521773 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514951    2564 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 14 02:52:33.521773 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514954    2564 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 14 02:52:33.521773 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514957    2564 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 14 02:52:33.521773 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514960    2564 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 14 02:52:33.521773 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514963    2564 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 14 02:52:33.521773 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514970    2564 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 14 02:52:33.521773 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514973    2564 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 14 02:52:33.522430 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514980    2564 flags.go:64] FLAG: --node-status-max-images="50"
Sep 14 02:52:33.522430 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514983    2564 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 14 02:52:33.522430 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514986    2564 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 14 02:52:33.522430 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514989    2564 flags.go:64] FLAG: --pod-cidr=""
Sep 14 02:52:33.522430 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.514996    2564 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 14 02:52:33.522430 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.515002    2564 flags.go:64] FLAG: --pod-manifest-path=""
Sep 14 02:52:33.522430 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.515006    2564 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 14 02:52:33.522430 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.515009    2564 flags.go:64] FLAG: --pods-per-core="0"
Sep 14 02:52:33.522430 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.515012    2564 flags.go:64] FLAG: --port="10250"
Sep 14 02:52:33.522430 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.515015    2564 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 14 02:52:33.522430 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.515018    2564 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e6e1fb65f1f7f81e"
Sep 14 02:52:33.522430 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.515021    2564 flags.go:64] FLAG: --qos-reserved=""
Sep 14 02:52:33.522430 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.515024    2564 flags.go:64] FLAG: --read-only-port="10255"
Sep 14 02:52:33.522430 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.515026    2564 flags.go:64] FLAG: --register-node="true"
Sep 14 02:52:33.522430 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.515029    2564 flags.go:64] FLAG: --register-with-taints=""
Sep 14 02:52:33.522430 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.515033    2564 flags.go:64] FLAG: --registry-burst="10"
Sep 14 02:52:33.522430 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.515036    2564 flags.go:64] FLAG: --registry-qps="5"
Sep 14 02:52:33.522430 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.515039    2564 flags.go:64] FLAG: --reserved-cpus=""
Sep 14 02:52:33.522430 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.515041    2564 flags.go:64] FLAG: --reserved-memory=""
Sep 14 02:52:33.522430 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.515045    2564 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 14 02:52:33.522430 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.515048    2564 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 14 02:52:33.522430 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.515051    2564 flags.go:64] FLAG: --rotate-certificates="false"
Sep 14 02:52:33.522430 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.515054    2564 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 14 02:52:33.522430 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.515057    2564 flags.go:64] FLAG: --runonce="false"
Sep 14 02:52:33.522430 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.515059    2564 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 14 02:52:33.523050 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.515063    2564 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 14 02:52:33.523050 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.515066    2564 flags.go:64] FLAG: --seccomp-default="false"
Sep 14 02:52:33.523050 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.515069    2564 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 14 02:52:33.523050 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.515072    2564 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 14 02:52:33.523050 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.515075    2564 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 14 02:52:33.523050 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.515080    2564 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 14 02:52:33.523050 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.515084    2564 flags.go:64] FLAG: --storage-driver-password="root"
Sep 14 02:52:33.523050 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.515086    2564 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 14 02:52:33.523050 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.515089    2564 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 14 02:52:33.523050 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.515092    2564 flags.go:64] FLAG: --storage-driver-user="root"
Sep 14 02:52:33.523050 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.515095    2564 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 14 02:52:33.523050 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.515098    2564 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 14 02:52:33.523050 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.515107    2564 flags.go:64] FLAG: --system-cgroups=""
Sep 14 02:52:33.523050 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.515110    2564 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 14 02:52:33.523050 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.515116    2564 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 14 02:52:33.523050 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.515119    2564 flags.go:64] FLAG: --tls-cert-file=""
Sep 14 02:52:33.523050 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.515121    2564 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 14 02:52:33.523050 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.515130    2564 flags.go:64] FLAG: --tls-min-version=""
Sep 14 02:52:33.523050 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.515133    2564 flags.go:64] FLAG: --tls-private-key-file=""
Sep 14 02:52:33.523050 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.515136    2564 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 14 02:52:33.523050 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.515138    2564 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 14 02:52:33.523050 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.515141    2564 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 14 02:52:33.523050 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.515144    2564 flags.go:64] FLAG: --v="2"
Sep 14 02:52:33.523050 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.515148    2564 flags.go:64] FLAG: --version="false"
Sep 14 02:52:33.523050 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.515151    2564 flags.go:64] FLAG: --vmodule=""
Sep 14 02:52:33.523678 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.515155    2564 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 14 02:52:33.523678 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.515158    2564 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 14 02:52:33.523678 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515284    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 14 02:52:33.523678 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515290    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 14 02:52:33.523678 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515293    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 14 02:52:33.523678 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515296    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 14 02:52:33.523678 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515299    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 14 02:52:33.523678 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515302    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 14 02:52:33.523678 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515305    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 14 02:52:33.523678 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515309    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 14 02:52:33.523678 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515313    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 14 02:52:33.523678 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515317    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 14 02:52:33.523678 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515321    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 14 02:52:33.523678 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515324    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 14 02:52:33.523678 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515327    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 14 02:52:33.523678 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515330    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 14 02:52:33.523678 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515333    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 14 02:52:33.523678 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515335    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 14 02:52:33.523678 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515338    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 14 02:52:33.524160 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515341    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 14 02:52:33.524160 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515344    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 14 02:52:33.524160 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515346    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 14 02:52:33.524160 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515349    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 14 02:52:33.524160 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515352    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 14 02:52:33.524160 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515355    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 14 02:52:33.524160 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515358    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 14 02:52:33.524160 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515360    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 14 02:52:33.524160 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515363    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 14 02:52:33.524160 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515366    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 14 02:52:33.524160 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515369    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 14 02:52:33.524160 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515371    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 14 02:52:33.524160 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515374    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 14 02:52:33.524160 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515377    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 14 02:52:33.524160 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515379    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 14 02:52:33.524160 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515382    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 14 02:52:33.524160 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515384    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 14 02:52:33.524160 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515387    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 14 02:52:33.524160 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515390    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 14 02:52:33.524619 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515392    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 14 02:52:33.524619 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515395    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 14 02:52:33.524619 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515398    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 14 02:52:33.524619 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515400    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 14 02:52:33.524619 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515403    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 14 02:52:33.524619 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515406    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 14 02:52:33.524619 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515409    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 14 02:52:33.524619 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515412    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 14 02:52:33.524619 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515415    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 14 02:52:33.524619 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515419    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 14 02:52:33.524619 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515422    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 14 02:52:33.524619 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515425    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 14 02:52:33.524619 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515428    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 14 02:52:33.524619 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515431    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 14 02:52:33.524619 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515434    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 14 02:52:33.524619 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515437    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 14 02:52:33.524619 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515440    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 14 02:52:33.524619 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515443    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 14 02:52:33.524619 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515445    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 14 02:52:33.524619 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515448    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 14 02:52:33.525115 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515450    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 14 02:52:33.525115 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515453    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 14 02:52:33.525115 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515456    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 14 02:52:33.525115 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515458    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 14 02:52:33.525115 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515461    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 14 02:52:33.525115 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515463    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 14 02:52:33.525115 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515466    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 14 02:52:33.525115 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515468    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 14 02:52:33.525115 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515471    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 14 02:52:33.525115 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515473    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 14 02:52:33.525115 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515476    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 14 02:52:33.525115 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515479    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 14 02:52:33.525115 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515481    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 14 02:52:33.525115 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515484    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 14 02:52:33.525115 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515486    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 14 02:52:33.525115 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515489    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 14 02:52:33.525115 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515492    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 14 02:52:33.525115 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515494    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 14 02:52:33.525115 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515498    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 14 02:52:33.525115 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515501    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 14 02:52:33.525585 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515504    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 14 02:52:33.525585 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515506    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 14 02:52:33.525585 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515509    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 14 02:52:33.525585 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515511    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 14 02:52:33.525585 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515514    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 14 02:52:33.525585 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515517    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 14 02:52:33.525585 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515520    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 14 02:52:33.525585 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515522    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 14 02:52:33.525585 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515526    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 14 02:52:33.525585 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515528    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 14 02:52:33.525585 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515531    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 14 02:52:33.525585 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515533    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 14 02:52:33.525585 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515536    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 14 02:52:33.525585 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515539    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 14 02:52:33.525585 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515541    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 14 02:52:33.525585 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515544    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 14 02:52:33.525585 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515546    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 14 02:52:33.525585 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515549    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 14 02:52:33.525585 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515551    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 14 02:52:33.525585 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515554    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 14 02:52:33.526081 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515556    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 14 02:52:33.526081 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515559    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 14 02:52:33.526081 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.515562    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 14 02:52:33.526081 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.516272    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 14 02:52:33.526081 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.523337    2564 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 14 02:52:33.526081 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.523350    2564 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 14 02:52:33.526081 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.525088    2564 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 14 02:52:33.526081 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.525100    2564 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 14 02:52:33.526081 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525156    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 14 02:52:33.526081 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525161    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 14 02:52:33.526081 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525165    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 14 02:52:33.526081 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525168    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 14 02:52:33.526081 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525171    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 14 02:52:33.526081 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525173    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 14 02:52:33.526081 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525177    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 14 02:52:33.526464 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525180    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 14 02:52:33.526464 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525183    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 14 02:52:33.526464 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525186    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 14 02:52:33.526464 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525189    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 14 02:52:33.526464 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525191    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 14 02:52:33.526464 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525194    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 14 02:52:33.526464 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525197    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 14 02:52:33.526464 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525200    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 14 02:52:33.526464 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525203    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 14 02:52:33.526464 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525205    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 14 02:52:33.526464 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525208    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 14 02:52:33.526464 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525211    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 14 02:52:33.526464 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525213    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 14 02:52:33.526464 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525216    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 14 02:52:33.526464 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525219    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 14 02:52:33.526464 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525222    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 14 02:52:33.526464 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525224    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 14 02:52:33.526464 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525227    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 14 02:52:33.526464 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525229    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 14 02:52:33.526464 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525232    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 14 02:52:33.526464 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525235    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 14 02:52:33.527004 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525238    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 14 02:52:33.527004 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525242    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 14 02:52:33.527004 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525244    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 14 02:52:33.527004 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525247    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 14 02:52:33.527004 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525250    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 14 02:52:33.527004 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525252    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 14 02:52:33.527004 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525255    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 14 02:52:33.527004 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525258    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 14 02:52:33.527004 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525260    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 14 02:52:33.527004 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525263    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 14 02:52:33.527004 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525265    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 14 02:52:33.527004 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525267    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 14 02:52:33.527004 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525270    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 14 02:52:33.527004 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525272    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 14 02:52:33.527004 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525275    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 14 02:52:33.527004 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525278    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 14 02:52:33.527004 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525280    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 14 02:52:33.527004 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525283    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 14 02:52:33.527004 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525286    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 14 02:52:33.527004 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525289    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 14 02:52:33.527477 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525292    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 14 02:52:33.527477 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525294    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 14 02:52:33.527477 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525297    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 14 02:52:33.527477 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525300    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 14 02:52:33.527477 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525302    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 14 02:52:33.527477 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525305    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 14 02:52:33.527477 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525307    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 14 02:52:33.527477 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525310    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 14 02:52:33.527477 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525313    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 14 02:52:33.527477 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525315    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 14 02:52:33.527477 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525318    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 14 02:52:33.527477 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525321    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 14 02:52:33.527477 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525323    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 14 02:52:33.527477 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525326    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 14 02:52:33.527477 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525329    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 14 02:52:33.527477 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525332    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 14 02:52:33.527477 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525335    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 14 02:52:33.527477 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525338    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 14 02:52:33.527477 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525340    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 14 02:52:33.528010 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525343    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 14 02:52:33.528010 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525345    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 14 02:52:33.528010 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525348    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 14 02:52:33.528010 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525350    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 14 02:52:33.528010 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525354    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 14 02:52:33.528010 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525358    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 14 02:52:33.528010 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525361    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 14 02:52:33.528010 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525364    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 14 02:52:33.528010 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525367    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 14 02:52:33.528010 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525369    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 14 02:52:33.528010 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525373    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 14 02:52:33.528010 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525375    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 14 02:52:33.528010 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525378    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 14 02:52:33.528010 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525381    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 14 02:52:33.528010 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525383    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 14 02:52:33.528010 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525386    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 14 02:52:33.528010 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525388    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 14 02:52:33.528010 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525391    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 14 02:52:33.528010 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525393    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 14 02:52:33.528476 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525396    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 14 02:52:33.528476 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525398    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 14 02:52:33.528476 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525401    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 14 02:52:33.528476 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525404    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 14 02:52:33.528476 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525408    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 14 02:52:33.528476 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525412    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 14 02:52:33.528476 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525417    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 14 02:52:33.528476 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525420    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 14 02:52:33.528476 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525423    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 14 02:52:33.528476 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525426    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 14 02:52:33.528476 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525429    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 14 02:52:33.528476 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525432    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 14 02:52:33.528476 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525434    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 14 02:52:33.528476 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.525440    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 14 02:52:33.528828 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525547    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 14 02:52:33.528828 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525552    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 14 02:52:33.528828 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525555    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 14 02:52:33.528828 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525558    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 14 02:52:33.528828 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525560    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 14 02:52:33.528828 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525563    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 14 02:52:33.528828 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525565    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 14 02:52:33.528828 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525568    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 14 02:52:33.528828 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525571    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 14 02:52:33.528828 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525573    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 14 02:52:33.528828 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525576    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 14 02:52:33.528828 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525579    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 14 02:52:33.528828 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525581    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 14 02:52:33.528828 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525584    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 14 02:52:33.528828 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525587    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 14 02:52:33.528828 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525590    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 14 02:52:33.528828 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525592    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 14 02:52:33.528828 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525595    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 14 02:52:33.528828 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525598    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 14 02:52:33.529312 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525601    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 14 02:52:33.529312 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525604    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 14 02:52:33.529312 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525606    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 14 02:52:33.529312 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525609    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 14 02:52:33.529312 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525611    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 14 02:52:33.529312 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525614    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 14 02:52:33.529312 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525617    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 14 02:52:33.529312 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525620    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 14 02:52:33.529312 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525622    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 14 02:52:33.529312 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525625    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 14 02:52:33.529312 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525628    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 14 02:52:33.529312 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525631    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 14 02:52:33.529312 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525633    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 14 02:52:33.529312 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525636    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 14 02:52:33.529312 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525638    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 14 02:52:33.529312 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525641    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 14 02:52:33.529312 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525643    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 14 02:52:33.529312 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525646    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 14 02:52:33.529312 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525649    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 14 02:52:33.529312 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525651    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 14 02:52:33.529788 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525654    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 14 02:52:33.529788 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525657    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 14 02:52:33.529788 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525659    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 14 02:52:33.529788 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525662    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 14 02:52:33.529788 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525665    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 14 02:52:33.529788 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525669    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 14 02:52:33.529788 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525672    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 14 02:52:33.529788 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525675    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 14 02:52:33.529788 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525678    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 14 02:52:33.529788 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525681    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 14 02:52:33.529788 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525684    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 14 02:52:33.529788 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525686    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 14 02:52:33.529788 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525689    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 14 02:52:33.529788 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525692    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 14 02:52:33.529788 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525695    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 14 02:52:33.529788 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525697    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 14 02:52:33.529788 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525700    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 14 02:52:33.529788 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525703    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 14 02:52:33.529788 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525705    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 14 02:52:33.529788 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525708    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 14 02:52:33.530279 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525710    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 14 02:52:33.530279 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525713    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 14 02:52:33.530279 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525716    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 14 02:52:33.530279 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525718    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 14 02:52:33.530279 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525721    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 14 02:52:33.530279 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525723    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 14 02:52:33.530279 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525726    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 14 02:52:33.530279 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525728    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 14 02:52:33.530279 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525731    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 14 02:52:33.530279 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525733    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 14 02:52:33.530279 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525736    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 14 02:52:33.530279 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525738    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 14 02:52:33.530279 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525741    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 14 02:52:33.530279 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525743    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 14 02:52:33.530279 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525746    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 14 02:52:33.530279 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525748    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 14 02:52:33.530279 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525751    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 14 02:52:33.530279 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525753    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 14 02:52:33.530279 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525756    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 14 02:52:33.530279 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525758    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 14 02:52:33.530739 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525761    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 14 02:52:33.530739 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525765    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 14 02:52:33.530739 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525768    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 14 02:52:33.530739 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525771    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 14 02:52:33.530739 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525774    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 14 02:52:33.530739 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525777    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 14 02:52:33.530739 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525779    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 14 02:52:33.530739 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525782    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 14 02:52:33.530739 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525784    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 14 02:52:33.530739 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525787    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 14 02:52:33.530739 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525790    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 14 02:52:33.530739 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525792    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 14 02:52:33.530739 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525795    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 14 02:52:33.530739 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525797    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 14 02:52:33.530739 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525800    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 14 02:52:33.530739 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525803    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 14 02:52:33.530739 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525806    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 14 02:52:33.530739 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525808    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 14 02:52:33.530739 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525811    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 14 02:52:33.531196 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:33.525814    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 14 02:52:33.531196 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.525820    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 14 02:52:33.531196 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.525950    2564 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 14 02:52:33.531196 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.527914    2564 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 14 02:52:33.531196 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.528980    2564 server.go:1019] "Starting client certificate rotation"
Sep 14 02:52:33.531196 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.529080    2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 14 02:52:33.531196 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.529116    2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 14 02:52:33.531501 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.531488    2564 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 14 02:52:33.535887 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.535844    2564 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 14 02:52:33.549904 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.549889    2564 log.go:25] "Validated CRI v1 runtime API"
Sep 14 02:52:33.554318 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.554299    2564 log.go:25] "Validated CRI v1 image API"
Sep 14 02:52:33.555318 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.555284    2564 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 14 02:52:33.559305 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.559283    2564 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8111996a-01c6-4e8a-bfe4-b99fd3b4e968:/dev/nvme0n1p4 8e3f3503-bf42-4004-80b5-4acaf36c818f:/dev/nvme0n1p3]
Sep 14 02:52:33.559371 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.559305    2564 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 14 02:52:33.564326 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.564212    2564 manager.go:216] Machine: {Timestamp:2026-09-14 02:52:33.562896165 +0000 UTC m=+0.346528308 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100960 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2cfc6cff28a8aad3817fa91512b842 SystemUUID:ec2cfc6c-ff28-a8aa-d381-7fa91512b842 BootID:98487bd2-d551-4313-bedd-e45d75bca470 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c0:8e:9f:f1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c0:8e:9f:f1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:f2:02:ea:00:02:b7 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 14 02:52:33.564326 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.564323    2564 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 14 02:52:33.564430 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.564398    2564 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 14 02:52:33.565722 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.565706    2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 14 02:52:33.568482 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.568452    2564 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 14 02:52:33.568608 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.568484    2564 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-4.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 14 02:52:33.569816 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.569805    2564 topology_manager.go:138] "Creating topology manager with none policy"
Sep 14 02:52:33.569865 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.569819    2564 container_manager_linux.go:309] "Creating device plugin manager"
Sep 14 02:52:33.569865 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.569827    2564 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 14 02:52:33.569865 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.569842    2564 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 14 02:52:33.570553 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.570544    2564 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 14 02:52:33.574514 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.574502    2564 state_mem.go:36] "Initialized new in-memory state store"
Sep 14 02:52:33.574627 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.574617    2564 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 14 02:52:33.574666 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.574646    2564 kubelet.go:486] "Attempting to sync node with API server"
Sep 14 02:52:33.574666 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.574663    2564 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 14 02:52:33.574727 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.574677    2564 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 14 02:52:33.574727 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.574686    2564 kubelet.go:398] "Adding apiserver pod source"
Sep 14 02:52:33.574727 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.574701    2564 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 14 02:52:33.575082 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.575065    2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 14 02:52:33.575122 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.575091    2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 14 02:52:33.577199 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.577184    2564 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 14 02:52:33.579320 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.579306    2564 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 14 02:52:33.579373 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.579336    2564 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 14 02:52:33.580635 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.580625    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 14 02:52:33.580696 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.580640    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 14 02:52:33.580696 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.580646    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 14 02:52:33.580696 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.580652    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 14 02:52:33.580696 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.580661    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 14 02:52:33.580696 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.580683    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 14 02:52:33.580832 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.580700    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 14 02:52:33.580832 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.580706    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 14 02:52:33.580832 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.580712    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 14 02:52:33.580832 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.580717    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 14 02:52:33.580832 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.580730    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 14 02:52:33.580832 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.580735    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 14 02:52:33.581531 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.581495    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 14 02:52:33.581684 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.581663    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 14 02:52:33.585266 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.585250    2564 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-4.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 14 02:52:33.585762 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.585748    2564 server.go:1268] "Started kubelet"
Sep 14 02:52:33.585834 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.585810    2564 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 14 02:52:33.585907 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.585843    2564 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 14 02:52:33.585959 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.585924    2564 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 14 02:52:33.586036 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:33.586009    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-4.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 14 02:52:33.586283 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:33.585883    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 14 02:52:33.586556 ip-10-0-141-4 systemd[1]: Started Kubernetes Kubelet.
Sep 14 02:52:33.586911 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.586896    2564 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 14 02:52:33.587091 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.587077    2564 server.go:310] "Adding debug handlers to kubelet server"
Sep 14 02:52:33.591631 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.591615    2564 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 14 02:52:33.591739 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.591725    2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 14 02:52:33.591797 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:33.591773    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-4.ec2.internal\" not found"
Sep 14 02:52:33.592106 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:33.591249    2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-4.ec2.internal.18d5104ddec5f42b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-4.ec2.internal,UID:ip-10-0-141-4.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-4.ec2.internal,},FirstTimestamp:2026-09-14 02:52:33.585681451 +0000 UTC m=+0.369313596,LastTimestamp:2026-09-14 02:52:33.585681451 +0000 UTC m=+0.369313596,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-4.ec2.internal,}"
Sep 14 02:52:33.592206 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.592189    2564 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 14 02:52:33.592311 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.592292    2564 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 14 02:52:33.592401 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.592315    2564 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 14 02:52:33.592452 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.592412    2564 reconstruct.go:97] "Volume reconstruction finished"
Sep 14 02:52:33.592452 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.592422    2564 reconciler.go:29] "Reconciler: start to sync state"
Sep 14 02:52:33.592838 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.592813    2564 factory.go:55] Registering systemd factory
Sep 14 02:52:33.592937 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.592851    2564 factory.go:223] Registration of the systemd container factory successfully
Sep 14 02:52:33.593095 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.593076    2564 factory.go:153] Registering CRI-O factory
Sep 14 02:52:33.593160 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.593098    2564 factory.go:223] Registration of the crio container factory successfully
Sep 14 02:52:33.593160 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.593155    2564 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 14 02:52:33.593228 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.593173    2564 factory.go:103] Registering Raw factory
Sep 14 02:52:33.593228 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.593189    2564 manager.go:1196] Started watching for new ooms in manager
Sep 14 02:52:33.593605 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.593589    2564 manager.go:349] Starting recovery of all containers
Sep 14 02:52:33.593727 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:33.593658    2564 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 14 02:52:33.600515 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:33.600493    2564 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-4.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 14 02:52:33.600622 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:33.600603    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 14 02:52:33.605105 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.605086    2564 manager.go:354] Recovery completed
Sep 14 02:52:33.610894 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.610880    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 14 02:52:33.612481 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.612457    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-4.ec2.internal" event="NodeHasSufficientMemory"
Sep 14 02:52:33.612547 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.612492    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-4.ec2.internal" event="NodeHasNoDiskPressure"
Sep 14 02:52:33.612547 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.612501    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-4.ec2.internal" event="NodeHasSufficientPID"
Sep 14 02:52:33.613120 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.613107    2564 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 14 02:52:33.613120 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.613116    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 14 02:52:33.613214 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.613118    2564 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 14 02:52:33.613214 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.613193    2564 state_mem.go:36] "Initialized new in-memory state store"
Sep 14 02:52:33.613722 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.613704    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-4.ec2.internal" event="NodeHasSufficientMemory"
Sep 14 02:52:33.613811 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.613732    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-4.ec2.internal" event="NodeHasNoDiskPressure"
Sep 14 02:52:33.613811 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.613746    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-4.ec2.internal" event="NodeHasSufficientPID"
Sep 14 02:52:33.614609 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:33.614545    2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-4.ec2.internal.18d5104de05edcc1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-4.ec2.internal,UID:ip-10-0-141-4.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-4.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-4.ec2.internal,},FirstTimestamp:2026-09-14 02:52:33.612479681 +0000 UTC m=+0.396111828,LastTimestamp:2026-09-14 02:52:33.612479681 +0000 UTC m=+0.396111828,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-4.ec2.internal,}"
Sep 14 02:52:33.615354 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.615344    2564 policy_none.go:49] "None policy: Start"
Sep 14 02:52:33.615396 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.615357    2564 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 14 02:52:33.615396 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.615364    2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-d779q"
Sep 14 02:52:33.615396 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.615368    2564 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 14 02:52:33.617296 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.617285    2564 policy_none.go:47] "Start"
Sep 14 02:52:33.625073 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.625056    2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-d779q"
Sep 14 02:52:33.625403 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:33.625339    2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-4.ec2.internal.18d5104de05f1dea  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-4.ec2.internal,UID:ip-10-0-141-4.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-141-4.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-141-4.ec2.internal,},FirstTimestamp:2026-09-14 02:52:33.612496362 +0000 UTC m=+0.396128506,LastTimestamp:2026-09-14 02:52:33.612496362 +0000 UTC m=+0.396128506,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-4.ec2.internal,}"
Sep 14 02:52:33.644397 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.644382    2564 manager.go:337] "Starting Device Plugin manager"
Sep 14 02:52:33.665114 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:33.644420    2564 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 14 02:52:33.665114 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.644433    2564 server.go:87] "Starting device plugin registration server"
Sep 14 02:52:33.665114 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.644651    2564 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 14 02:52:33.665114 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.644663    2564 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 14 02:52:33.665114 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.644761    2564 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 14 02:52:33.665114 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.644839    2564 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 14 02:52:33.665114 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.644848    2564 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 14 02:52:33.665114 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:33.645432    2564 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 14 02:52:33.665114 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:33.645481    2564 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-4.ec2.internal\" not found"
Sep 14 02:52:33.687661 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.687631    2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 14 02:52:33.688745 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.688725    2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 14 02:52:33.688823 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.688767    2564 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 14 02:52:33.688823 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.688787    2564 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 14 02:52:33.688920 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:33.688825    2564 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 14 02:52:33.691440 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.691421    2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 14 02:52:33.745545 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.745517    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 14 02:52:33.746578 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.746561    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-4.ec2.internal" event="NodeHasSufficientMemory"
Sep 14 02:52:33.746683 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.746592    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-4.ec2.internal" event="NodeHasNoDiskPressure"
Sep 14 02:52:33.746683 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.746609    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-4.ec2.internal" event="NodeHasSufficientPID"
Sep 14 02:52:33.746683 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.746635    2564 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-4.ec2.internal"
Sep 14 02:52:33.756139 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.756121    2564 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-4.ec2.internal"
Sep 14 02:52:33.756220 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:33.756150    2564 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-4.ec2.internal\": node \"ip-10-0-141-4.ec2.internal\" not found"
Sep 14 02:52:33.771665 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:33.771608    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-4.ec2.internal\" not found"
Sep 14 02:52:33.789182 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.789151    2564 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-4.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-4.ec2.internal"]
Sep 14 02:52:33.789269 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.789240    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 14 02:52:33.790153 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.790138    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-4.ec2.internal" event="NodeHasSufficientMemory"
Sep 14 02:52:33.790230 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.790168    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-4.ec2.internal" event="NodeHasNoDiskPressure"
Sep 14 02:52:33.790230 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.790178    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-4.ec2.internal" event="NodeHasSufficientPID"
Sep 14 02:52:33.792463 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.792450    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 14 02:52:33.792610 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.792595    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-4.ec2.internal"
Sep 14 02:52:33.792668 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.792632    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 14 02:52:33.793190 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.793169    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-4.ec2.internal" event="NodeHasSufficientMemory"
Sep 14 02:52:33.793190 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.793179    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-4.ec2.internal" event="NodeHasSufficientMemory"
Sep 14 02:52:33.793283 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.793196    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-4.ec2.internal" event="NodeHasNoDiskPressure"
Sep 14 02:52:33.793283 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.793199    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-4.ec2.internal" event="NodeHasNoDiskPressure"
Sep 14 02:52:33.793283 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.793206    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-4.ec2.internal" event="NodeHasSufficientPID"
Sep 14 02:52:33.793283 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.793211    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-4.ec2.internal" event="NodeHasSufficientPID"
Sep 14 02:52:33.793439 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.793418    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f38ae5f0895b8efd6b111a875262c0a6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-4.ec2.internal\" (UID: \"f38ae5f0895b8efd6b111a875262c0a6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-4.ec2.internal"
Sep 14 02:52:33.793479 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.793445    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f38ae5f0895b8efd6b111a875262c0a6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-4.ec2.internal\" (UID: \"f38ae5f0895b8efd6b111a875262c0a6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-4.ec2.internal"
Sep 14 02:52:33.795378 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.795361    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-4.ec2.internal"
Sep 14 02:52:33.795457 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.795388    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 14 02:52:33.796063 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.796048    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-4.ec2.internal" event="NodeHasSufficientMemory"
Sep 14 02:52:33.796140 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.796071    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-4.ec2.internal" event="NodeHasNoDiskPressure"
Sep 14 02:52:33.796140 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.796084    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-4.ec2.internal" event="NodeHasSufficientPID"
Sep 14 02:52:33.815543 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:33.815523    2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-4.ec2.internal\" not found" node="ip-10-0-141-4.ec2.internal"
Sep 14 02:52:33.820186 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:33.820170    2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-4.ec2.internal\" not found" node="ip-10-0-141-4.ec2.internal"
Sep 14 02:52:33.871848 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:33.871818    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-4.ec2.internal\" not found"
Sep 14 02:52:33.893705 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.893674    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f38ae5f0895b8efd6b111a875262c0a6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-4.ec2.internal\" (UID: \"f38ae5f0895b8efd6b111a875262c0a6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-4.ec2.internal"
Sep 14 02:52:33.893881 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.893704    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f38ae5f0895b8efd6b111a875262c0a6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-4.ec2.internal\" (UID: \"f38ae5f0895b8efd6b111a875262c0a6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-4.ec2.internal"
Sep 14 02:52:33.893881 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.893739    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d52ca59bd77793b4e757e0f76d119156-config\") pod \"kube-apiserver-proxy-ip-10-0-141-4.ec2.internal\" (UID: \"d52ca59bd77793b4e757e0f76d119156\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-4.ec2.internal"
Sep 14 02:52:33.893881 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.893777    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f38ae5f0895b8efd6b111a875262c0a6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-4.ec2.internal\" (UID: \"f38ae5f0895b8efd6b111a875262c0a6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-4.ec2.internal"
Sep 14 02:52:33.893881 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.893784    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f38ae5f0895b8efd6b111a875262c0a6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-4.ec2.internal\" (UID: \"f38ae5f0895b8efd6b111a875262c0a6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-4.ec2.internal"
Sep 14 02:52:33.972828 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:33.972785    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-4.ec2.internal\" not found"
Sep 14 02:52:33.994114 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.994087    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d52ca59bd77793b4e757e0f76d119156-config\") pod \"kube-apiserver-proxy-ip-10-0-141-4.ec2.internal\" (UID: \"d52ca59bd77793b4e757e0f76d119156\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-4.ec2.internal"
Sep 14 02:52:33.994181 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:33.994133    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d52ca59bd77793b4e757e0f76d119156-config\") pod \"kube-apiserver-proxy-ip-10-0-141-4.ec2.internal\" (UID: \"d52ca59bd77793b4e757e0f76d119156\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-4.ec2.internal"
Sep 14 02:52:34.072973 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:34.072916    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-4.ec2.internal\" not found"
Sep 14 02:52:34.118521 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:34.118488    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-4.ec2.internal"
Sep 14 02:52:34.123884 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:34.123284    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-4.ec2.internal"
Sep 14 02:52:34.173870 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:34.173684    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-4.ec2.internal\" not found"
Sep 14 02:52:35.676733 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:35.676707    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-4.ec2.internal\" not found"
Sep 14 02:52:35.777269 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:35.777190    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-4.ec2.internal\" not found"
Sep 14 02:52:35.877682 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:35.877651    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-4.ec2.internal\" not found"
Sep 14 02:52:35.931122 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:35.931095    2564 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 14 02:52:35.931553 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:35.931243    2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 14 02:52:35.978429 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:35.978385    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-4.ec2.internal\" not found"
Sep 14 02:52:36.032612 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:36.032526    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-13 02:47:35 +0000 UTC" deadline="2028-03-10 23:41:34.826343516 +0000 UTC"
Sep 14 02:52:36.032612 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:36.032566    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13052h48m58.793782687s"
Sep 14 02:52:36.058549 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:36.058524    2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 14 02:52:36.094647 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:36.094625    2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-4.ec2.internal"
Sep 14 02:52:36.111171 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:36.111146    2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 14 02:52:36.113083 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:36.113065    2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-4.ec2.internal"
Sep 14 02:52:36.129797 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:36.129780    2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 14 02:52:36.160218 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:36.160199    2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 14 02:52:36.162707 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:36.162686    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf38ae5f0895b8efd6b111a875262c0a6.slice/crio-489fe28920796112c14634a0c43ee0eb46bbf5768fcd5b5d9df9c7429bbed77f WatchSource:0}: Error finding container 489fe28920796112c14634a0c43ee0eb46bbf5768fcd5b5d9df9c7429bbed77f: Status 404 returned error can't find the container with id 489fe28920796112c14634a0c43ee0eb46bbf5768fcd5b5d9df9c7429bbed77f
Sep 14 02:52:36.163515 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:36.163496    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd52ca59bd77793b4e757e0f76d119156.slice/crio-96d513729cb10517af9734a6a1a13df336316e82a0d3219ec8bb24e33396bd42 WatchSource:0}: Error finding container 96d513729cb10517af9734a6a1a13df336316e82a0d3219ec8bb24e33396bd42: Status 404 returned error can't find the container with id 96d513729cb10517af9734a6a1a13df336316e82a0d3219ec8bb24e33396bd42
Sep 14 02:52:36.169544 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:36.169530    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 14 02:52:36.482969 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:36.482884    2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 14 02:52:36.979290 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:36.979257    2564 apiserver.go:52] "Watching apiserver"
Sep 14 02:52:36.986950 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:36.986920    2564 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 14 02:52:36.989468 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:36.989434    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f4hxl","openshift-cluster-node-tuning-operator/tuned-4t7c6","openshift-image-registry/node-ca-whvw4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-4.ec2.internal","openshift-multus/multus-f55dq","openshift-multus/network-metrics-daemon-8lf5z","kube-system/kube-apiserver-proxy-ip-10-0-141-4.ec2.internal","openshift-dns/node-resolver-c57s5","openshift-multus/multus-additional-cni-plugins-qx7v5","openshift-network-diagnostics/network-check-target-kq5nf","openshift-network-operator/iptables-alerter-bm7vz","openshift-ovn-kubernetes/ovnkube-node-qm7jd","kube-system/global-pull-secret-syncer-zhkkf","kube-system/konnectivity-agent-2hr6w"]
Sep 14 02:52:36.991927 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:36.991900    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhkkf"
Sep 14 02:52:36.992045 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:36.991988    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhkkf" podUID="18063d29-58fa-4599-8e9c-93c829fa0fb7"
Sep 14 02:52:36.994244 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:36.994167    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4t7c6"
Sep 14 02:52:36.994457 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:36.994432    2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 14 02:52:36.996893 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:36.996459    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-whvw4"
Sep 14 02:52:36.997233 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:36.997215    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-8sw8k\""
Sep 14 02:52:36.997319 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:36.997257    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 14 02:52:36.997373 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:36.997345    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 14 02:52:36.998692 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:36.998671    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f4hxl"
Sep 14 02:52:36.999355 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:36.999338    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-fbmbl\""
Sep 14 02:52:36.999631 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:36.999596    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 14 02:52:36.999815 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:36.999799    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 14 02:52:36.999877 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:36.999821    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 14 02:52:37.001579 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.001559    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-f55dq"
Sep 14 02:52:37.002963 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.002914    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-n9hw9\""
Sep 14 02:52:37.002963 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.002930    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 14 02:52:37.003518 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.003043    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 14 02:52:37.003518 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.003088    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 14 02:52:37.004166 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.004011    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4813e57c-ad63-4564-8f34-707c6438e999-etc-sysctl-d\") pod \"tuned-4t7c6\" (UID: \"4813e57c-ad63-4564-8f34-707c6438e999\") " pod="openshift-cluster-node-tuning-operator/tuned-4t7c6"
Sep 14 02:52:37.004166 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.003998    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8lf5z"
Sep 14 02:52:37.004166 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.004044    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4813e57c-ad63-4564-8f34-707c6438e999-var-lib-kubelet\") pod \"tuned-4t7c6\" (UID: \"4813e57c-ad63-4564-8f34-707c6438e999\") " pod="openshift-cluster-node-tuning-operator/tuned-4t7c6"
Sep 14 02:52:37.004166 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.004070    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4813e57c-ad63-4564-8f34-707c6438e999-etc-kubernetes\") pod \"tuned-4t7c6\" (UID: \"4813e57c-ad63-4564-8f34-707c6438e999\") " pod="openshift-cluster-node-tuning-operator/tuned-4t7c6"
Sep 14 02:52:37.004166 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.004092    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4813e57c-ad63-4564-8f34-707c6438e999-etc-sysctl-conf\") pod \"tuned-4t7c6\" (UID: \"4813e57c-ad63-4564-8f34-707c6438e999\") " pod="openshift-cluster-node-tuning-operator/tuned-4t7c6"
Sep 14 02:52:37.004166 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:37.004092    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8lf5z" podUID="7029d1df-1b4b-40f6-95e1-621b70070086"
Sep 14 02:52:37.004166 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.004113    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4813e57c-ad63-4564-8f34-707c6438e999-sys\") pod \"tuned-4t7c6\" (UID: \"4813e57c-ad63-4564-8f34-707c6438e999\") " pod="openshift-cluster-node-tuning-operator/tuned-4t7c6"
Sep 14 02:52:37.004166 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.004134    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4813e57c-ad63-4564-8f34-707c6438e999-tmp\") pod \"tuned-4t7c6\" (UID: \"4813e57c-ad63-4564-8f34-707c6438e999\") " pod="openshift-cluster-node-tuning-operator/tuned-4t7c6"
Sep 14 02:52:37.004166 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.004157    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c3ff549c-82fc-4dec-b893-6aa2b2fa1259-host\") pod \"node-ca-whvw4\" (UID: \"c3ff549c-82fc-4dec-b893-6aa2b2fa1259\") " pod="openshift-image-registry/node-ca-whvw4"
Sep 14 02:52:37.004652 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.004181    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c3ff549c-82fc-4dec-b893-6aa2b2fa1259-serviceca\") pod \"node-ca-whvw4\" (UID: \"c3ff549c-82fc-4dec-b893-6aa2b2fa1259\") " pod="openshift-image-registry/node-ca-whvw4"
Sep 14 02:52:37.004652 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.004221    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4813e57c-ad63-4564-8f34-707c6438e999-etc-sysconfig\") pod \"tuned-4t7c6\" (UID: \"4813e57c-ad63-4564-8f34-707c6438e999\") " pod="openshift-cluster-node-tuning-operator/tuned-4t7c6"
Sep 14 02:52:37.004652 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.004295    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4813e57c-ad63-4564-8f34-707c6438e999-lib-modules\") pod \"tuned-4t7c6\" (UID: \"4813e57c-ad63-4564-8f34-707c6438e999\") " pod="openshift-cluster-node-tuning-operator/tuned-4t7c6"
Sep 14 02:52:37.004652 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.004318    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4813e57c-ad63-4564-8f34-707c6438e999-host\") pod \"tuned-4t7c6\" (UID: \"4813e57c-ad63-4564-8f34-707c6438e999\") " pod="openshift-cluster-node-tuning-operator/tuned-4t7c6"
Sep 14 02:52:37.004652 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.004347    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4813e57c-ad63-4564-8f34-707c6438e999-etc-tuned\") pod \"tuned-4t7c6\" (UID: \"4813e57c-ad63-4564-8f34-707c6438e999\") " pod="openshift-cluster-node-tuning-operator/tuned-4t7c6"
Sep 14 02:52:37.004652 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.004394    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpvjc\" (UniqueName: \"kubernetes.io/projected/c3ff549c-82fc-4dec-b893-6aa2b2fa1259-kube-api-access-hpvjc\") pod \"node-ca-whvw4\" (UID: \"c3ff549c-82fc-4dec-b893-6aa2b2fa1259\") " pod="openshift-image-registry/node-ca-whvw4"
Sep 14 02:52:37.004652 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.004440    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/18063d29-58fa-4599-8e9c-93c829fa0fb7-kubelet-config\") pod \"global-pull-secret-syncer-zhkkf\" (UID: \"18063d29-58fa-4599-8e9c-93c829fa0fb7\") " pod="kube-system/global-pull-secret-syncer-zhkkf"
Sep 14 02:52:37.004652 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.004456    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/18063d29-58fa-4599-8e9c-93c829fa0fb7-original-pull-secret\") pod \"global-pull-secret-syncer-zhkkf\" (UID: \"18063d29-58fa-4599-8e9c-93c829fa0fb7\") " pod="kube-system/global-pull-secret-syncer-zhkkf"
Sep 14 02:52:37.004652 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.004476    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4813e57c-ad63-4564-8f34-707c6438e999-etc-systemd\") pod \"tuned-4t7c6\" (UID: \"4813e57c-ad63-4564-8f34-707c6438e999\") " pod="openshift-cluster-node-tuning-operator/tuned-4t7c6"
Sep 14 02:52:37.004652 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.004515    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4813e57c-ad63-4564-8f34-707c6438e999-run\") pod \"tuned-4t7c6\" (UID: \"4813e57c-ad63-4564-8f34-707c6438e999\") " pod="openshift-cluster-node-tuning-operator/tuned-4t7c6"
Sep 14 02:52:37.004652 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.004546    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npw76\" (UniqueName: \"kubernetes.io/projected/4813e57c-ad63-4564-8f34-707c6438e999-kube-api-access-npw76\") pod \"tuned-4t7c6\" (UID: \"4813e57c-ad63-4564-8f34-707c6438e999\") " pod="openshift-cluster-node-tuning-operator/tuned-4t7c6"
Sep 14 02:52:37.004652 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.004592    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/18063d29-58fa-4599-8e9c-93c829fa0fb7-dbus\") pod \"global-pull-secret-syncer-zhkkf\" (UID: \"18063d29-58fa-4599-8e9c-93c829fa0fb7\") " pod="kube-system/global-pull-secret-syncer-zhkkf"
Sep 14 02:52:37.004652 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.004624    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4813e57c-ad63-4564-8f34-707c6438e999-etc-modprobe-d\") pod \"tuned-4t7c6\" (UID: \"4813e57c-ad63-4564-8f34-707c6438e999\") " pod="openshift-cluster-node-tuning-operator/tuned-4t7c6"
Sep 14 02:52:37.006468 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.006398    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2hr6w"
Sep 14 02:52:37.007564 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.007545    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 14 02:52:37.008168 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.008020    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 14 02:52:37.008247 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.008176    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 14 02:52:37.008247 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.008223    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 14 02:52:37.008543 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.008526    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-gchdp\""
Sep 14 02:52:37.008693 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.008675    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-c57s5"
Sep 14 02:52:37.010308 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.010289    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 14 02:52:37.010624 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.010599    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-8kxlt\""
Sep 14 02:52:37.011584 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.011563    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 14 02:52:37.012739 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.012717    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-n8p5b\""
Sep 14 02:52:37.013445 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.013073    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qx7v5"
Sep 14 02:52:37.013651 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.013633    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 14 02:52:37.014041 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.014024    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 14 02:52:37.015787 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.015766    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 14 02:52:37.016308 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.016285    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-qrkf7\""
Sep 14 02:52:37.017107 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.016899    2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 14 02:52:37.017107 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.017079    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 14 02:52:37.017510 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.017491    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kq5nf"
Sep 14 02:52:37.017592 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:37.017562    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kq5nf" podUID="8804d811-79f4-4612-bf40-38c08a04d2c8"
Sep 14 02:52:37.020012 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.019568    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bm7vz"
Sep 14 02:52:37.022081 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.022058    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd"
Sep 14 02:52:37.023463 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.023441    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 14 02:52:37.023553 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.023537    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 14 02:52:37.023668 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.023658    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qll24\""
Sep 14 02:52:37.023770 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.023756    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 14 02:52:37.028805 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.028785    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 14 02:52:37.029355 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.029323    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 14 02:52:37.029355 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.029329    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qxr9b\""
Sep 14 02:52:37.029546 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.029525    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 14 02:52:37.029792 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.029739    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 14 02:52:37.029949 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.029934    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 14 02:52:37.033880 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.033668    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 14 02:52:37.043413 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.043368    2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-cg4ql"
Sep 14 02:52:37.052653 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.052626    2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-cg4ql"
Sep 14 02:52:37.095858 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.095827    2564 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 14 02:52:37.099356 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.099299    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-4.ec2.internal" event={"ID":"f38ae5f0895b8efd6b111a875262c0a6","Type":"ContainerStarted","Data":"489fe28920796112c14634a0c43ee0eb46bbf5768fcd5b5d9df9c7429bbed77f"}
Sep 14 02:52:37.100813 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.100780    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-4.ec2.internal" event={"ID":"d52ca59bd77793b4e757e0f76d119156","Type":"ContainerStarted","Data":"96d513729cb10517af9734a6a1a13df336316e82a0d3219ec8bb24e33396bd42"}
Sep 14 02:52:37.105083 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.105046    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ec25ba7d-9bc2-4f39-a9cf-57894311b7cc-device-dir\") pod \"aws-ebs-csi-driver-node-f4hxl\" (UID: \"ec25ba7d-9bc2-4f39-a9cf-57894311b7cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f4hxl"
Sep 14 02:52:37.105083 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.105086    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/43421977-8a08-4a29-869d-a165accae989-multus-conf-dir\") pod \"multus-f55dq\" (UID: \"43421977-8a08-4a29-869d-a165accae989\") " pod="openshift-multus/multus-f55dq"
Sep 14 02:52:37.105291 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.105109    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/09389d2f-4da9-4e12-91e1-d616caa472d7-agent-certs\") pod \"konnectivity-agent-2hr6w\" (UID: \"09389d2f-4da9-4e12-91e1-d616caa472d7\") " pod="kube-system/konnectivity-agent-2hr6w"
Sep 14 02:52:37.105291 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.105140    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85099e4c-154c-4a21-8e4a-c2ef17f17b25-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qm7jd\" (UID: \"85099e4c-154c-4a21-8e4a-c2ef17f17b25\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd"
Sep 14 02:52:37.105291 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.105167    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/85099e4c-154c-4a21-8e4a-c2ef17f17b25-ovn-node-metrics-cert\") pod \"ovnkube-node-qm7jd\" (UID: \"85099e4c-154c-4a21-8e4a-c2ef17f17b25\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd"
Sep 14 02:52:37.105291 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.105198    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4813e57c-ad63-4564-8f34-707c6438e999-lib-modules\") pod \"tuned-4t7c6\" (UID: \"4813e57c-ad63-4564-8f34-707c6438e999\") " pod="openshift-cluster-node-tuning-operator/tuned-4t7c6"
Sep 14 02:52:37.105291 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.105222    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4813e57c-ad63-4564-8f34-707c6438e999-etc-tuned\") pod \"tuned-4t7c6\" (UID: \"4813e57c-ad63-4564-8f34-707c6438e999\") " pod="openshift-cluster-node-tuning-operator/tuned-4t7c6"
Sep 14 02:52:37.105291 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.105244    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b92669c6-6693-41c9-8d20-b4ac5bffe96f-system-cni-dir\") pod \"multus-additional-cni-plugins-qx7v5\" (UID: \"b92669c6-6693-41c9-8d20-b4ac5bffe96f\") " pod="openshift-multus/multus-additional-cni-plugins-qx7v5"
Sep 14 02:52:37.105291 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.105271    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b92669c6-6693-41c9-8d20-b4ac5bffe96f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qx7v5\" (UID: \"b92669c6-6693-41c9-8d20-b4ac5bffe96f\") " pod="openshift-multus/multus-additional-cni-plugins-qx7v5"
Sep 14 02:52:37.105630 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.105297    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/43421977-8a08-4a29-869d-a165accae989-multus-cni-dir\") pod \"multus-f55dq\" (UID: \"43421977-8a08-4a29-869d-a165accae989\") " pod="openshift-multus/multus-f55dq"
Sep 14 02:52:37.105630 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.105319    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/85099e4c-154c-4a21-8e4a-c2ef17f17b25-ovnkube-config\") pod \"ovnkube-node-qm7jd\" (UID: \"85099e4c-154c-4a21-8e4a-c2ef17f17b25\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd"
Sep 14 02:52:37.105630 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.105343    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/85099e4c-154c-4a21-8e4a-c2ef17f17b25-env-overrides\") pod \"ovnkube-node-qm7jd\" (UID: \"85099e4c-154c-4a21-8e4a-c2ef17f17b25\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd"
Sep 14 02:52:37.105630 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.105366    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/85099e4c-154c-4a21-8e4a-c2ef17f17b25-ovnkube-script-lib\") pod \"ovnkube-node-qm7jd\" (UID: \"85099e4c-154c-4a21-8e4a-c2ef17f17b25\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd"
Sep 14 02:52:37.105630 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.105430    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npw76\" (UniqueName: \"kubernetes.io/projected/4813e57c-ad63-4564-8f34-707c6438e999-kube-api-access-npw76\") pod \"tuned-4t7c6\" (UID: \"4813e57c-ad63-4564-8f34-707c6438e999\") " pod="openshift-cluster-node-tuning-operator/tuned-4t7c6"
Sep 14 02:52:37.105630 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.105484    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b92669c6-6693-41c9-8d20-b4ac5bffe96f-cnibin\") pod \"multus-additional-cni-plugins-qx7v5\" (UID: \"b92669c6-6693-41c9-8d20-b4ac5bffe96f\") " pod="openshift-multus/multus-additional-cni-plugins-qx7v5"
Sep 14 02:52:37.105630 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.105526    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b92669c6-6693-41c9-8d20-b4ac5bffe96f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qx7v5\" (UID: \"b92669c6-6693-41c9-8d20-b4ac5bffe96f\") " pod="openshift-multus/multus-additional-cni-plugins-qx7v5"
Sep 14 02:52:37.105630 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.105563    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fa0ee9a6-d143-4c3d-8d35-306c30cec60a-iptables-alerter-script\") pod \"iptables-alerter-bm7vz\" (UID: \"fa0ee9a6-d143-4c3d-8d35-306c30cec60a\") " pod="openshift-network-operator/iptables-alerter-bm7vz"
Sep 14 02:52:37.105630 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.105590    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/789839ca-54f5-4cab-815c-777c86244553-tmp-dir\") pod \"node-resolver-c57s5\" (UID: \"789839ca-54f5-4cab-815c-777c86244553\") " pod="openshift-dns/node-resolver-c57s5"
Sep 14 02:52:37.105630 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.105614    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dg7c\" (UniqueName: \"kubernetes.io/projected/789839ca-54f5-4cab-815c-777c86244553-kube-api-access-9dg7c\") pod \"node-resolver-c57s5\" (UID: \"789839ca-54f5-4cab-815c-777c86244553\") " pod="openshift-dns/node-resolver-c57s5"
Sep 14 02:52:37.106136 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.105640    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/85099e4c-154c-4a21-8e4a-c2ef17f17b25-etc-openvswitch\") pod \"ovnkube-node-qm7jd\" (UID: \"85099e4c-154c-4a21-8e4a-c2ef17f17b25\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd"
Sep 14 02:52:37.106136 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.105638    2564 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 14 02:52:37.106136 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.105665    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4813e57c-ad63-4564-8f34-707c6438e999-etc-sysctl-conf\") pod \"tuned-4t7c6\" (UID: \"4813e57c-ad63-4564-8f34-707c6438e999\") " pod="openshift-cluster-node-tuning-operator/tuned-4t7c6"
Sep 14 02:52:37.106136 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.105702    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c3ff549c-82fc-4dec-b893-6aa2b2fa1259-host\") pod \"node-ca-whvw4\" (UID: \"c3ff549c-82fc-4dec-b893-6aa2b2fa1259\") " pod="openshift-image-registry/node-ca-whvw4"
Sep 14 02:52:37.106136 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.105728    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c3ff549c-82fc-4dec-b893-6aa2b2fa1259-serviceca\") pod \"node-ca-whvw4\" (UID: \"c3ff549c-82fc-4dec-b893-6aa2b2fa1259\") " pod="openshift-image-registry/node-ca-whvw4"
Sep 14 02:52:37.106136 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.105752    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ec25ba7d-9bc2-4f39-a9cf-57894311b7cc-etc-selinux\") pod \"aws-ebs-csi-driver-node-f4hxl\" (UID: \"ec25ba7d-9bc2-4f39-a9cf-57894311b7cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f4hxl"
Sep 14 02:52:37.106136 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.105841    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ec25ba7d-9bc2-4f39-a9cf-57894311b7cc-sys-fs\") pod \"aws-ebs-csi-driver-node-f4hxl\" (UID: \"ec25ba7d-9bc2-4f39-a9cf-57894311b7cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f4hxl"
Sep 14 02:52:37.106136 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.105872    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/85099e4c-154c-4a21-8e4a-c2ef17f17b25-host-kubelet\") pod \"ovnkube-node-qm7jd\" (UID: \"85099e4c-154c-4a21-8e4a-c2ef17f17b25\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd"
Sep 14 02:52:37.106136 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.105881    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4813e57c-ad63-4564-8f34-707c6438e999-etc-sysctl-conf\") pod \"tuned-4t7c6\" (UID: \"4813e57c-ad63-4564-8f34-707c6438e999\") " pod="openshift-cluster-node-tuning-operator/tuned-4t7c6"
Sep 14 02:52:37.106136 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.105425    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4813e57c-ad63-4564-8f34-707c6438e999-lib-modules\") pod \"tuned-4t7c6\" (UID: \"4813e57c-ad63-4564-8f34-707c6438e999\") " pod="openshift-cluster-node-tuning-operator/tuned-4t7c6"
Sep 14 02:52:37.106136 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.105916    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4813e57c-ad63-4564-8f34-707c6438e999-host\") pod \"tuned-4t7c6\" (UID: \"4813e57c-ad63-4564-8f34-707c6438e999\") " pod="openshift-cluster-node-tuning-operator/tuned-4t7c6"
Sep 14 02:52:37.106136 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.105945    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpvjc\" (UniqueName: \"kubernetes.io/projected/c3ff549c-82fc-4dec-b893-6aa2b2fa1259-kube-api-access-hpvjc\") pod \"node-ca-whvw4\" (UID: \"c3ff549c-82fc-4dec-b893-6aa2b2fa1259\") " pod="openshift-image-registry/node-ca-whvw4"
Sep 14 02:52:37.106136 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.106007    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4813e57c-ad63-4564-8f34-707c6438e999-host\") pod \"tuned-4t7c6\" (UID: \"4813e57c-ad63-4564-8f34-707c6438e999\") " pod="openshift-cluster-node-tuning-operator/tuned-4t7c6"
Sep 14 02:52:37.106136 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.106036    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/43421977-8a08-4a29-869d-a165accae989-cnibin\") pod \"multus-f55dq\" (UID: \"43421977-8a08-4a29-869d-a165accae989\") " pod="openshift-multus/multus-f55dq"
Sep 14 02:52:37.106136 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.106120    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/85099e4c-154c-4a21-8e4a-c2ef17f17b25-run-systemd\") pod \"ovnkube-node-qm7jd\" (UID: \"85099e4c-154c-4a21-8e4a-c2ef17f17b25\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd"
Sep 14 02:52:37.106731 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.106158    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/85099e4c-154c-4a21-8e4a-c2ef17f17b25-run-ovn\") pod \"ovnkube-node-qm7jd\" (UID: \"85099e4c-154c-4a21-8e4a-c2ef17f17b25\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd"
Sep 14 02:52:37.106731 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.106192    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/85099e4c-154c-4a21-8e4a-c2ef17f17b25-host-cni-netd\") pod \"ovnkube-node-qm7jd\" (UID: \"85099e4c-154c-4a21-8e4a-c2ef17f17b25\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd"
Sep 14 02:52:37.106731 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.106245    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/789839ca-54f5-4cab-815c-777c86244553-hosts-file\") pod \"node-resolver-c57s5\" (UID: \"789839ca-54f5-4cab-815c-777c86244553\") " pod="openshift-dns/node-resolver-c57s5"
Sep 14 02:52:37.106731 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.105807    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c3ff549c-82fc-4dec-b893-6aa2b2fa1259-host\") pod \"node-ca-whvw4\" (UID: \"c3ff549c-82fc-4dec-b893-6aa2b2fa1259\") " pod="openshift-image-registry/node-ca-whvw4"
Sep 14 02:52:37.106731 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.106288    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/43421977-8a08-4a29-869d-a165accae989-system-cni-dir\") pod \"multus-f55dq\" (UID: \"43421977-8a08-4a29-869d-a165accae989\") " pod="openshift-multus/multus-f55dq"
Sep 14 02:52:37.106731 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.106315    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c3ff549c-82fc-4dec-b893-6aa2b2fa1259-serviceca\") pod \"node-ca-whvw4\" (UID: \"c3ff549c-82fc-4dec-b893-6aa2b2fa1259\") " pod="openshift-image-registry/node-ca-whvw4"
Sep 14 02:52:37.106731 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.106328    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/09389d2f-4da9-4e12-91e1-d616caa472d7-konnectivity-ca\") pod \"konnectivity-agent-2hr6w\" (UID: \"09389d2f-4da9-4e12-91e1-d616caa472d7\") " pod="kube-system/konnectivity-agent-2hr6w"
Sep 14 02:52:37.106731 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.106367    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/85099e4c-154c-4a21-8e4a-c2ef17f17b25-node-log\") pod \"ovnkube-node-qm7jd\" (UID: \"85099e4c-154c-4a21-8e4a-c2ef17f17b25\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd"
Sep 14 02:52:37.106731 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.106398    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/18063d29-58fa-4599-8e9c-93c829fa0fb7-dbus\") pod \"global-pull-secret-syncer-zhkkf\" (UID: \"18063d29-58fa-4599-8e9c-93c829fa0fb7\") " pod="kube-system/global-pull-secret-syncer-zhkkf"
Sep 14 02:52:37.106731 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.106441    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/43421977-8a08-4a29-869d-a165accae989-host-var-lib-kubelet\") pod \"multus-f55dq\" (UID: \"43421977-8a08-4a29-869d-a165accae989\") " pod="openshift-multus/multus-f55dq"
Sep 14 02:52:37.106731 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.106476    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/85099e4c-154c-4a21-8e4a-c2ef17f17b25-host-run-netns\") pod \"ovnkube-node-qm7jd\" (UID: \"85099e4c-154c-4a21-8e4a-c2ef17f17b25\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd"
Sep 14 02:52:37.106731 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.106535    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4813e57c-ad63-4564-8f34-707c6438e999-etc-kubernetes\") pod \"tuned-4t7c6\" (UID: \"4813e57c-ad63-4564-8f34-707c6438e999\") " pod="openshift-cluster-node-tuning-operator/tuned-4t7c6"
Sep 14 02:52:37.106731 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.106588    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4813e57c-ad63-4564-8f34-707c6438e999-etc-kubernetes\") pod \"tuned-4t7c6\" (UID: \"4813e57c-ad63-4564-8f34-707c6438e999\") " pod="openshift-cluster-node-tuning-operator/tuned-4t7c6"
Sep 14 02:52:37.106731 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.106615    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/18063d29-58fa-4599-8e9c-93c829fa0fb7-dbus\") pod \"global-pull-secret-syncer-zhkkf\" (UID: \"18063d29-58fa-4599-8e9c-93c829fa0fb7\") " pod="kube-system/global-pull-secret-syncer-zhkkf"
Sep 14 02:52:37.106731 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.106626    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4813e57c-ad63-4564-8f34-707c6438e999-tmp\") pod \"tuned-4t7c6\" (UID: \"4813e57c-ad63-4564-8f34-707c6438e999\") " pod="openshift-cluster-node-tuning-operator/tuned-4t7c6"
Sep 14 02:52:37.106731 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.106659    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/43421977-8a08-4a29-869d-a165accae989-cni-binary-copy\") pod \"multus-f55dq\" (UID: \"43421977-8a08-4a29-869d-a165accae989\") " pod="openshift-multus/multus-f55dq"
Sep 14 02:52:37.107432 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.106740    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/85099e4c-154c-4a21-8e4a-c2ef17f17b25-log-socket\") pod \"ovnkube-node-qm7jd\" (UID: \"85099e4c-154c-4a21-8e4a-c2ef17f17b25\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd"
Sep 14 02:52:37.107432 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.106778    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85099e4c-154c-4a21-8e4a-c2ef17f17b25-host-run-ovn-kubernetes\") pod \"ovnkube-node-qm7jd\" (UID: \"85099e4c-154c-4a21-8e4a-c2ef17f17b25\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd"
Sep 14 02:52:37.107432 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.106820    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/85099e4c-154c-4a21-8e4a-c2ef17f17b25-host-cni-bin\") pod \"ovnkube-node-qm7jd\" (UID: \"85099e4c-154c-4a21-8e4a-c2ef17f17b25\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd"
Sep 14 02:52:37.107432 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.106855    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4813e57c-ad63-4564-8f34-707c6438e999-etc-sysconfig\") pod \"tuned-4t7c6\" (UID: \"4813e57c-ad63-4564-8f34-707c6438e999\") " pod="openshift-cluster-node-tuning-operator/tuned-4t7c6"
Sep 14 02:52:37.107432 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.106894    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbdks\" (UniqueName: \"kubernetes.io/projected/7029d1df-1b4b-40f6-95e1-621b70070086-kube-api-access-zbdks\") pod \"network-metrics-daemon-8lf5z\" (UID: \"7029d1df-1b4b-40f6-95e1-621b70070086\") " pod="openshift-multus/network-metrics-daemon-8lf5z"
Sep 14 02:52:37.107432 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.106911    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4813e57c-ad63-4564-8f34-707c6438e999-etc-sysconfig\") pod \"tuned-4t7c6\" (UID: \"4813e57c-ad63-4564-8f34-707c6438e999\") " pod="openshift-cluster-node-tuning-operator/tuned-4t7c6"
Sep 14 02:52:37.107432 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.106935    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm48c\" (UniqueName: \"kubernetes.io/projected/b92669c6-6693-41c9-8d20-b4ac5bffe96f-kube-api-access-cm48c\") pod \"multus-additional-cni-plugins-qx7v5\" (UID: \"b92669c6-6693-41c9-8d20-b4ac5bffe96f\") " pod="openshift-multus/multus-additional-cni-plugins-qx7v5"
Sep 14 02:52:37.107432 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.106962    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/43421977-8a08-4a29-869d-a165accae989-host-run-netns\") pod \"multus-f55dq\" (UID: \"43421977-8a08-4a29-869d-a165accae989\") " pod="openshift-multus/multus-f55dq"
Sep 14 02:52:37.107432 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.106987    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/85099e4c-154c-4a21-8e4a-c2ef17f17b25-systemd-units\") pod \"ovnkube-node-qm7jd\" (UID: \"85099e4c-154c-4a21-8e4a-c2ef17f17b25\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd"
Sep 14 02:52:37.107432 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.107017    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4813e57c-ad63-4564-8f34-707c6438e999-etc-systemd\") pod \"tuned-4t7c6\" (UID: \"4813e57c-ad63-4564-8f34-707c6438e999\") " pod="openshift-cluster-node-tuning-operator/tuned-4t7c6"
Sep 14 02:52:37.107432 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.107052    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ec25ba7d-9bc2-4f39-a9cf-57894311b7cc-registration-dir\") pod \"aws-ebs-csi-driver-node-f4hxl\" (UID: \"ec25ba7d-9bc2-4f39-a9cf-57894311b7cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f4hxl"
Sep 14 02:52:37.107432 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.107083    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4813e57c-ad63-4564-8f34-707c6438e999-etc-systemd\") pod \"tuned-4t7c6\" (UID: \"4813e57c-ad63-4564-8f34-707c6438e999\") " pod="openshift-cluster-node-tuning-operator/tuned-4t7c6"
Sep 14 02:52:37.107432 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.107124    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/43421977-8a08-4a29-869d-a165accae989-multus-socket-dir-parent\") pod \"multus-f55dq\" (UID: \"43421977-8a08-4a29-869d-a165accae989\") " pod="openshift-multus/multus-f55dq"
Sep 14 02:52:37.107432 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.107166    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4813e57c-ad63-4564-8f34-707c6438e999-etc-modprobe-d\") pod \"tuned-4t7c6\" (UID: \"4813e57c-ad63-4564-8f34-707c6438e999\") " pod="openshift-cluster-node-tuning-operator/tuned-4t7c6"
Sep 14 02:52:37.107432 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.107213    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4813e57c-ad63-4564-8f34-707c6438e999-etc-sysctl-d\") pod \"tuned-4t7c6\" (UID: \"4813e57c-ad63-4564-8f34-707c6438e999\") " pod="openshift-cluster-node-tuning-operator/tuned-4t7c6"
Sep 14 02:52:37.107432 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.107295    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/43421977-8a08-4a29-869d-a165accae989-hostroot\") pod \"multus-f55dq\" (UID: \"43421977-8a08-4a29-869d-a165accae989\") " pod="openshift-multus/multus-f55dq"
Sep 14 02:52:37.107432 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.107297    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4813e57c-ad63-4564-8f34-707c6438e999-etc-modprobe-d\") pod \"tuned-4t7c6\" (UID: \"4813e57c-ad63-4564-8f34-707c6438e999\") " pod="openshift-cluster-node-tuning-operator/tuned-4t7c6"
Sep 14 02:52:37.108053 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.107327    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fa0ee9a6-d143-4c3d-8d35-306c30cec60a-host-slash\") pod \"iptables-alerter-bm7vz\" (UID: \"fa0ee9a6-d143-4c3d-8d35-306c30cec60a\") " pod="openshift-network-operator/iptables-alerter-bm7vz"
Sep 14 02:52:37.108053 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.107330    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4813e57c-ad63-4564-8f34-707c6438e999-etc-sysctl-d\") pod \"tuned-4t7c6\" (UID: \"4813e57c-ad63-4564-8f34-707c6438e999\") " pod="openshift-cluster-node-tuning-operator/tuned-4t7c6"
Sep 14 02:52:37.108053 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.107352    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc6kv\" (UniqueName: \"kubernetes.io/projected/85099e4c-154c-4a21-8e4a-c2ef17f17b25-kube-api-access-hc6kv\") pod \"ovnkube-node-qm7jd\" (UID: \"85099e4c-154c-4a21-8e4a-c2ef17f17b25\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd"
Sep 14 02:52:37.108053 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.107381    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b92669c6-6693-41c9-8d20-b4ac5bffe96f-cni-binary-copy\") pod \"multus-additional-cni-plugins-qx7v5\" (UID: \"b92669c6-6693-41c9-8d20-b4ac5bffe96f\") " pod="openshift-multus/multus-additional-cni-plugins-qx7v5"
Sep 14 02:52:37.108053 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.107419    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/43421977-8a08-4a29-869d-a165accae989-etc-kubernetes\") pod \"multus-f55dq\" (UID: \"43421977-8a08-4a29-869d-a165accae989\") " pod="openshift-multus/multus-f55dq"
Sep 14 02:52:37.108053 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.107448    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/85099e4c-154c-4a21-8e4a-c2ef17f17b25-var-lib-openvswitch\") pod \"ovnkube-node-qm7jd\" (UID: \"85099e4c-154c-4a21-8e4a-c2ef17f17b25\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd"
Sep 14 02:52:37.108053 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.107474    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b92669c6-6693-41c9-8d20-b4ac5bffe96f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qx7v5\" (UID: \"b92669c6-6693-41c9-8d20-b4ac5bffe96f\") " pod="openshift-multus/multus-additional-cni-plugins-qx7v5"
Sep 14 02:52:37.108053 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.107899    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/43421977-8a08-4a29-869d-a165accae989-host-run-k8s-cni-cncf-io\") pod \"multus-f55dq\" (UID: \"43421977-8a08-4a29-869d-a165accae989\") " pod="openshift-multus/multus-f55dq"
Sep 14 02:52:37.108053 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.107922    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7dms\" (UniqueName: \"kubernetes.io/projected/43421977-8a08-4a29-869d-a165accae989-kube-api-access-x7dms\") pod \"multus-f55dq\" (UID: \"43421977-8a08-4a29-869d-a165accae989\") " pod="openshift-multus/multus-f55dq"
Sep 14 02:52:37.108053 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.107948    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/18063d29-58fa-4599-8e9c-93c829fa0fb7-kubelet-config\") pod \"global-pull-secret-syncer-zhkkf\" (UID: \"18063d29-58fa-4599-8e9c-93c829fa0fb7\") " pod="kube-system/global-pull-secret-syncer-zhkkf"
Sep 14 02:52:37.108053 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.107973    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/18063d29-58fa-4599-8e9c-93c829fa0fb7-original-pull-secret\") pod \"global-pull-secret-syncer-zhkkf\" (UID: \"18063d29-58fa-4599-8e9c-93c829fa0fb7\") " pod="kube-system/global-pull-secret-syncer-zhkkf"
Sep 14 02:52:37.108053 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.107997    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4813e57c-ad63-4564-8f34-707c6438e999-run\") pod \"tuned-4t7c6\" (UID: \"4813e57c-ad63-4564-8f34-707c6438e999\") " pod="openshift-cluster-node-tuning-operator/tuned-4t7c6"
Sep 14 02:52:37.108053 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.108022    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7029d1df-1b4b-40f6-95e1-621b70070086-metrics-certs\") pod \"network-metrics-daemon-8lf5z\" (UID: \"7029d1df-1b4b-40f6-95e1-621b70070086\") " pod="openshift-multus/network-metrics-daemon-8lf5z"
Sep 14 02:52:37.108053 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.108046    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ec25ba7d-9bc2-4f39-a9cf-57894311b7cc-socket-dir\") pod \"aws-ebs-csi-driver-node-f4hxl\" (UID: \"ec25ba7d-9bc2-4f39-a9cf-57894311b7cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f4hxl"
Sep 14 02:52:37.108595 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.108075    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtwkp\" (UniqueName: \"kubernetes.io/projected/ec25ba7d-9bc2-4f39-a9cf-57894311b7cc-kube-api-access-rtwkp\") pod \"aws-ebs-csi-driver-node-f4hxl\" (UID: \"ec25ba7d-9bc2-4f39-a9cf-57894311b7cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f4hxl"
Sep 14 02:52:37.108595 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.108105    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/43421977-8a08-4a29-869d-a165accae989-host-run-multus-certs\") pod \"multus-f55dq\" (UID: \"43421977-8a08-4a29-869d-a165accae989\") " pod="openshift-multus/multus-f55dq"
Sep 14 02:52:37.108595 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.108126    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn8sw\" (UniqueName: \"kubernetes.io/projected/8804d811-79f4-4612-bf40-38c08a04d2c8-kube-api-access-tn8sw\") pod \"network-check-target-kq5nf\" (UID: \"8804d811-79f4-4612-bf40-38c08a04d2c8\") " pod="openshift-network-diagnostics/network-check-target-kq5nf"
Sep 14 02:52:37.108595 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.108158    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4813e57c-ad63-4564-8f34-707c6438e999-var-lib-kubelet\") pod \"tuned-4t7c6\" (UID: \"4813e57c-ad63-4564-8f34-707c6438e999\") " pod="openshift-cluster-node-tuning-operator/tuned-4t7c6"
Sep 14 02:52:37.108595 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.108179    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ec25ba7d-9bc2-4f39-a9cf-57894311b7cc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f4hxl\" (UID: \"ec25ba7d-9bc2-4f39-a9cf-57894311b7cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f4hxl"
Sep 14 02:52:37.108595 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.108200    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/43421977-8a08-4a29-869d-a165accae989-host-var-lib-cni-bin\") pod \"multus-f55dq\" (UID: \"43421977-8a08-4a29-869d-a165accae989\") " pod="openshift-multus/multus-f55dq"
Sep 14 02:52:37.108595 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.108222    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/43421977-8a08-4a29-869d-a165accae989-host-var-lib-cni-multus\") pod \"multus-f55dq\" (UID: \"43421977-8a08-4a29-869d-a165accae989\") " pod="openshift-multus/multus-f55dq"
Sep 14 02:52:37.108595 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.108246    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/43421977-8a08-4a29-869d-a165accae989-multus-daemon-config\") pod \"multus-f55dq\" (UID: \"43421977-8a08-4a29-869d-a165accae989\") " pod="openshift-multus/multus-f55dq"
Sep 14 02:52:37.108595 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.108269    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfpk4\" (UniqueName: \"kubernetes.io/projected/fa0ee9a6-d143-4c3d-8d35-306c30cec60a-kube-api-access-bfpk4\") pod \"iptables-alerter-bm7vz\" (UID: \"fa0ee9a6-d143-4c3d-8d35-306c30cec60a\") " pod="openshift-network-operator/iptables-alerter-bm7vz"
Sep 14 02:52:37.108595 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.108291    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/85099e4c-154c-4a21-8e4a-c2ef17f17b25-host-slash\") pod \"ovnkube-node-qm7jd\" (UID: \"85099e4c-154c-4a21-8e4a-c2ef17f17b25\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd"
Sep 14 02:52:37.108595 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.108314    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/85099e4c-154c-4a21-8e4a-c2ef17f17b25-run-openvswitch\") pod \"ovnkube-node-qm7jd\" (UID: \"85099e4c-154c-4a21-8e4a-c2ef17f17b25\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd"
Sep 14 02:52:37.108595 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.108336    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4813e57c-ad63-4564-8f34-707c6438e999-sys\") pod \"tuned-4t7c6\" (UID: \"4813e57c-ad63-4564-8f34-707c6438e999\") " pod="openshift-cluster-node-tuning-operator/tuned-4t7c6"
Sep 14 02:52:37.108595 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.108456    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4813e57c-ad63-4564-8f34-707c6438e999-sys\") pod \"tuned-4t7c6\" (UID: \"4813e57c-ad63-4564-8f34-707c6438e999\") " pod="openshift-cluster-node-tuning-operator/tuned-4t7c6"
Sep 14 02:52:37.108595 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.108510    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/18063d29-58fa-4599-8e9c-93c829fa0fb7-kubelet-config\") pod \"global-pull-secret-syncer-zhkkf\" (UID: \"18063d29-58fa-4599-8e9c-93c829fa0fb7\") " pod="kube-system/global-pull-secret-syncer-zhkkf"
Sep 14 02:52:37.108595 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.108533    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4813e57c-ad63-4564-8f34-707c6438e999-run\") pod \"tuned-4t7c6\" (UID: \"4813e57c-ad63-4564-8f34-707c6438e999\") " pod="openshift-cluster-node-tuning-operator/tuned-4t7c6"
Sep 14 02:52:37.108595 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:37.108599    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 14 02:52:37.109292 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.108636    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4813e57c-ad63-4564-8f34-707c6438e999-var-lib-kubelet\") pod \"tuned-4t7c6\" (UID: \"4813e57c-ad63-4564-8f34-707c6438e999\") " pod="openshift-cluster-node-tuning-operator/tuned-4t7c6"
Sep 14 02:52:37.109292 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:37.108691    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18063d29-58fa-4599-8e9c-93c829fa0fb7-original-pull-secret podName:18063d29-58fa-4599-8e9c-93c829fa0fb7 nodeName:}" failed. No retries permitted until 2026-09-14 02:52:37.608651266 +0000 UTC m=+2.989735447 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/18063d29-58fa-4599-8e9c-93c829fa0fb7-original-pull-secret") pod "global-pull-secret-syncer-zhkkf" (UID: "18063d29-58fa-4599-8e9c-93c829fa0fb7") : object "kube-system"/"original-pull-secret" not registered
Sep 14 02:52:37.110135 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.110088    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4813e57c-ad63-4564-8f34-707c6438e999-etc-tuned\") pod \"tuned-4t7c6\" (UID: \"4813e57c-ad63-4564-8f34-707c6438e999\") " pod="openshift-cluster-node-tuning-operator/tuned-4t7c6"
Sep 14 02:52:37.110329 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.110306    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4813e57c-ad63-4564-8f34-707c6438e999-tmp\") pod \"tuned-4t7c6\" (UID: \"4813e57c-ad63-4564-8f34-707c6438e999\") " pod="openshift-cluster-node-tuning-operator/tuned-4t7c6"
Sep 14 02:52:37.127182 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.127142    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-npw76\" (UniqueName: \"kubernetes.io/projected/4813e57c-ad63-4564-8f34-707c6438e999-kube-api-access-npw76\") pod \"tuned-4t7c6\" (UID: \"4813e57c-ad63-4564-8f34-707c6438e999\") " pod="openshift-cluster-node-tuning-operator/tuned-4t7c6"
Sep 14 02:52:37.127745 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.127643    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpvjc\" (UniqueName: \"kubernetes.io/projected/c3ff549c-82fc-4dec-b893-6aa2b2fa1259-kube-api-access-hpvjc\") pod \"node-ca-whvw4\" (UID: \"c3ff549c-82fc-4dec-b893-6aa2b2fa1259\") " pod="openshift-image-registry/node-ca-whvw4"
Sep 14 02:52:37.208960 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.208920    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfpk4\" (UniqueName: \"kubernetes.io/projected/fa0ee9a6-d143-4c3d-8d35-306c30cec60a-kube-api-access-bfpk4\") pod \"iptables-alerter-bm7vz\" (UID: \"fa0ee9a6-d143-4c3d-8d35-306c30cec60a\") " pod="openshift-network-operator/iptables-alerter-bm7vz"
Sep 14 02:52:37.209135 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.208966    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/85099e4c-154c-4a21-8e4a-c2ef17f17b25-host-slash\") pod \"ovnkube-node-qm7jd\" (UID: \"85099e4c-154c-4a21-8e4a-c2ef17f17b25\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd"
Sep 14 02:52:37.209135 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.208992    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/85099e4c-154c-4a21-8e4a-c2ef17f17b25-run-openvswitch\") pod \"ovnkube-node-qm7jd\" (UID: \"85099e4c-154c-4a21-8e4a-c2ef17f17b25\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd"
Sep 14 02:52:37.209135 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.209048    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/85099e4c-154c-4a21-8e4a-c2ef17f17b25-run-openvswitch\") pod \"ovnkube-node-qm7jd\" (UID: \"85099e4c-154c-4a21-8e4a-c2ef17f17b25\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd"
Sep 14 02:52:37.209135 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.209053    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/85099e4c-154c-4a21-8e4a-c2ef17f17b25-host-slash\") pod \"ovnkube-node-qm7jd\" (UID: \"85099e4c-154c-4a21-8e4a-c2ef17f17b25\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd"
Sep 14 02:52:37.209345 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.209170    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ec25ba7d-9bc2-4f39-a9cf-57894311b7cc-device-dir\") pod \"aws-ebs-csi-driver-node-f4hxl\" (UID: \"ec25ba7d-9bc2-4f39-a9cf-57894311b7cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f4hxl"
Sep 14 02:52:37.209345 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.209207    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/43421977-8a08-4a29-869d-a165accae989-multus-conf-dir\") pod \"multus-f55dq\" (UID: \"43421977-8a08-4a29-869d-a165accae989\") " pod="openshift-multus/multus-f55dq"
Sep 14 02:52:37.209345 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.209234    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/09389d2f-4da9-4e12-91e1-d616caa472d7-agent-certs\") pod \"konnectivity-agent-2hr6w\" (UID: \"09389d2f-4da9-4e12-91e1-d616caa472d7\") " pod="kube-system/konnectivity-agent-2hr6w"
Sep 14 02:52:37.209345 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.209260    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85099e4c-154c-4a21-8e4a-c2ef17f17b25-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qm7jd\" (UID: \"85099e4c-154c-4a21-8e4a-c2ef17f17b25\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd"
Sep 14 02:52:37.209345 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.209286    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/85099e4c-154c-4a21-8e4a-c2ef17f17b25-ovn-node-metrics-cert\") pod \"ovnkube-node-qm7jd\" (UID: \"85099e4c-154c-4a21-8e4a-c2ef17f17b25\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd"
Sep 14 02:52:37.209345 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.209310    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b92669c6-6693-41c9-8d20-b4ac5bffe96f-system-cni-dir\") pod \"multus-additional-cni-plugins-qx7v5\" (UID: \"b92669c6-6693-41c9-8d20-b4ac5bffe96f\") " pod="openshift-multus/multus-additional-cni-plugins-qx7v5"
Sep 14 02:52:37.209345 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.209322    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/43421977-8a08-4a29-869d-a165accae989-multus-conf-dir\") pod \"multus-f55dq\" (UID: \"43421977-8a08-4a29-869d-a165accae989\") " pod="openshift-multus/multus-f55dq"
Sep 14 02:52:37.209345 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.209326    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b92669c6-6693-41c9-8d20-b4ac5bffe96f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qx7v5\" (UID: \"b92669c6-6693-41c9-8d20-b4ac5bffe96f\") " pod="openshift-multus/multus-additional-cni-plugins-qx7v5"
Sep 14 02:52:37.209753 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.209371    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/43421977-8a08-4a29-869d-a165accae989-multus-cni-dir\") pod \"multus-f55dq\" (UID: \"43421977-8a08-4a29-869d-a165accae989\") " pod="openshift-multus/multus-f55dq"
Sep 14 02:52:37.209753 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.209398    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/85099e4c-154c-4a21-8e4a-c2ef17f17b25-ovnkube-config\") pod \"ovnkube-node-qm7jd\" (UID: \"85099e4c-154c-4a21-8e4a-c2ef17f17b25\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd"
Sep 14 02:52:37.209753 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.209439    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/85099e4c-154c-4a21-8e4a-c2ef17f17b25-env-overrides\") pod \"ovnkube-node-qm7jd\" (UID: \"85099e4c-154c-4a21-8e4a-c2ef17f17b25\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd"
Sep 14 02:52:37.209753 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.209464    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/85099e4c-154c-4a21-8e4a-c2ef17f17b25-ovnkube-script-lib\") pod \"ovnkube-node-qm7jd\" (UID: \"85099e4c-154c-4a21-8e4a-c2ef17f17b25\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd"
Sep 14 02:52:37.209753 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.209492    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b92669c6-6693-41c9-8d20-b4ac5bffe96f-cnibin\") pod \"multus-additional-cni-plugins-qx7v5\" (UID: \"b92669c6-6693-41c9-8d20-b4ac5bffe96f\") " pod="openshift-multus/multus-additional-cni-plugins-qx7v5"
Sep 14 02:52:37.209753 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.209516    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b92669c6-6693-41c9-8d20-b4ac5bffe96f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qx7v5\" (UID: \"b92669c6-6693-41c9-8d20-b4ac5bffe96f\") " pod="openshift-multus/multus-additional-cni-plugins-qx7v5"
Sep 14 02:52:37.209753 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.209541    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fa0ee9a6-d143-4c3d-8d35-306c30cec60a-iptables-alerter-script\") pod \"iptables-alerter-bm7vz\" (UID: \"fa0ee9a6-d143-4c3d-8d35-306c30cec60a\") " pod="openshift-network-operator/iptables-alerter-bm7vz"
Sep 14 02:52:37.209753 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.209569    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/789839ca-54f5-4cab-815c-777c86244553-tmp-dir\") pod \"node-resolver-c57s5\" (UID: \"789839ca-54f5-4cab-815c-777c86244553\") " pod="openshift-dns/node-resolver-c57s5"
Sep 14 02:52:37.209753 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.209592    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dg7c\" (UniqueName: \"kubernetes.io/projected/789839ca-54f5-4cab-815c-777c86244553-kube-api-access-9dg7c\") pod \"node-resolver-c57s5\" (UID: \"789839ca-54f5-4cab-815c-777c86244553\") " pod="openshift-dns/node-resolver-c57s5"
Sep 14 02:52:37.209753 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.209632    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/85099e4c-154c-4a21-8e4a-c2ef17f17b25-etc-openvswitch\") pod \"ovnkube-node-qm7jd\" (UID: \"85099e4c-154c-4a21-8e4a-c2ef17f17b25\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd"
Sep 14 02:52:37.209753 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.209659    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ec25ba7d-9bc2-4f39-a9cf-57894311b7cc-etc-selinux\") pod \"aws-ebs-csi-driver-node-f4hxl\" (UID: \"ec25ba7d-9bc2-4f39-a9cf-57894311b7cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f4hxl"
Sep 14 02:52:37.209753 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.209740    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/43421977-8a08-4a29-869d-a165accae989-multus-cni-dir\") pod \"multus-f55dq\" (UID: \"43421977-8a08-4a29-869d-a165accae989\") " pod="openshift-multus/multus-f55dq"
Sep 14 02:52:37.210397 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.209774    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ec25ba7d-9bc2-4f39-a9cf-57894311b7cc-etc-selinux\") pod \"aws-ebs-csi-driver-node-f4hxl\" (UID: \"ec25ba7d-9bc2-4f39-a9cf-57894311b7cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f4hxl"
Sep 14 02:52:37.210397 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.209788    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ec25ba7d-9bc2-4f39-a9cf-57894311b7cc-sys-fs\") pod \"aws-ebs-csi-driver-node-f4hxl\" (UID: \"ec25ba7d-9bc2-4f39-a9cf-57894311b7cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f4hxl"
Sep 14 02:52:37.210397 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.209818    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/85099e4c-154c-4a21-8e4a-c2ef17f17b25-host-kubelet\") pod \"ovnkube-node-qm7jd\" (UID: \"85099e4c-154c-4a21-8e4a-c2ef17f17b25\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd"
Sep 14 02:52:37.210397 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.209843    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/43421977-8a08-4a29-869d-a165accae989-cnibin\") pod \"multus-f55dq\" (UID: \"43421977-8a08-4a29-869d-a165accae989\") " pod="openshift-multus/multus-f55dq"
Sep 14 02:52:37.210397 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.209865    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/85099e4c-154c-4a21-8e4a-c2ef17f17b25-run-systemd\") pod \"ovnkube-node-qm7jd\" (UID: \"85099e4c-154c-4a21-8e4a-c2ef17f17b25\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd"
Sep 14 02:52:37.210397 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.209888    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/85099e4c-154c-4a21-8e4a-c2ef17f17b25-run-ovn\") pod \"ovnkube-node-qm7jd\" (UID: \"85099e4c-154c-4a21-8e4a-c2ef17f17b25\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd"
Sep 14 02:52:37.210397 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.209907    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b92669c6-6693-41c9-8d20-b4ac5bffe96f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qx7v5\" (UID: \"b92669c6-6693-41c9-8d20-b4ac5bffe96f\") " pod="openshift-multus/multus-additional-cni-plugins-qx7v5"
Sep 14 02:52:37.210397 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.209907    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b92669c6-6693-41c9-8d20-b4ac5bffe96f-cnibin\") pod \"multus-additional-cni-plugins-qx7v5\" (UID: \"b92669c6-6693-41c9-8d20-b4ac5bffe96f\") " pod="openshift-multus/multus-additional-cni-plugins-qx7v5"
Sep 14 02:52:37.210397 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.209910    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/85099e4c-154c-4a21-8e4a-c2ef17f17b25-host-cni-netd\") pod \"ovnkube-node-qm7jd\" (UID: \"85099e4c-154c-4a21-8e4a-c2ef17f17b25\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd"
Sep 14 02:52:37.210397 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.209943    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/85099e4c-154c-4a21-8e4a-c2ef17f17b25-host-cni-netd\") pod \"ovnkube-node-qm7jd\" (UID: \"85099e4c-154c-4a21-8e4a-c2ef17f17b25\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd"
Sep 14 02:52:37.210397 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.209952    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/789839ca-54f5-4cab-815c-777c86244553-hosts-file\") pod \"node-resolver-c57s5\" (UID: \"789839ca-54f5-4cab-815c-777c86244553\") " pod="openshift-dns/node-resolver-c57s5"
Sep 14 02:52:37.210397 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.209970    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/43421977-8a08-4a29-869d-a165accae989-system-cni-dir\") pod \"multus-f55dq\" (UID: \"43421977-8a08-4a29-869d-a165accae989\") " pod="openshift-multus/multus-f55dq"
Sep 14 02:52:37.210397 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.209985    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/09389d2f-4da9-4e12-91e1-d616caa472d7-konnectivity-ca\") pod \"konnectivity-agent-2hr6w\" (UID: \"09389d2f-4da9-4e12-91e1-d616caa472d7\") " pod="kube-system/konnectivity-agent-2hr6w"
Sep 14 02:52:37.210397 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.210000    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/85099e4c-154c-4a21-8e4a-c2ef17f17b25-node-log\") pod \"ovnkube-node-qm7jd\" (UID: \"85099e4c-154c-4a21-8e4a-c2ef17f17b25\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd"
Sep 14 02:52:37.210397 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.210016    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/43421977-8a08-4a29-869d-a165accae989-host-var-lib-kubelet\") pod \"multus-f55dq\" (UID: \"43421977-8a08-4a29-869d-a165accae989\") " pod="openshift-multus/multus-f55dq"
Sep 14 02:52:37.210397 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.210032    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/85099e4c-154c-4a21-8e4a-c2ef17f17b25-host-run-netns\") pod \"ovnkube-node-qm7jd\" (UID: \"85099e4c-154c-4a21-8e4a-c2ef17f17b25\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd"
Sep 14 02:52:37.210397 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.210048    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/43421977-8a08-4a29-869d-a165accae989-cni-binary-copy\") pod \"multus-f55dq\" (UID: \"43421977-8a08-4a29-869d-a165accae989\") " pod="openshift-multus/multus-f55dq"
Sep 14 02:52:37.210397 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.210070    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/85099e4c-154c-4a21-8e4a-c2ef17f17b25-log-socket\") pod \"ovnkube-node-qm7jd\" (UID: \"85099e4c-154c-4a21-8e4a-c2ef17f17b25\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd"
Sep 14 02:52:37.211265 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.210073    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/789839ca-54f5-4cab-815c-777c86244553-tmp-dir\") pod \"node-resolver-c57s5\" (UID: \"789839ca-54f5-4cab-815c-777c86244553\") " pod="openshift-dns/node-resolver-c57s5"
Sep 14 02:52:37.211265 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.210085    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85099e4c-154c-4a21-8e4a-c2ef17f17b25-host-run-ovn-kubernetes\") pod \"ovnkube-node-qm7jd\" (UID: \"85099e4c-154c-4a21-8e4a-c2ef17f17b25\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd"
Sep 14 02:52:37.211265 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.210123    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/85099e4c-154c-4a21-8e4a-c2ef17f17b25-host-cni-bin\") pod \"ovnkube-node-qm7jd\" (UID: \"85099e4c-154c-4a21-8e4a-c2ef17f17b25\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd"
Sep 14 02:52:37.211265 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.210135    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/85099e4c-154c-4a21-8e4a-c2ef17f17b25-etc-openvswitch\") pod \"ovnkube-node-qm7jd\" (UID: \"85099e4c-154c-4a21-8e4a-c2ef17f17b25\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd"
Sep 14 02:52:37.211265 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.210140    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbdks\" (UniqueName: \"kubernetes.io/projected/7029d1df-1b4b-40f6-95e1-621b70070086-kube-api-access-zbdks\") pod \"network-metrics-daemon-8lf5z\" (UID: \"7029d1df-1b4b-40f6-95e1-621b70070086\") " pod="openshift-multus/network-metrics-daemon-8lf5z"
Sep 14 02:52:37.211265 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.210168    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fa0ee9a6-d143-4c3d-8d35-306c30cec60a-iptables-alerter-script\") pod \"iptables-alerter-bm7vz\" (UID: \"fa0ee9a6-d143-4c3d-8d35-306c30cec60a\") " pod="openshift-network-operator/iptables-alerter-bm7vz"
Sep 14 02:52:37.211265 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.210179    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm48c\" (UniqueName: \"kubernetes.io/projected/b92669c6-6693-41c9-8d20-b4ac5bffe96f-kube-api-access-cm48c\") pod \"multus-additional-cni-plugins-qx7v5\" (UID: \"b92669c6-6693-41c9-8d20-b4ac5bffe96f\") " pod="openshift-multus/multus-additional-cni-plugins-qx7v5"
Sep 14 02:52:37.211265 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.210217    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/43421977-8a08-4a29-869d-a165accae989-host-run-netns\") pod \"multus-f55dq\" (UID: \"43421977-8a08-4a29-869d-a165accae989\") " pod="openshift-multus/multus-f55dq"
Sep 14 02:52:37.211265 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.210250    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/85099e4c-154c-4a21-8e4a-c2ef17f17b25-systemd-units\") pod \"ovnkube-node-qm7jd\" (UID: \"85099e4c-154c-4a21-8e4a-c2ef17f17b25\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd"
Sep 14 02:52:37.211265 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.210274    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ec25ba7d-9bc2-4f39-a9cf-57894311b7cc-registration-dir\") pod \"aws-ebs-csi-driver-node-f4hxl\" (UID: \"ec25ba7d-9bc2-4f39-a9cf-57894311b7cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f4hxl"
Sep 14 02:52:37.211265 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.210299    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/43421977-8a08-4a29-869d-a165accae989-multus-socket-dir-parent\") pod \"multus-f55dq\" (UID: \"43421977-8a08-4a29-869d-a165accae989\") " pod="openshift-multus/multus-f55dq"
Sep 14 02:52:37.211265 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.210335    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/43421977-8a08-4a29-869d-a165accae989-hostroot\") pod \"multus-f55dq\" (UID: \"43421977-8a08-4a29-869d-a165accae989\") " pod="openshift-multus/multus-f55dq"
Sep 14 02:52:37.211265 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.210344    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b92669c6-6693-41c9-8d20-b4ac5bffe96f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qx7v5\" (UID: \"b92669c6-6693-41c9-8d20-b4ac5bffe96f\") " pod="openshift-multus/multus-additional-cni-plugins-qx7v5"
Sep 14 02:52:37.211265 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.210364    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/43421977-8a08-4a29-869d-a165accae989-multus-socket-dir-parent\") pod \"multus-f55dq\" (UID: \"43421977-8a08-4a29-869d-a165accae989\") " pod="openshift-multus/multus-f55dq"
Sep 14 02:52:37.211265 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.209283    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ec25ba7d-9bc2-4f39-a9cf-57894311b7cc-device-dir\") pod \"aws-ebs-csi-driver-node-f4hxl\" (UID: \"ec25ba7d-9bc2-4f39-a9cf-57894311b7cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f4hxl"
Sep 14 02:52:37.211265 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.210377    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fa0ee9a6-d143-4c3d-8d35-306c30cec60a-host-slash\") pod \"iptables-alerter-bm7vz\" (UID: \"fa0ee9a6-d143-4c3d-8d35-306c30cec60a\") " pod="openshift-network-operator/iptables-alerter-bm7vz"
Sep 14 02:52:37.211265 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.210453    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ec25ba7d-9bc2-4f39-a9cf-57894311b7cc-sys-fs\") pod \"aws-ebs-csi-driver-node-f4hxl\" (UID: \"ec25ba7d-9bc2-4f39-a9cf-57894311b7cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f4hxl"
Sep 14 02:52:37.212156 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.210454    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b92669c6-6693-41c9-8d20-b4ac5bffe96f-system-cni-dir\") pod \"multus-additional-cni-plugins-qx7v5\" (UID: \"b92669c6-6693-41c9-8d20-b4ac5bffe96f\") " pod="openshift-multus/multus-additional-cni-plugins-qx7v5"
Sep 14 02:52:37.212156 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.210485    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc6kv\" (UniqueName: \"kubernetes.io/projected/85099e4c-154c-4a21-8e4a-c2ef17f17b25-kube-api-access-hc6kv\") pod \"ovnkube-node-qm7jd\" (UID: \"85099e4c-154c-4a21-8e4a-c2ef17f17b25\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd"
Sep 14 02:52:37.212156 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.210510    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/789839ca-54f5-4cab-815c-777c86244553-hosts-file\") pod \"node-resolver-c57s5\" (UID: \"789839ca-54f5-4cab-815c-777c86244553\") " pod="openshift-dns/node-resolver-c57s5"
Sep 14 02:52:37.212156 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.210512    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b92669c6-6693-41c9-8d20-b4ac5bffe96f-cni-binary-copy\") pod \"multus-additional-cni-plugins-qx7v5\" (UID: \"b92669c6-6693-41c9-8d20-b4ac5bffe96f\") " pod="openshift-multus/multus-additional-cni-plugins-qx7v5"
Sep 14 02:52:37.212156 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.210542    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/43421977-8a08-4a29-869d-a165accae989-etc-kubernetes\") pod \"multus-f55dq\" (UID: \"43421977-8a08-4a29-869d-a165accae989\") " pod="openshift-multus/multus-f55dq"
Sep 14 02:52:37.212156 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.210602    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/85099e4c-154c-4a21-8e4a-c2ef17f17b25-var-lib-openvswitch\") pod \"ovnkube-node-qm7jd\" (UID: \"85099e4c-154c-4a21-8e4a-c2ef17f17b25\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd"
Sep 14 02:52:37.212156 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.210630    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b92669c6-6693-41c9-8d20-b4ac5bffe96f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qx7v5\" (UID: \"b92669c6-6693-41c9-8d20-b4ac5bffe96f\") " pod="openshift-multus/multus-additional-cni-plugins-qx7v5"
Sep 14 02:52:37.212156 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.210657    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/43421977-8a08-4a29-869d-a165accae989-host-run-k8s-cni-cncf-io\") pod \"multus-f55dq\" (UID: \"43421977-8a08-4a29-869d-a165accae989\") " pod="openshift-multus/multus-f55dq"
Sep 14 02:52:37.212156 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.210697    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7dms\" (UniqueName: \"kubernetes.io/projected/43421977-8a08-4a29-869d-a165accae989-kube-api-access-x7dms\") pod \"multus-f55dq\" (UID: \"43421977-8a08-4a29-869d-a165accae989\") " pod="openshift-multus/multus-f55dq"
Sep 14 02:52:37.212156 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.210739    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7029d1df-1b4b-40f6-95e1-621b70070086-metrics-certs\") pod \"network-metrics-daemon-8lf5z\" (UID: \"7029d1df-1b4b-40f6-95e1-621b70070086\") " pod="openshift-multus/network-metrics-daemon-8lf5z"
Sep 14 02:52:37.212156 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.210766    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ec25ba7d-9bc2-4f39-a9cf-57894311b7cc-socket-dir\") pod \"aws-ebs-csi-driver-node-f4hxl\" (UID: \"ec25ba7d-9bc2-4f39-a9cf-57894311b7cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f4hxl"
Sep 14 02:52:37.212156 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.210797    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtwkp\" (UniqueName: \"kubernetes.io/projected/ec25ba7d-9bc2-4f39-a9cf-57894311b7cc-kube-api-access-rtwkp\") pod \"aws-ebs-csi-driver-node-f4hxl\" (UID: \"ec25ba7d-9bc2-4f39-a9cf-57894311b7cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f4hxl"
Sep 14 02:52:37.212156 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.210798    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/85099e4c-154c-4a21-8e4a-c2ef17f17b25-env-overrides\") pod \"ovnkube-node-qm7jd\" (UID: \"85099e4c-154c-4a21-8e4a-c2ef17f17b25\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd"
Sep 14 02:52:37.212156 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.210824    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/43421977-8a08-4a29-869d-a165accae989-host-run-multus-certs\") pod \"multus-f55dq\" (UID: \"43421977-8a08-4a29-869d-a165accae989\") " pod="openshift-multus/multus-f55dq"
Sep 14 02:52:37.212156 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.210849    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn8sw\" (UniqueName: \"kubernetes.io/projected/8804d811-79f4-4612-bf40-38c08a04d2c8-kube-api-access-tn8sw\") pod \"network-check-target-kq5nf\" (UID: \"8804d811-79f4-4612-bf40-38c08a04d2c8\") " pod="openshift-network-diagnostics/network-check-target-kq5nf"
Sep 14 02:52:37.212156 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.210852    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fa0ee9a6-d143-4c3d-8d35-306c30cec60a-host-slash\") pod \"iptables-alerter-bm7vz\" (UID: \"fa0ee9a6-d143-4c3d-8d35-306c30cec60a\") " pod="openshift-network-operator/iptables-alerter-bm7vz"
Sep 14 02:52:37.212156 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.210859    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/85099e4c-154c-4a21-8e4a-c2ef17f17b25-ovnkube-script-lib\") pod \"ovnkube-node-qm7jd\" (UID: \"85099e4c-154c-4a21-8e4a-c2ef17f17b25\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd"
Sep 14 02:52:37.213009 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.210876    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ec25ba7d-9bc2-4f39-a9cf-57894311b7cc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f4hxl\" (UID: \"ec25ba7d-9bc2-4f39-a9cf-57894311b7cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f4hxl"
Sep 14 02:52:37.213009 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.210902    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/43421977-8a08-4a29-869d-a165accae989-host-var-lib-cni-bin\") pod \"multus-f55dq\" (UID: \"43421977-8a08-4a29-869d-a165accae989\") " pod="openshift-multus/multus-f55dq"
Sep 14 02:52:37.213009 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.210928    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/43421977-8a08-4a29-869d-a165accae989-host-var-lib-cni-multus\") pod \"multus-f55dq\" (UID: \"43421977-8a08-4a29-869d-a165accae989\") " pod="openshift-multus/multus-f55dq"
Sep 14 02:52:37.213009 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.210954    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/43421977-8a08-4a29-869d-a165accae989-multus-daemon-config\") pod \"multus-f55dq\" (UID: \"43421977-8a08-4a29-869d-a165accae989\") " pod="openshift-multus/multus-f55dq"
Sep 14 02:52:37.213009 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.210969    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ec25ba7d-9bc2-4f39-a9cf-57894311b7cc-registration-dir\") pod \"aws-ebs-csi-driver-node-f4hxl\" (UID: \"ec25ba7d-9bc2-4f39-a9cf-57894311b7cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f4hxl"
Sep 14 02:52:37.213009 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.211048    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/09389d2f-4da9-4e12-91e1-d616caa472d7-konnectivity-ca\") pod \"konnectivity-agent-2hr6w\" (UID: \"09389d2f-4da9-4e12-91e1-d616caa472d7\") " pod="kube-system/konnectivity-agent-2hr6w"
Sep 14 02:52:37.213009 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.209980    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85099e4c-154c-4a21-8e4a-c2ef17f17b25-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qm7jd\" (UID: \"85099e4c-154c-4a21-8e4a-c2ef17f17b25\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd"
Sep 14 02:52:37.213009 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.211142    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/85099e4c-154c-4a21-8e4a-c2ef17f17b25-node-log\") pod \"ovnkube-node-qm7jd\" (UID: \"85099e4c-154c-4a21-8e4a-c2ef17f17b25\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd"
Sep 14 02:52:37.213009 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.211198    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/43421977-8a08-4a29-869d-a165accae989-host-var-lib-kubelet\") pod \"multus-f55dq\" (UID: \"43421977-8a08-4a29-869d-a165accae989\") " pod="openshift-multus/multus-f55dq"
Sep 14 02:52:37.213009 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.211276    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/85099e4c-154c-4a21-8e4a-c2ef17f17b25-ovnkube-config\") pod \"ovnkube-node-qm7jd\" (UID: \"85099e4c-154c-4a21-8e4a-c2ef17f17b25\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd"
Sep 14 02:52:37.213009 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.211321    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/85099e4c-154c-4a21-8e4a-c2ef17f17b25-host-kubelet\") pod \"ovnkube-node-qm7jd\" (UID: \"85099e4c-154c-4a21-8e4a-c2ef17f17b25\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd"
Sep 14 02:52:37.213009 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.211362    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/85099e4c-154c-4a21-8e4a-c2ef17f17b25-run-systemd\") pod \"ovnkube-node-qm7jd\" (UID: \"85099e4c-154c-4a21-8e4a-c2ef17f17b25\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd"
Sep 14 02:52:37.213009 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.211423    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b92669c6-6693-41c9-8d20-b4ac5bffe96f-cni-binary-copy\") pod \"multus-additional-cni-plugins-qx7v5\" (UID: \"b92669c6-6693-41c9-8d20-b4ac5bffe96f\") " pod="openshift-multus/multus-additional-cni-plugins-qx7v5"
Sep 14 02:52:37.213009 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.211431    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/43421977-8a08-4a29-869d-a165accae989-cnibin\") pod \"multus-f55dq\" (UID: \"43421977-8a08-4a29-869d-a165accae989\") " pod="openshift-multus/multus-f55dq"
Sep 14 02:52:37.213009 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.211431    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ec25ba7d-9bc2-4f39-a9cf-57894311b7cc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f4hxl\" (UID: \"ec25ba7d-9bc2-4f39-a9cf-57894311b7cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f4hxl"
Sep 14 02:52:37.213009 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.211471    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/43421977-8a08-4a29-869d-a165accae989-host-var-lib-cni-bin\") pod \"multus-f55dq\" (UID: \"43421977-8a08-4a29-869d-a165accae989\") " pod="openshift-multus/multus-f55dq"
Sep 14 02:52:37.213009 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.210903    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/43421977-8a08-4a29-869d-a165accae989-host-run-netns\") pod \"multus-f55dq\" (UID: \"43421977-8a08-4a29-869d-a165accae989\") " pod="openshift-multus/multus-f55dq"
Sep 14 02:52:37.213009 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.211513    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/85099e4c-154c-4a21-8e4a-c2ef17f17b25-log-socket\") pod \"ovnkube-node-qm7jd\" (UID: \"85099e4c-154c-4a21-8e4a-c2ef17f17b25\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd"
Sep 14 02:52:37.213859 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.211519    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/43421977-8a08-4a29-869d-a165accae989-host-var-lib-cni-multus\") pod \"multus-f55dq\" (UID: \"43421977-8a08-4a29-869d-a165accae989\") " pod="openshift-multus/multus-f55dq"
Sep 14 02:52:37.213859 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.211522    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/43421977-8a08-4a29-869d-a165accae989-etc-kubernetes\") pod \"multus-f55dq\" (UID: \"43421977-8a08-4a29-869d-a165accae989\") " pod="openshift-multus/multus-f55dq"
Sep 14 02:52:37.213859 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.210930    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/85099e4c-154c-4a21-8e4a-c2ef17f17b25-systemd-units\") pod \"ovnkube-node-qm7jd\" (UID: \"85099e4c-154c-4a21-8e4a-c2ef17f17b25\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd"
Sep 14 02:52:37.213859 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.211550    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/85099e4c-154c-4a21-8e4a-c2ef17f17b25-var-lib-openvswitch\") pod \"ovnkube-node-qm7jd\" (UID: \"85099e4c-154c-4a21-8e4a-c2ef17f17b25\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd"
Sep 14 02:52:37.213859 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.211555    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/85099e4c-154c-4a21-8e4a-c2ef17f17b25-host-cni-bin\") pod \"ovnkube-node-qm7jd\" (UID: \"85099e4c-154c-4a21-8e4a-c2ef17f17b25\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd"
Sep 14 02:52:37.213859 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.211594    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/85099e4c-154c-4a21-8e4a-c2ef17f17b25-run-ovn\") pod \"ovnkube-node-qm7jd\" (UID: \"85099e4c-154c-4a21-8e4a-c2ef17f17b25\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd"
Sep 14 02:52:37.213859 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.211597    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/43421977-8a08-4a29-869d-a165accae989-system-cni-dir\") pod \"multus-f55dq\" (UID: \"43421977-8a08-4a29-869d-a165accae989\") " pod="openshift-multus/multus-f55dq"
Sep 14 02:52:37.213859 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.211630    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/43421977-8a08-4a29-869d-a165accae989-hostroot\") pod \"multus-f55dq\" (UID: \"43421977-8a08-4a29-869d-a165accae989\") " pod="openshift-multus/multus-f55dq"
Sep 14 02:52:37.213859 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.211670    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/43421977-8a08-4a29-869d-a165accae989-host-run-multus-certs\") pod \"multus-f55dq\" (UID: \"43421977-8a08-4a29-869d-a165accae989\") " pod="openshift-multus/multus-f55dq"
Sep 14 02:52:37.213859 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.211720    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ec25ba7d-9bc2-4f39-a9cf-57894311b7cc-socket-dir\") pod \"aws-ebs-csi-driver-node-f4hxl\" (UID: \"ec25ba7d-9bc2-4f39-a9cf-57894311b7cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f4hxl"
Sep 14 02:52:37.213859 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.211729    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/85099e4c-154c-4a21-8e4a-c2ef17f17b25-host-run-netns\") pod \"ovnkube-node-qm7jd\" (UID: \"85099e4c-154c-4a21-8e4a-c2ef17f17b25\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd"
Sep 14 02:52:37.213859 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.211756    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85099e4c-154c-4a21-8e4a-c2ef17f17b25-host-run-ovn-kubernetes\") pod \"ovnkube-node-qm7jd\" (UID: \"85099e4c-154c-4a21-8e4a-c2ef17f17b25\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd"
Sep 14 02:52:37.213859 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:37.211804    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 02:52:37.213859 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.211971    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/43421977-8a08-4a29-869d-a165accae989-host-run-k8s-cni-cncf-io\") pod \"multus-f55dq\" (UID: \"43421977-8a08-4a29-869d-a165accae989\") " pod="openshift-multus/multus-f55dq"
Sep 14 02:52:37.213859 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:37.212005    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7029d1df-1b4b-40f6-95e1-621b70070086-metrics-certs podName:7029d1df-1b4b-40f6-95e1-621b70070086 nodeName:}" failed. No retries permitted until 2026-09-14 02:52:37.711985969 +0000 UTC m=+3.093070165 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7029d1df-1b4b-40f6-95e1-621b70070086-metrics-certs") pod "network-metrics-daemon-8lf5z" (UID: "7029d1df-1b4b-40f6-95e1-621b70070086") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 02:52:37.213859 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.212055    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b92669c6-6693-41c9-8d20-b4ac5bffe96f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qx7v5\" (UID: \"b92669c6-6693-41c9-8d20-b4ac5bffe96f\") " pod="openshift-multus/multus-additional-cni-plugins-qx7v5"
Sep 14 02:52:37.213859 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.212147    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/43421977-8a08-4a29-869d-a165accae989-multus-daemon-config\") pod \"multus-f55dq\" (UID: \"43421977-8a08-4a29-869d-a165accae989\") " pod="openshift-multus/multus-f55dq"
Sep 14 02:52:37.214679 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.212212    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/09389d2f-4da9-4e12-91e1-d616caa472d7-agent-certs\") pod \"konnectivity-agent-2hr6w\" (UID: \"09389d2f-4da9-4e12-91e1-d616caa472d7\") " pod="kube-system/konnectivity-agent-2hr6w"
Sep 14 02:52:37.214679 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.212306    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/43421977-8a08-4a29-869d-a165accae989-cni-binary-copy\") pod \"multus-f55dq\" (UID: \"43421977-8a08-4a29-869d-a165accae989\") " pod="openshift-multus/multus-f55dq"
Sep 14 02:52:37.214679 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.213235    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/85099e4c-154c-4a21-8e4a-c2ef17f17b25-ovn-node-metrics-cert\") pod \"ovnkube-node-qm7jd\" (UID: \"85099e4c-154c-4a21-8e4a-c2ef17f17b25\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd"
Sep 14 02:52:37.219632 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.219607    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfpk4\" (UniqueName: \"kubernetes.io/projected/fa0ee9a6-d143-4c3d-8d35-306c30cec60a-kube-api-access-bfpk4\") pod \"iptables-alerter-bm7vz\" (UID: \"fa0ee9a6-d143-4c3d-8d35-306c30cec60a\") " pod="openshift-network-operator/iptables-alerter-bm7vz"
Sep 14 02:52:37.221908 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:37.221878    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 14 02:52:37.221908 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:37.221906    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 14 02:52:37.222139 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:37.221921    2564 projected.go:196] Error preparing data for projected volume kube-api-access-tn8sw for pod openshift-network-diagnostics/network-check-target-kq5nf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 02:52:37.222139 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:37.221995    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8804d811-79f4-4612-bf40-38c08a04d2c8-kube-api-access-tn8sw podName:8804d811-79f4-4612-bf40-38c08a04d2c8 nodeName:}" failed. No retries permitted until 2026-09-14 02:52:37.72197767 +0000 UTC m=+3.103061847 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-tn8sw" (UniqueName: "kubernetes.io/projected/8804d811-79f4-4612-bf40-38c08a04d2c8-kube-api-access-tn8sw") pod "network-check-target-kq5nf" (UID: "8804d811-79f4-4612-bf40-38c08a04d2c8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 02:52:37.224772 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.224743    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dg7c\" (UniqueName: \"kubernetes.io/projected/789839ca-54f5-4cab-815c-777c86244553-kube-api-access-9dg7c\") pod \"node-resolver-c57s5\" (UID: \"789839ca-54f5-4cab-815c-777c86244553\") " pod="openshift-dns/node-resolver-c57s5"
Sep 14 02:52:37.224886 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.224811    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7dms\" (UniqueName: \"kubernetes.io/projected/43421977-8a08-4a29-869d-a165accae989-kube-api-access-x7dms\") pod \"multus-f55dq\" (UID: \"43421977-8a08-4a29-869d-a165accae989\") " pod="openshift-multus/multus-f55dq"
Sep 14 02:52:37.224886 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.224852    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm48c\" (UniqueName: \"kubernetes.io/projected/b92669c6-6693-41c9-8d20-b4ac5bffe96f-kube-api-access-cm48c\") pod \"multus-additional-cni-plugins-qx7v5\" (UID: \"b92669c6-6693-41c9-8d20-b4ac5bffe96f\") " pod="openshift-multus/multus-additional-cni-plugins-qx7v5"
Sep 14 02:52:37.225470 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.225447    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtwkp\" (UniqueName: \"kubernetes.io/projected/ec25ba7d-9bc2-4f39-a9cf-57894311b7cc-kube-api-access-rtwkp\") pod \"aws-ebs-csi-driver-node-f4hxl\" (UID: \"ec25ba7d-9bc2-4f39-a9cf-57894311b7cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f4hxl"
Sep 14 02:52:37.225767 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.225745    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbdks\" (UniqueName: \"kubernetes.io/projected/7029d1df-1b4b-40f6-95e1-621b70070086-kube-api-access-zbdks\") pod \"network-metrics-daemon-8lf5z\" (UID: \"7029d1df-1b4b-40f6-95e1-621b70070086\") " pod="openshift-multus/network-metrics-daemon-8lf5z"
Sep 14 02:52:37.226141 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.226119    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc6kv\" (UniqueName: \"kubernetes.io/projected/85099e4c-154c-4a21-8e4a-c2ef17f17b25-kube-api-access-hc6kv\") pod \"ovnkube-node-qm7jd\" (UID: \"85099e4c-154c-4a21-8e4a-c2ef17f17b25\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd"
Sep 14 02:52:37.313608 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.313574    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4t7c6"
Sep 14 02:52:37.323056 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.323027    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f4hxl"
Sep 14 02:52:37.332065 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.332029    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-whvw4"
Sep 14 02:52:37.337960 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.337934    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-f55dq"
Sep 14 02:52:37.338236 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.338210    2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 14 02:52:37.345862 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.345829    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2hr6w"
Sep 14 02:52:37.352804 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.352767    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-c57s5"
Sep 14 02:52:37.359825 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.359794    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qx7v5"
Sep 14 02:52:37.365725 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.365694    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bm7vz"
Sep 14 02:52:37.371239 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.371221    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd"
Sep 14 02:52:37.614828 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.614739    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/18063d29-58fa-4599-8e9c-93c829fa0fb7-original-pull-secret\") pod \"global-pull-secret-syncer-zhkkf\" (UID: \"18063d29-58fa-4599-8e9c-93c829fa0fb7\") " pod="kube-system/global-pull-secret-syncer-zhkkf"
Sep 14 02:52:37.614998 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:37.614911    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 14 02:52:37.614998 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:37.614986    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18063d29-58fa-4599-8e9c-93c829fa0fb7-original-pull-secret podName:18063d29-58fa-4599-8e9c-93c829fa0fb7 nodeName:}" failed. No retries permitted until 2026-09-14 02:52:38.61496394 +0000 UTC m=+3.996048121 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/18063d29-58fa-4599-8e9c-93c829fa0fb7-original-pull-secret") pod "global-pull-secret-syncer-zhkkf" (UID: "18063d29-58fa-4599-8e9c-93c829fa0fb7") : object "kube-system"/"original-pull-secret" not registered
Sep 14 02:52:37.715842 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.715811    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7029d1df-1b4b-40f6-95e1-621b70070086-metrics-certs\") pod \"network-metrics-daemon-8lf5z\" (UID: \"7029d1df-1b4b-40f6-95e1-621b70070086\") " pod="openshift-multus/network-metrics-daemon-8lf5z"
Sep 14 02:52:37.716003 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:37.715932    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 02:52:37.716003 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:37.715994    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7029d1df-1b4b-40f6-95e1-621b70070086-metrics-certs podName:7029d1df-1b4b-40f6-95e1-621b70070086 nodeName:}" failed. No retries permitted until 2026-09-14 02:52:38.715978836 +0000 UTC m=+4.097063026 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7029d1df-1b4b-40f6-95e1-621b70070086-metrics-certs") pod "network-metrics-daemon-8lf5z" (UID: "7029d1df-1b4b-40f6-95e1-621b70070086") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 02:52:37.816824 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:37.816782    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn8sw\" (UniqueName: \"kubernetes.io/projected/8804d811-79f4-4612-bf40-38c08a04d2c8-kube-api-access-tn8sw\") pod \"network-check-target-kq5nf\" (UID: \"8804d811-79f4-4612-bf40-38c08a04d2c8\") " pod="openshift-network-diagnostics/network-check-target-kq5nf"
Sep 14 02:52:37.816995 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:37.816964    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 14 02:52:37.816995 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:37.816983    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 14 02:52:37.816995 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:37.816993    2564 projected.go:196] Error preparing data for projected volume kube-api-access-tn8sw for pod openshift-network-diagnostics/network-check-target-kq5nf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 02:52:37.817126 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:37.817058    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8804d811-79f4-4612-bf40-38c08a04d2c8-kube-api-access-tn8sw podName:8804d811-79f4-4612-bf40-38c08a04d2c8 nodeName:}" failed. No retries permitted until 2026-09-14 02:52:38.817038816 +0000 UTC m=+4.198123012 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-tn8sw" (UniqueName: "kubernetes.io/projected/8804d811-79f4-4612-bf40-38c08a04d2c8-kube-api-access-tn8sw") pod "network-check-target-kq5nf" (UID: "8804d811-79f4-4612-bf40-38c08a04d2c8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 02:52:38.047273 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:38.047176    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec25ba7d_9bc2_4f39_a9cf_57894311b7cc.slice/crio-99931da2eb19c40222848029b84babbc7484ba7a257e2bcf3a57b65b35dc6772 WatchSource:0}: Error finding container 99931da2eb19c40222848029b84babbc7484ba7a257e2bcf3a57b65b35dc6772: Status 404 returned error can't find the container with id 99931da2eb19c40222848029b84babbc7484ba7a257e2bcf3a57b65b35dc6772
Sep 14 02:52:38.050259 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:38.049979    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09389d2f_4da9_4e12_91e1_d616caa472d7.slice/crio-32a854ccfecab72a773ed0a71b457397837d4252ac2d4bd9546b550a8ad9b468 WatchSource:0}: Error finding container 32a854ccfecab72a773ed0a71b457397837d4252ac2d4bd9546b550a8ad9b468: Status 404 returned error can't find the container with id 32a854ccfecab72a773ed0a71b457397837d4252ac2d4bd9546b550a8ad9b468
Sep 14 02:52:38.054623 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:38.054594    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod789839ca_54f5_4cab_815c_777c86244553.slice/crio-54f4e89481fa0c820a1d80eefd20a60dabdfaba1210e319dab32f4bc915cf2c6 WatchSource:0}: Error finding container 54f4e89481fa0c820a1d80eefd20a60dabdfaba1210e319dab32f4bc915cf2c6: Status 404 returned error can't find the container with id 54f4e89481fa0c820a1d80eefd20a60dabdfaba1210e319dab32f4bc915cf2c6
Sep 14 02:52:38.055600 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:38.055544    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-13 02:47:37 +0000 UTC" deadline="2028-06-19 17:29:44.57984227 +0000 UTC"
Sep 14 02:52:38.055698 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:38.055600    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15470h37m6.524246456s"
Sep 14 02:52:38.055698 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:38.055688    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa0ee9a6_d143_4c3d_8d35_306c30cec60a.slice/crio-9e6fcfaac4f6edc973e7d88f1a835c7089ac3d8d3bd84cae6698d4af62ba3cd4 WatchSource:0}: Error finding container 9e6fcfaac4f6edc973e7d88f1a835c7089ac3d8d3bd84cae6698d4af62ba3cd4: Status 404 returned error can't find the container with id 9e6fcfaac4f6edc973e7d88f1a835c7089ac3d8d3bd84cae6698d4af62ba3cd4
Sep 14 02:52:38.056471 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:38.056453    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85099e4c_154c_4a21_8e4a_c2ef17f17b25.slice/crio-94a4d4a865affe26b797563fbb9e659d76a837b753dd1653f6e5bcb2c974c7e3 WatchSource:0}: Error finding container 94a4d4a865affe26b797563fbb9e659d76a837b753dd1653f6e5bcb2c974c7e3: Status 404 returned error can't find the container with id 94a4d4a865affe26b797563fbb9e659d76a837b753dd1653f6e5bcb2c974c7e3
Sep 14 02:52:38.079078 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:38.079052    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4813e57c_ad63_4564_8f34_707c6438e999.slice/crio-574daae7f1c76304d7a5fc7b374582cb927ef72bb8d1fcfdb555995496f648f6 WatchSource:0}: Error finding container 574daae7f1c76304d7a5fc7b374582cb927ef72bb8d1fcfdb555995496f648f6: Status 404 returned error can't find the container with id 574daae7f1c76304d7a5fc7b374582cb927ef72bb8d1fcfdb555995496f648f6
Sep 14 02:52:38.079776 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:38.079756    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43421977_8a08_4a29_869d_a165accae989.slice/crio-31102965bf2e654e84152cd6372fb6ce5f6cd49f4beddcb4786ba12f1487fc36 WatchSource:0}: Error finding container 31102965bf2e654e84152cd6372fb6ce5f6cd49f4beddcb4786ba12f1487fc36: Status 404 returned error can't find the container with id 31102965bf2e654e84152cd6372fb6ce5f6cd49f4beddcb4786ba12f1487fc36
Sep 14 02:52:38.080589 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:52:38.080556    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb92669c6_6693_41c9_8d20_b4ac5bffe96f.slice/crio-782f83b2d70b059ca4eff7f4fbdb3c59a10ccf79b71ec39ef4d9223dbefc99d5 WatchSource:0}: Error finding container 782f83b2d70b059ca4eff7f4fbdb3c59a10ccf79b71ec39ef4d9223dbefc99d5: Status 404 returned error can't find the container with id 782f83b2d70b059ca4eff7f4fbdb3c59a10ccf79b71ec39ef4d9223dbefc99d5
Sep 14 02:52:38.091768 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:38.091745    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhkkf"
Sep 14 02:52:38.091860 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:38.091837    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhkkf" podUID="18063d29-58fa-4599-8e9c-93c829fa0fb7"
Sep 14 02:52:38.106452 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:38.106420    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd" event={"ID":"85099e4c-154c-4a21-8e4a-c2ef17f17b25","Type":"ContainerStarted","Data":"94a4d4a865affe26b797563fbb9e659d76a837b753dd1653f6e5bcb2c974c7e3"}
Sep 14 02:52:38.110074 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:38.109992    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qx7v5" event={"ID":"b92669c6-6693-41c9-8d20-b4ac5bffe96f","Type":"ContainerStarted","Data":"782f83b2d70b059ca4eff7f4fbdb3c59a10ccf79b71ec39ef4d9223dbefc99d5"}
Sep 14 02:52:38.112258 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:38.112233    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-f55dq" event={"ID":"43421977-8a08-4a29-869d-a165accae989","Type":"ContainerStarted","Data":"31102965bf2e654e84152cd6372fb6ce5f6cd49f4beddcb4786ba12f1487fc36"}
Sep 14 02:52:38.113946 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:38.113920    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4t7c6" event={"ID":"4813e57c-ad63-4564-8f34-707c6438e999","Type":"ContainerStarted","Data":"574daae7f1c76304d7a5fc7b374582cb927ef72bb8d1fcfdb555995496f648f6"}
Sep 14 02:52:38.115224 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:38.115189    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-whvw4" event={"ID":"c3ff549c-82fc-4dec-b893-6aa2b2fa1259","Type":"ContainerStarted","Data":"f51c1c47000e65998a82e0644b547d3484966d87b551fdb8c3ad624b0369d8cf"}
Sep 14 02:52:38.117834 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:38.117811    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bm7vz" event={"ID":"fa0ee9a6-d143-4c3d-8d35-306c30cec60a","Type":"ContainerStarted","Data":"9e6fcfaac4f6edc973e7d88f1a835c7089ac3d8d3bd84cae6698d4af62ba3cd4"}
Sep 14 02:52:38.121267 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:38.121246    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2hr6w" event={"ID":"09389d2f-4da9-4e12-91e1-d616caa472d7","Type":"ContainerStarted","Data":"32a854ccfecab72a773ed0a71b457397837d4252ac2d4bd9546b550a8ad9b468"}
Sep 14 02:52:38.122574 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:38.122523    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-c57s5" event={"ID":"789839ca-54f5-4cab-815c-777c86244553","Type":"ContainerStarted","Data":"54f4e89481fa0c820a1d80eefd20a60dabdfaba1210e319dab32f4bc915cf2c6"}
Sep 14 02:52:38.123845 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:38.123818    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f4hxl" event={"ID":"ec25ba7d-9bc2-4f39-a9cf-57894311b7cc","Type":"ContainerStarted","Data":"99931da2eb19c40222848029b84babbc7484ba7a257e2bcf3a57b65b35dc6772"}
Sep 14 02:52:38.623519 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:38.623166    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/18063d29-58fa-4599-8e9c-93c829fa0fb7-original-pull-secret\") pod \"global-pull-secret-syncer-zhkkf\" (UID: \"18063d29-58fa-4599-8e9c-93c829fa0fb7\") " pod="kube-system/global-pull-secret-syncer-zhkkf"
Sep 14 02:52:38.623519 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:38.623437    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 14 02:52:38.623519 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:38.623515    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18063d29-58fa-4599-8e9c-93c829fa0fb7-original-pull-secret podName:18063d29-58fa-4599-8e9c-93c829fa0fb7 nodeName:}" failed. No retries permitted until 2026-09-14 02:52:40.623495552 +0000 UTC m=+6.004579732 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/18063d29-58fa-4599-8e9c-93c829fa0fb7-original-pull-secret") pod "global-pull-secret-syncer-zhkkf" (UID: "18063d29-58fa-4599-8e9c-93c829fa0fb7") : object "kube-system"/"original-pull-secret" not registered
Sep 14 02:52:38.724126 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:38.724084    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7029d1df-1b4b-40f6-95e1-621b70070086-metrics-certs\") pod \"network-metrics-daemon-8lf5z\" (UID: \"7029d1df-1b4b-40f6-95e1-621b70070086\") " pod="openshift-multus/network-metrics-daemon-8lf5z"
Sep 14 02:52:38.724318 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:38.724295    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 02:52:38.724389 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:38.724375    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7029d1df-1b4b-40f6-95e1-621b70070086-metrics-certs podName:7029d1df-1b4b-40f6-95e1-621b70070086 nodeName:}" failed. No retries permitted until 2026-09-14 02:52:40.724354828 +0000 UTC m=+6.105439009 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7029d1df-1b4b-40f6-95e1-621b70070086-metrics-certs") pod "network-metrics-daemon-8lf5z" (UID: "7029d1df-1b4b-40f6-95e1-621b70070086") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 02:52:38.825313 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:38.825277    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn8sw\" (UniqueName: \"kubernetes.io/projected/8804d811-79f4-4612-bf40-38c08a04d2c8-kube-api-access-tn8sw\") pod \"network-check-target-kq5nf\" (UID: \"8804d811-79f4-4612-bf40-38c08a04d2c8\") " pod="openshift-network-diagnostics/network-check-target-kq5nf"
Sep 14 02:52:38.825602 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:38.825583    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 14 02:52:38.825665 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:38.825611    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 14 02:52:38.825665 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:38.825623    2564 projected.go:196] Error preparing data for projected volume kube-api-access-tn8sw for pod openshift-network-diagnostics/network-check-target-kq5nf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 02:52:38.825759 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:38.825687    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8804d811-79f4-4612-bf40-38c08a04d2c8-kube-api-access-tn8sw podName:8804d811-79f4-4612-bf40-38c08a04d2c8 nodeName:}" failed. No retries permitted until 2026-09-14 02:52:40.825667394 +0000 UTC m=+6.206751587 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-tn8sw" (UniqueName: "kubernetes.io/projected/8804d811-79f4-4612-bf40-38c08a04d2c8-kube-api-access-tn8sw") pod "network-check-target-kq5nf" (UID: "8804d811-79f4-4612-bf40-38c08a04d2c8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 02:52:39.056786 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:39.056740    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-13 02:47:37 +0000 UTC" deadline="2028-06-28 07:35:23.439587322 +0000 UTC"
Sep 14 02:52:39.056786 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:39.056783    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15676h42m44.382808726s"
Sep 14 02:52:39.092220 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:39.092183    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kq5nf"
Sep 14 02:52:39.092380 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:39.092311    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kq5nf" podUID="8804d811-79f4-4612-bf40-38c08a04d2c8"
Sep 14 02:52:39.094917 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:39.094892    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8lf5z"
Sep 14 02:52:39.095052 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:39.095019    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8lf5z" podUID="7029d1df-1b4b-40f6-95e1-621b70070086"
Sep 14 02:52:39.136185 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:39.136147    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-4.ec2.internal" event={"ID":"d52ca59bd77793b4e757e0f76d119156","Type":"ContainerStarted","Data":"d20c5dae93175b247cc08330ecaff5808dbbb6b185ad7efd8e24763e7cfd9535"}
Sep 14 02:52:39.155137 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:39.155100    2564 generic.go:356] "Generic (PLEG): container finished" podID="f38ae5f0895b8efd6b111a875262c0a6" containerID="309e57048090874eef522596ce969567ff4418b67f19d6c49d0d5ebe6c1310a9" exitCode=0
Sep 14 02:52:39.155326 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:39.155152    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-4.ec2.internal" event={"ID":"f38ae5f0895b8efd6b111a875262c0a6","Type":"ContainerDied","Data":"309e57048090874eef522596ce969567ff4418b67f19d6c49d0d5ebe6c1310a9"}
Sep 14 02:52:39.171098 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:39.171043    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-4.ec2.internal" podStartSLOduration=3.171020692 podStartE2EDuration="3.171020692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 02:52:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:52:39.152390272 +0000 UTC m=+4.533474472" watchObservedRunningTime="2026-09-14 02:52:39.171020692 +0000 UTC m=+4.552104893"
Sep 14 02:52:40.092588 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:40.092552    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhkkf"
Sep 14 02:52:40.093041 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:40.092689    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhkkf" podUID="18063d29-58fa-4599-8e9c-93c829fa0fb7"
Sep 14 02:52:40.167550 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:40.167464    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-4.ec2.internal" event={"ID":"f38ae5f0895b8efd6b111a875262c0a6","Type":"ContainerStarted","Data":"a247c2e50a255f6924912822aec7875b41f916bb255d7ad2215f0ee08794aaf4"}
Sep 14 02:52:40.187765 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:40.187708    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-4.ec2.internal" podStartSLOduration=4.187687099 podStartE2EDuration="4.187687099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 02:52:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:52:40.185787173 +0000 UTC m=+5.566871377" watchObservedRunningTime="2026-09-14 02:52:40.187687099 +0000 UTC m=+5.568771303"
Sep 14 02:52:40.643743 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:40.643699    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/18063d29-58fa-4599-8e9c-93c829fa0fb7-original-pull-secret\") pod \"global-pull-secret-syncer-zhkkf\" (UID: \"18063d29-58fa-4599-8e9c-93c829fa0fb7\") " pod="kube-system/global-pull-secret-syncer-zhkkf"
Sep 14 02:52:40.643940 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:40.643921    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 14 02:52:40.644023 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:40.643986    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18063d29-58fa-4599-8e9c-93c829fa0fb7-original-pull-secret podName:18063d29-58fa-4599-8e9c-93c829fa0fb7 nodeName:}" failed. No retries permitted until 2026-09-14 02:52:44.643969161 +0000 UTC m=+10.025053345 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/18063d29-58fa-4599-8e9c-93c829fa0fb7-original-pull-secret") pod "global-pull-secret-syncer-zhkkf" (UID: "18063d29-58fa-4599-8e9c-93c829fa0fb7") : object "kube-system"/"original-pull-secret" not registered
Sep 14 02:52:40.744583 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:40.744541    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7029d1df-1b4b-40f6-95e1-621b70070086-metrics-certs\") pod \"network-metrics-daemon-8lf5z\" (UID: \"7029d1df-1b4b-40f6-95e1-621b70070086\") " pod="openshift-multus/network-metrics-daemon-8lf5z"
Sep 14 02:52:40.744753 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:40.744711    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 02:52:40.744812 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:40.744777    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7029d1df-1b4b-40f6-95e1-621b70070086-metrics-certs podName:7029d1df-1b4b-40f6-95e1-621b70070086 nodeName:}" failed. No retries permitted until 2026-09-14 02:52:44.744757368 +0000 UTC m=+10.125841573 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7029d1df-1b4b-40f6-95e1-621b70070086-metrics-certs") pod "network-metrics-daemon-8lf5z" (UID: "7029d1df-1b4b-40f6-95e1-621b70070086") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 02:52:40.845200 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:40.845161    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn8sw\" (UniqueName: \"kubernetes.io/projected/8804d811-79f4-4612-bf40-38c08a04d2c8-kube-api-access-tn8sw\") pod \"network-check-target-kq5nf\" (UID: \"8804d811-79f4-4612-bf40-38c08a04d2c8\") " pod="openshift-network-diagnostics/network-check-target-kq5nf"
Sep 14 02:52:40.845373 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:40.845323    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 14 02:52:40.845373 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:40.845343    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 14 02:52:40.845373 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:40.845356    2564 projected.go:196] Error preparing data for projected volume kube-api-access-tn8sw for pod openshift-network-diagnostics/network-check-target-kq5nf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 02:52:40.845572 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:40.845429    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8804d811-79f4-4612-bf40-38c08a04d2c8-kube-api-access-tn8sw podName:8804d811-79f4-4612-bf40-38c08a04d2c8 nodeName:}" failed. No retries permitted until 2026-09-14 02:52:44.845397109 +0000 UTC m=+10.226481288 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-tn8sw" (UniqueName: "kubernetes.io/projected/8804d811-79f4-4612-bf40-38c08a04d2c8-kube-api-access-tn8sw") pod "network-check-target-kq5nf" (UID: "8804d811-79f4-4612-bf40-38c08a04d2c8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 02:52:41.092810 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:41.092558    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8lf5z"
Sep 14 02:52:41.092810 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:41.092724    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8lf5z" podUID="7029d1df-1b4b-40f6-95e1-621b70070086"
Sep 14 02:52:41.092810 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:41.092749    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kq5nf"
Sep 14 02:52:41.093378 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:41.092870    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kq5nf" podUID="8804d811-79f4-4612-bf40-38c08a04d2c8"
Sep 14 02:52:42.092508 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:42.092476    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhkkf"
Sep 14 02:52:42.092691 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:42.092603    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhkkf" podUID="18063d29-58fa-4599-8e9c-93c829fa0fb7"
Sep 14 02:52:43.092461 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:43.092426    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kq5nf"
Sep 14 02:52:43.092947 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:43.092578    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8lf5z"
Sep 14 02:52:43.092947 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:43.092666    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kq5nf" podUID="8804d811-79f4-4612-bf40-38c08a04d2c8"
Sep 14 02:52:43.092947 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:43.092736    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8lf5z" podUID="7029d1df-1b4b-40f6-95e1-621b70070086"
Sep 14 02:52:44.092469 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:44.092426    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhkkf"
Sep 14 02:52:44.092943 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:44.092552    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhkkf" podUID="18063d29-58fa-4599-8e9c-93c829fa0fb7"
Sep 14 02:52:44.678644 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:44.678551    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/18063d29-58fa-4599-8e9c-93c829fa0fb7-original-pull-secret\") pod \"global-pull-secret-syncer-zhkkf\" (UID: \"18063d29-58fa-4599-8e9c-93c829fa0fb7\") " pod="kube-system/global-pull-secret-syncer-zhkkf"
Sep 14 02:52:44.678821 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:44.678725    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 14 02:52:44.678821 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:44.678802    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18063d29-58fa-4599-8e9c-93c829fa0fb7-original-pull-secret podName:18063d29-58fa-4599-8e9c-93c829fa0fb7 nodeName:}" failed. No retries permitted until 2026-09-14 02:52:52.678782494 +0000 UTC m=+18.059866677 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/18063d29-58fa-4599-8e9c-93c829fa0fb7-original-pull-secret") pod "global-pull-secret-syncer-zhkkf" (UID: "18063d29-58fa-4599-8e9c-93c829fa0fb7") : object "kube-system"/"original-pull-secret" not registered
Sep 14 02:52:44.779986 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:44.779954    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7029d1df-1b4b-40f6-95e1-621b70070086-metrics-certs\") pod \"network-metrics-daemon-8lf5z\" (UID: \"7029d1df-1b4b-40f6-95e1-621b70070086\") " pod="openshift-multus/network-metrics-daemon-8lf5z"
Sep 14 02:52:44.780234 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:44.780130    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 02:52:44.780234 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:44.780200    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7029d1df-1b4b-40f6-95e1-621b70070086-metrics-certs podName:7029d1df-1b4b-40f6-95e1-621b70070086 nodeName:}" failed. No retries permitted until 2026-09-14 02:52:52.780181768 +0000 UTC m=+18.161265959 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7029d1df-1b4b-40f6-95e1-621b70070086-metrics-certs") pod "network-metrics-daemon-8lf5z" (UID: "7029d1df-1b4b-40f6-95e1-621b70070086") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 02:52:44.880809 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:44.880777    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn8sw\" (UniqueName: \"kubernetes.io/projected/8804d811-79f4-4612-bf40-38c08a04d2c8-kube-api-access-tn8sw\") pod \"network-check-target-kq5nf\" (UID: \"8804d811-79f4-4612-bf40-38c08a04d2c8\") " pod="openshift-network-diagnostics/network-check-target-kq5nf"
Sep 14 02:52:44.880943 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:44.880917    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 14 02:52:44.880943 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:44.880932    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 14 02:52:44.880943 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:44.880940    2564 projected.go:196] Error preparing data for projected volume kube-api-access-tn8sw for pod openshift-network-diagnostics/network-check-target-kq5nf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 02:52:44.881055 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:44.880987    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8804d811-79f4-4612-bf40-38c08a04d2c8-kube-api-access-tn8sw podName:8804d811-79f4-4612-bf40-38c08a04d2c8 nodeName:}" failed. No retries permitted until 2026-09-14 02:52:52.880975566 +0000 UTC m=+18.262059757 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-tn8sw" (UniqueName: "kubernetes.io/projected/8804d811-79f4-4612-bf40-38c08a04d2c8-kube-api-access-tn8sw") pod "network-check-target-kq5nf" (UID: "8804d811-79f4-4612-bf40-38c08a04d2c8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 02:52:45.092633 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:45.092597    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8lf5z"
Sep 14 02:52:45.093085 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:45.092703    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8lf5z" podUID="7029d1df-1b4b-40f6-95e1-621b70070086"
Sep 14 02:52:45.093149 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:45.093126    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kq5nf"
Sep 14 02:52:45.093248 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:45.093214    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kq5nf" podUID="8804d811-79f4-4612-bf40-38c08a04d2c8"
Sep 14 02:52:46.092215 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:46.092183    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhkkf"
Sep 14 02:52:46.092357 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:46.092312    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhkkf" podUID="18063d29-58fa-4599-8e9c-93c829fa0fb7"
Sep 14 02:52:47.092536 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:47.092502    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8lf5z"
Sep 14 02:52:47.093055 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:47.092502    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kq5nf"
Sep 14 02:52:47.093055 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:47.092627    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8lf5z" podUID="7029d1df-1b4b-40f6-95e1-621b70070086"
Sep 14 02:52:47.093055 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:47.092726    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kq5nf" podUID="8804d811-79f4-4612-bf40-38c08a04d2c8"
Sep 14 02:52:48.092486 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:48.092448    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhkkf"
Sep 14 02:52:48.092650 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:48.092579    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhkkf" podUID="18063d29-58fa-4599-8e9c-93c829fa0fb7"
Sep 14 02:52:49.092095 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:49.092058    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kq5nf"
Sep 14 02:52:49.092273 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:49.092065    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8lf5z"
Sep 14 02:52:49.092273 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:49.092183    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kq5nf" podUID="8804d811-79f4-4612-bf40-38c08a04d2c8"
Sep 14 02:52:49.092378 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:49.092283    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8lf5z" podUID="7029d1df-1b4b-40f6-95e1-621b70070086"
Sep 14 02:52:50.091909 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:50.091867    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhkkf"
Sep 14 02:52:50.092391 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:50.091998    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhkkf" podUID="18063d29-58fa-4599-8e9c-93c829fa0fb7"
Sep 14 02:52:51.093039 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:51.092989    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kq5nf"
Sep 14 02:52:51.093511 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:51.093121    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kq5nf" podUID="8804d811-79f4-4612-bf40-38c08a04d2c8"
Sep 14 02:52:51.093511 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:51.093128    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8lf5z"
Sep 14 02:52:51.093511 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:51.093243    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8lf5z" podUID="7029d1df-1b4b-40f6-95e1-621b70070086"
Sep 14 02:52:52.091863 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:52.091820    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhkkf"
Sep 14 02:52:52.092089 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:52.091968    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhkkf" podUID="18063d29-58fa-4599-8e9c-93c829fa0fb7"
Sep 14 02:52:52.740783 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:52.740748    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/18063d29-58fa-4599-8e9c-93c829fa0fb7-original-pull-secret\") pod \"global-pull-secret-syncer-zhkkf\" (UID: \"18063d29-58fa-4599-8e9c-93c829fa0fb7\") " pod="kube-system/global-pull-secret-syncer-zhkkf"
Sep 14 02:52:52.741252 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:52.740929    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 14 02:52:52.741252 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:52.741009    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18063d29-58fa-4599-8e9c-93c829fa0fb7-original-pull-secret podName:18063d29-58fa-4599-8e9c-93c829fa0fb7 nodeName:}" failed. No retries permitted until 2026-09-14 02:53:08.740988153 +0000 UTC m=+34.122072338 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/18063d29-58fa-4599-8e9c-93c829fa0fb7-original-pull-secret") pod "global-pull-secret-syncer-zhkkf" (UID: "18063d29-58fa-4599-8e9c-93c829fa0fb7") : object "kube-system"/"original-pull-secret" not registered
Sep 14 02:52:52.841697 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:52.841651    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7029d1df-1b4b-40f6-95e1-621b70070086-metrics-certs\") pod \"network-metrics-daemon-8lf5z\" (UID: \"7029d1df-1b4b-40f6-95e1-621b70070086\") " pod="openshift-multus/network-metrics-daemon-8lf5z"
Sep 14 02:52:52.841885 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:52.841803    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 02:52:52.841885 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:52.841883    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7029d1df-1b4b-40f6-95e1-621b70070086-metrics-certs podName:7029d1df-1b4b-40f6-95e1-621b70070086 nodeName:}" failed. No retries permitted until 2026-09-14 02:53:08.841857807 +0000 UTC m=+34.222941999 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7029d1df-1b4b-40f6-95e1-621b70070086-metrics-certs") pod "network-metrics-daemon-8lf5z" (UID: "7029d1df-1b4b-40f6-95e1-621b70070086") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 02:52:52.942637 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:52.942596    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn8sw\" (UniqueName: \"kubernetes.io/projected/8804d811-79f4-4612-bf40-38c08a04d2c8-kube-api-access-tn8sw\") pod \"network-check-target-kq5nf\" (UID: \"8804d811-79f4-4612-bf40-38c08a04d2c8\") " pod="openshift-network-diagnostics/network-check-target-kq5nf"
Sep 14 02:52:52.942833 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:52.942772    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 14 02:52:52.942833 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:52.942797    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 14 02:52:52.942833 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:52.942812    2564 projected.go:196] Error preparing data for projected volume kube-api-access-tn8sw for pod openshift-network-diagnostics/network-check-target-kq5nf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 02:52:52.942990 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:52.942879    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8804d811-79f4-4612-bf40-38c08a04d2c8-kube-api-access-tn8sw podName:8804d811-79f4-4612-bf40-38c08a04d2c8 nodeName:}" failed. No retries permitted until 2026-09-14 02:53:08.942861269 +0000 UTC m=+34.323945447 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-tn8sw" (UniqueName: "kubernetes.io/projected/8804d811-79f4-4612-bf40-38c08a04d2c8-kube-api-access-tn8sw") pod "network-check-target-kq5nf" (UID: "8804d811-79f4-4612-bf40-38c08a04d2c8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 02:52:53.092352 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:53.092316    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kq5nf"
Sep 14 02:52:53.092547 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:53.092318    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8lf5z"
Sep 14 02:52:53.092547 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:53.092486    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kq5nf" podUID="8804d811-79f4-4612-bf40-38c08a04d2c8"
Sep 14 02:52:53.092638 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:53.092619    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8lf5z" podUID="7029d1df-1b4b-40f6-95e1-621b70070086"
Sep 14 02:52:54.091825 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:54.091791    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhkkf"
Sep 14 02:52:54.092224 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:54.091900    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhkkf" podUID="18063d29-58fa-4599-8e9c-93c829fa0fb7"
Sep 14 02:52:55.092948 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:55.092914    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8lf5z"
Sep 14 02:52:55.093376 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:55.093086    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8lf5z" podUID="7029d1df-1b4b-40f6-95e1-621b70070086"
Sep 14 02:52:55.093376 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:55.093136    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kq5nf"
Sep 14 02:52:55.093376 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:55.093223    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kq5nf" podUID="8804d811-79f4-4612-bf40-38c08a04d2c8"
Sep 14 02:52:56.092042 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:56.091866    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhkkf"
Sep 14 02:52:56.092231 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:56.092156    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhkkf" podUID="18063d29-58fa-4599-8e9c-93c829fa0fb7"
Sep 14 02:52:56.200291 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:56.200260    2564 generic.go:356] "Generic (PLEG): container finished" podID="b92669c6-6693-41c9-8d20-b4ac5bffe96f" containerID="6cd3bf18964e6b601f1cd1e7e4a250bf0e61ade61f7381d26bdee4e1ae4ba6d5" exitCode=0
Sep 14 02:52:56.200859 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:56.200335    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qx7v5" event={"ID":"b92669c6-6693-41c9-8d20-b4ac5bffe96f","Type":"ContainerDied","Data":"6cd3bf18964e6b601f1cd1e7e4a250bf0e61ade61f7381d26bdee4e1ae4ba6d5"}
Sep 14 02:52:56.202009 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:56.201974    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-f55dq" event={"ID":"43421977-8a08-4a29-869d-a165accae989","Type":"ContainerStarted","Data":"04471716500a095612acc155be3b76af2a02a1b72399ce05375e96d7aba241a1"}
Sep 14 02:52:56.203542 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:56.203519    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4t7c6" event={"ID":"4813e57c-ad63-4564-8f34-707c6438e999","Type":"ContainerStarted","Data":"8711a79a09a8e75fea8d04c047efee3de2194a2dead9a6e8f80913dbf054d325"}
Sep 14 02:52:56.204956 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:56.204926    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-whvw4" event={"ID":"c3ff549c-82fc-4dec-b893-6aa2b2fa1259","Type":"ContainerStarted","Data":"af2fa4f035dc5fb9adf4c3a8062f50882ca672cb4ec84752e073373e8cb40041"}
Sep 14 02:52:56.206362 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:56.206341    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2hr6w" event={"ID":"09389d2f-4da9-4e12-91e1-d616caa472d7","Type":"ContainerStarted","Data":"53b24f14a26d213c1bd923a29fd1de01b52c6728c87c7ba8fca708c9ed5e768e"}
Sep 14 02:52:56.207722 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:56.207696    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-c57s5" event={"ID":"789839ca-54f5-4cab-815c-777c86244553","Type":"ContainerStarted","Data":"f0c01bd93ad08a144368698b85518c25b4f7fb1151de306a7d554a3373eefdc5"}
Sep 14 02:52:56.209174 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:56.209153    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f4hxl" event={"ID":"ec25ba7d-9bc2-4f39-a9cf-57894311b7cc","Type":"ContainerStarted","Data":"90deb9b40399b33dc534adbe328e2307158bd000f7507e8e6f81ed2867d90427"}
Sep 14 02:52:56.281314 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:56.281272    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-f55dq" podStartSLOduration=3.595489722 podStartE2EDuration="21.281257645s" totalImagesPullingTime="17.685767923s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 02:52:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:52:56.280851959 +0000 UTC m=+21.661936158" watchObservedRunningTime="2026-09-14 02:52:56.281257645 +0000 UTC m=+21.662341843"
Sep 14 02:52:56.304089 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:56.304050    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-2hr6w" podStartSLOduration=7.713747579 podStartE2EDuration="21.304036129s" totalImagesPullingTime="13.59028855s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:52:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:52:56.303800899 +0000 UTC m=+21.684885091" watchObservedRunningTime="2026-09-14 02:52:56.304036129 +0000 UTC m=+21.685120340"
Sep 14 02:52:56.324815 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:56.324765    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-c57s5" podStartSLOduration=7.738318761 podStartE2EDuration="21.324750278s" totalImagesPullingTime="13.586431517s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 02:52:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:52:56.324716046 +0000 UTC m=+21.705800244" watchObservedRunningTime="2026-09-14 02:52:56.324750278 +0000 UTC m=+21.705834477"
Sep 14 02:52:56.404839 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:56.404632    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-4t7c6" podStartSLOduration=3.752148065 podStartE2EDuration="21.404610636s" totalImagesPullingTime="17.652462571s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 02:52:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:52:56.35953685 +0000 UTC m=+21.740621049" watchObservedRunningTime="2026-09-14 02:52:56.404610636 +0000 UTC m=+21.785694836"
Sep 14 02:52:56.404940 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:56.404918    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-whvw4" podStartSLOduration=3.749514047 podStartE2EDuration="21.404910119s" totalImagesPullingTime="17.655396072s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 02:52:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:52:56.404311199 +0000 UTC m=+21.785395398" watchObservedRunningTime="2026-09-14 02:52:56.404910119 +0000 UTC m=+21.785994320"
Sep 14 02:52:56.423729 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:56.423698    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:52:56.502638 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:56.502607    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:52:56.605178 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:56.605140    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:52:56.667684 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:56.667608    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:52:57.021135 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:57.021118    2564 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 14 02:52:57.051146 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:57.051126    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-2hr6w"
Sep 14 02:52:57.051700 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:57.051681    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-2hr6w"
Sep 14 02:52:57.057502 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:57.057382    2564 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-14T02:52:57.021130827Z","UUID":"c3a5f6df-4130-4008-a782-22c6cdf47475","Handler":null,"Name":"","Endpoint":""}
Sep 14 02:52:57.058788 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:57.058773    2564 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 14 02:52:57.058892 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:57.058803    2564 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 14 02:52:57.090927 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:57.090904    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-whvw4_c3ff549c-82fc-4dec-b893-6aa2b2fa1259/node-ca/0.log"
Sep 14 02:52:57.092117 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:57.092095    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8lf5z"
Sep 14 02:52:57.092233 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:57.092096    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kq5nf"
Sep 14 02:52:57.092292 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:57.092225    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8lf5z" podUID="7029d1df-1b4b-40f6-95e1-621b70070086"
Sep 14 02:52:57.092292 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:57.092260    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kq5nf" podUID="8804d811-79f4-4612-bf40-38c08a04d2c8"
Sep 14 02:52:57.214374 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:57.214294    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd" event={"ID":"85099e4c-154c-4a21-8e4a-c2ef17f17b25","Type":"ContainerStarted","Data":"464a7b7604787a8ff2eafb20192792af822565f6cc8cd4be3742a1763ff3de5b"}
Sep 14 02:52:57.215030 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:57.214375    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd" event={"ID":"85099e4c-154c-4a21-8e4a-c2ef17f17b25","Type":"ContainerStarted","Data":"f6b1f0c6b4730c7273c0ff12f44e6276088677efbb89c25dc008fdb28195df9a"}
Sep 14 02:52:57.215030 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:57.214437    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd" event={"ID":"85099e4c-154c-4a21-8e4a-c2ef17f17b25","Type":"ContainerStarted","Data":"f2f751be24f6550dbc39c4629bae1561f9c85b18932d16a129cac0bce7bf0b12"}
Sep 14 02:52:57.215030 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:57.214478    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd" event={"ID":"85099e4c-154c-4a21-8e4a-c2ef17f17b25","Type":"ContainerStarted","Data":"8c37e33c5296328a543b05b6cb072bd9f1b651d4671bbbd3cc9f551344b27094"}
Sep 14 02:52:57.215030 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:57.214517    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd" event={"ID":"85099e4c-154c-4a21-8e4a-c2ef17f17b25","Type":"ContainerStarted","Data":"14aecf32ccad53eac8f33e8560ad19c4f2a0b7b02a7bce7014132794045c73a1"}
Sep 14 02:52:57.215030 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:57.214559    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd" event={"ID":"85099e4c-154c-4a21-8e4a-c2ef17f17b25","Type":"ContainerStarted","Data":"d311681c2dd759f6d59e290e05d5cb9c1d4a47f5895b98ad51479c92e4772952"}
Sep 14 02:52:57.215809 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:57.215776    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bm7vz" event={"ID":"fa0ee9a6-d143-4c3d-8d35-306c30cec60a","Type":"ContainerStarted","Data":"eebf50c12418f2d3dd752b50e6ca578c39f0d45d6dd2fa448ae002860eb57299"}
Sep 14 02:52:57.217399 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:57.217379    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f4hxl" event={"ID":"ec25ba7d-9bc2-4f39-a9cf-57894311b7cc","Type":"ContainerStarted","Data":"b8f79a3be8d71e9af8e9b88e7643ed0d255d509c45a1d9c8c3983dae62a4730c"}
Sep 14 02:52:57.217612 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:57.217598    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-2hr6w"
Sep 14 02:52:57.218044 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:57.218027    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-2hr6w"
Sep 14 02:52:57.232814 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:57.232781    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-bm7vz" podStartSLOduration=4.570132227 podStartE2EDuration="22.2327693s" totalImagesPullingTime="17.662637073s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 02:52:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:52:57.232673758 +0000 UTC m=+22.613757958" watchObservedRunningTime="2026-09-14 02:52:57.2327693 +0000 UTC m=+22.613853482"
Sep 14 02:52:57.355534 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:57.355504    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:52:57.434331 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:57.434283    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:52:57.598397 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:57.598326    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:52:57.703501 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:57.703157    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:52:58.092166 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:58.092134    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhkkf"
Sep 14 02:52:58.092323 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:58.092250    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhkkf" podUID="18063d29-58fa-4599-8e9c-93c829fa0fb7"
Sep 14 02:52:58.221336 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:58.221261    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f4hxl" event={"ID":"ec25ba7d-9bc2-4f39-a9cf-57894311b7cc","Type":"ContainerStarted","Data":"7e456b250c631323e96d14b1bfb86a678b7beb4451b3d1b57042c8aa98b7436d"}
Sep 14 02:52:58.247101 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:58.247064    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f4hxl" podStartSLOduration=3.788276922 podStartE2EDuration="23.247050165s" totalImagesPullingTime="19.458773243s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 02:52:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:52:58.246983274 +0000 UTC m=+23.628067472" watchObservedRunningTime="2026-09-14 02:52:58.247050165 +0000 UTC m=+23.628134363"
Sep 14 02:52:58.302039 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:58.301987    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:52:58.380796 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:58.380765    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:52:58.439040 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:58.439000    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:52:59.092313 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:59.092284    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8lf5z"
Sep 14 02:52:59.092515 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:59.092430    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8lf5z" podUID="7029d1df-1b4b-40f6-95e1-621b70070086"
Sep 14 02:52:59.092515 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:59.092475    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kq5nf"
Sep 14 02:52:59.092637 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:52:59.092579    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kq5nf" podUID="8804d811-79f4-4612-bf40-38c08a04d2c8"
Sep 14 02:52:59.226832 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:52:59.226801    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd" event={"ID":"85099e4c-154c-4a21-8e4a-c2ef17f17b25","Type":"ContainerStarted","Data":"33f07dd5e64fc7a6038b7aab5b60b71c1ab4b3d0798b3d9275bf75fb16c5676e"}
Sep 14 02:53:00.091728 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:00.091494    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhkkf"
Sep 14 02:53:00.091893 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:53:00.091838    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhkkf" podUID="18063d29-58fa-4599-8e9c-93c829fa0fb7"
Sep 14 02:53:01.091860 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:01.091828    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kq5nf"
Sep 14 02:53:01.092313 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:53:01.091947    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kq5nf" podUID="8804d811-79f4-4612-bf40-38c08a04d2c8"
Sep 14 02:53:01.092313 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:01.091994    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8lf5z"
Sep 14 02:53:01.092313 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:53:01.092101    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8lf5z" podUID="7029d1df-1b4b-40f6-95e1-621b70070086"
Sep 14 02:53:01.946455 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:01.946223    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:02.025959 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:02.025931    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:02.092470 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:02.092436    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhkkf"
Sep 14 02:53:02.093110 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:53:02.092567    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhkkf" podUID="18063d29-58fa-4599-8e9c-93c829fa0fb7"
Sep 14 02:53:02.236552 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:02.236387    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd" event={"ID":"85099e4c-154c-4a21-8e4a-c2ef17f17b25","Type":"ContainerStarted","Data":"9a37f41a82effd7e925f3f19d71701438ffafca509c3892fb766d0887cc5a999"}
Sep 14 02:53:02.236707 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:02.236661    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd"
Sep 14 02:53:02.239061 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:02.239039    2564 generic.go:356] "Generic (PLEG): container finished" podID="b92669c6-6693-41c9-8d20-b4ac5bffe96f" containerID="b251179cfb4abc642d6d5ea155e2fbb094df5b0eb820d1890eca37f194c86e09" exitCode=0
Sep 14 02:53:02.239172 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:02.239078    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qx7v5" event={"ID":"b92669c6-6693-41c9-8d20-b4ac5bffe96f","Type":"ContainerDied","Data":"b251179cfb4abc642d6d5ea155e2fbb094df5b0eb820d1890eca37f194c86e09"}
Sep 14 02:53:02.253017 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:02.252994    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd"
Sep 14 02:53:02.269736 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:02.269691    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd" podStartSLOduration=9.222273802 podStartE2EDuration="27.269676945s" totalImagesPullingTime="18.047403143s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 02:52:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:53:02.267795183 +0000 UTC m=+27.648879381" watchObservedRunningTime="2026-09-14 02:53:02.269676945 +0000 UTC m=+27.650761144"
Sep 14 02:53:02.377890 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:02.377851    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:02.480793 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:02.480753    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:02.678590 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:02.678557    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:02.752908 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:02.752819    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:02.871013 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:02.870983    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:02.956863 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:02.956789    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:03.091874 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:03.091846    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kq5nf"
Sep 14 02:53:03.092040 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:03.091846    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8lf5z"
Sep 14 02:53:03.092040 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:53:03.091963    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kq5nf" podUID="8804d811-79f4-4612-bf40-38c08a04d2c8"
Sep 14 02:53:03.092040 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:53:03.092030    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8lf5z" podUID="7029d1df-1b4b-40f6-95e1-621b70070086"
Sep 14 02:53:03.152771 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:03.152738    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:03.242830 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:03.242728    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd"
Sep 14 02:53:03.242830 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:03.242818    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd"
Sep 14 02:53:03.254446 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:03.254248    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:03.259508 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:03.259483    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd"
Sep 14 02:53:03.522065 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:03.522033    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8lf5z"]
Sep 14 02:53:03.522232 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:03.522179    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8lf5z"
Sep 14 02:53:03.522296 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:53:03.522278    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8lf5z" podUID="7029d1df-1b4b-40f6-95e1-621b70070086"
Sep 14 02:53:03.524784 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:03.524755    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zhkkf"]
Sep 14 02:53:03.524908 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:03.524872    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhkkf"
Sep 14 02:53:03.525019 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:53:03.524970    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhkkf" podUID="18063d29-58fa-4599-8e9c-93c829fa0fb7"
Sep 14 02:53:03.525422 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:03.525381    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kq5nf"]
Sep 14 02:53:03.525528 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:03.525484    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kq5nf"
Sep 14 02:53:03.525596 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:53:03.525556    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kq5nf" podUID="8804d811-79f4-4612-bf40-38c08a04d2c8"
Sep 14 02:53:03.695179 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:03.695149    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:04.246689 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:04.246655    2564 generic.go:356] "Generic (PLEG): container finished" podID="b92669c6-6693-41c9-8d20-b4ac5bffe96f" containerID="e9068c347e457c00dc6eb9c1123f91a93d9173fb75153b24765171a80c893e79" exitCode=0
Sep 14 02:53:04.247094 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:04.246746    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qx7v5" event={"ID":"b92669c6-6693-41c9-8d20-b4ac5bffe96f","Type":"ContainerDied","Data":"e9068c347e457c00dc6eb9c1123f91a93d9173fb75153b24765171a80c893e79"}
Sep 14 02:53:04.310420 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:04.310376    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:04.411448 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:04.411416    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:05.093079 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:05.092850    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhkkf"
Sep 14 02:53:05.093205 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:05.092930    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8lf5z"
Sep 14 02:53:05.093205 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:53:05.093120    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhkkf" podUID="18063d29-58fa-4599-8e9c-93c829fa0fb7"
Sep 14 02:53:05.093205 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:05.092951    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kq5nf"
Sep 14 02:53:05.093205 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:53:05.093188    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8lf5z" podUID="7029d1df-1b4b-40f6-95e1-621b70070086"
Sep 14 02:53:05.093378 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:53:05.093281    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kq5nf" podUID="8804d811-79f4-4612-bf40-38c08a04d2c8"
Sep 14 02:53:05.252245 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:05.252219    2564 generic.go:356] "Generic (PLEG): container finished" podID="b92669c6-6693-41c9-8d20-b4ac5bffe96f" containerID="48366bb2cc84128d1f7ff27387ac5d20dbccd3f412e616d48e5e697c9987a418" exitCode=0
Sep 14 02:53:05.252622 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:05.252292    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qx7v5" event={"ID":"b92669c6-6693-41c9-8d20-b4ac5bffe96f","Type":"ContainerDied","Data":"48366bb2cc84128d1f7ff27387ac5d20dbccd3f412e616d48e5e697c9987a418"}
Sep 14 02:53:05.357418 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:05.357382    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:06.930640 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:06.930571    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:07.092214 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:07.092184    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8lf5z"
Sep 14 02:53:07.092214 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:07.092208    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kq5nf"
Sep 14 02:53:07.092483 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:07.092181    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhkkf"
Sep 14 02:53:07.092483 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:53:07.092316    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8lf5z" podUID="7029d1df-1b4b-40f6-95e1-621b70070086"
Sep 14 02:53:07.092483 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:53:07.092455    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhkkf" podUID="18063d29-58fa-4599-8e9c-93c829fa0fb7"
Sep 14 02:53:07.092622 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:53:07.092525    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kq5nf" podUID="8804d811-79f4-4612-bf40-38c08a04d2c8"
Sep 14 02:53:07.131911 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:07.131878    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:07.248548 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:07.248516    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:07.364892 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:07.364811    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:07.515676 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:07.515644    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:07.711782 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:07.711679    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:08.616298 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:08.616239    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:08.761790 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:08.761752    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/18063d29-58fa-4599-8e9c-93c829fa0fb7-original-pull-secret\") pod \"global-pull-secret-syncer-zhkkf\" (UID: \"18063d29-58fa-4599-8e9c-93c829fa0fb7\") " pod="kube-system/global-pull-secret-syncer-zhkkf"
Sep 14 02:53:08.761975 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:53:08.761940    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 14 02:53:08.762033 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:53:08.762024    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18063d29-58fa-4599-8e9c-93c829fa0fb7-original-pull-secret podName:18063d29-58fa-4599-8e9c-93c829fa0fb7 nodeName:}" failed. No retries permitted until 2026-09-14 02:53:40.762002793 +0000 UTC m=+66.143086988 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/18063d29-58fa-4599-8e9c-93c829fa0fb7-original-pull-secret") pod "global-pull-secret-syncer-zhkkf" (UID: "18063d29-58fa-4599-8e9c-93c829fa0fb7") : object "kube-system"/"original-pull-secret" not registered
Sep 14 02:53:08.830684 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:08.830641    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:08.862618 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:08.862589    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7029d1df-1b4b-40f6-95e1-621b70070086-metrics-certs\") pod \"network-metrics-daemon-8lf5z\" (UID: \"7029d1df-1b4b-40f6-95e1-621b70070086\") " pod="openshift-multus/network-metrics-daemon-8lf5z"
Sep 14 02:53:08.862779 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:53:08.862714    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 02:53:08.862779 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:53:08.862776    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7029d1df-1b4b-40f6-95e1-621b70070086-metrics-certs podName:7029d1df-1b4b-40f6-95e1-621b70070086 nodeName:}" failed. No retries permitted until 2026-09-14 02:53:40.862759209 +0000 UTC m=+66.243843403 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7029d1df-1b4b-40f6-95e1-621b70070086-metrics-certs") pod "network-metrics-daemon-8lf5z" (UID: "7029d1df-1b4b-40f6-95e1-621b70070086") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 02:53:08.963522 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:08.963448    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn8sw\" (UniqueName: \"kubernetes.io/projected/8804d811-79f4-4612-bf40-38c08a04d2c8-kube-api-access-tn8sw\") pod \"network-check-target-kq5nf\" (UID: \"8804d811-79f4-4612-bf40-38c08a04d2c8\") " pod="openshift-network-diagnostics/network-check-target-kq5nf"
Sep 14 02:53:08.963660 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:53:08.963586    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 14 02:53:08.963660 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:53:08.963612    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 14 02:53:08.963660 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:53:08.963621    2564 projected.go:196] Error preparing data for projected volume kube-api-access-tn8sw for pod openshift-network-diagnostics/network-check-target-kq5nf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 02:53:08.963799 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:53:08.963678    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8804d811-79f4-4612-bf40-38c08a04d2c8-kube-api-access-tn8sw podName:8804d811-79f4-4612-bf40-38c08a04d2c8 nodeName:}" failed. No retries permitted until 2026-09-14 02:53:40.963658973 +0000 UTC m=+66.344743152 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-tn8sw" (UniqueName: "kubernetes.io/projected/8804d811-79f4-4612-bf40-38c08a04d2c8-kube-api-access-tn8sw") pod "network-check-target-kq5nf" (UID: "8804d811-79f4-4612-bf40-38c08a04d2c8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 02:53:09.092098 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:09.092060    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kq5nf"
Sep 14 02:53:09.092098 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:09.092097    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8lf5z"
Sep 14 02:53:09.092332 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:09.092063    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhkkf"
Sep 14 02:53:09.092332 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:53:09.092182    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kq5nf" podUID="8804d811-79f4-4612-bf40-38c08a04d2c8"
Sep 14 02:53:09.092332 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:53:09.092268    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhkkf" podUID="18063d29-58fa-4599-8e9c-93c829fa0fb7"
Sep 14 02:53:09.092516 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:53:09.092430    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8lf5z" podUID="7029d1df-1b4b-40f6-95e1-621b70070086"
Sep 14 02:53:09.220040 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:09.219953    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:09.445500 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:09.445471    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:09.646379 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:09.646350    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:09.755556 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:09.755526    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:10.039777 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:10.039745    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:10.266963 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:10.266935    2564 generic.go:356] "Generic (PLEG): container finished" podID="b92669c6-6693-41c9-8d20-b4ac5bffe96f" containerID="6be2f27d75f39f5145a3d93cec4c0fce3d550ed253f866bb83d4456221f2185a" exitCode=0
Sep 14 02:53:10.267104 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:10.266983    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qx7v5" event={"ID":"b92669c6-6693-41c9-8d20-b4ac5bffe96f","Type":"ContainerDied","Data":"6be2f27d75f39f5145a3d93cec4c0fce3d550ed253f866bb83d4456221f2185a"}
Sep 14 02:53:10.418113 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:10.418084    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:10.609833 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:10.609758    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:10.710842 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:10.710789    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:11.041183 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:11.041153    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:11.092533 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:11.092508    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8lf5z"
Sep 14 02:53:11.092533 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:11.092538    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kq5nf"
Sep 14 02:53:11.092718 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:11.092508    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhkkf"
Sep 14 02:53:11.092718 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:53:11.092623    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8lf5z" podUID="7029d1df-1b4b-40f6-95e1-621b70070086"
Sep 14 02:53:11.092718 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:53:11.092683    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhkkf" podUID="18063d29-58fa-4599-8e9c-93c829fa0fb7"
Sep 14 02:53:11.092804 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:53:11.092748    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kq5nf" podUID="8804d811-79f4-4612-bf40-38c08a04d2c8"
Sep 14 02:53:11.256866 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:11.256829    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:11.272391 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:11.272368    2564 generic.go:356] "Generic (PLEG): container finished" podID="b92669c6-6693-41c9-8d20-b4ac5bffe96f" containerID="e1bcb1443149efcf205ef9d939d55d09a27fc5d71683773efab3440bf0449a31" exitCode=0
Sep 14 02:53:11.272515 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:11.272428    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qx7v5" event={"ID":"b92669c6-6693-41c9-8d20-b4ac5bffe96f","Type":"ContainerDied","Data":"e1bcb1443149efcf205ef9d939d55d09a27fc5d71683773efab3440bf0449a31"}
Sep 14 02:53:11.622179 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:11.622108    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:11.704515 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:11.704483    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:11.813426 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:11.813369    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:12.225780 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:12.225745    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:12.278315 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:12.278281    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qx7v5" event={"ID":"b92669c6-6693-41c9-8d20-b4ac5bffe96f","Type":"ContainerStarted","Data":"386dd10ebbd2a488b7b276fea3efc905a749d7e3957a0def9180e399c2ab77d8"}
Sep 14 02:53:12.343638 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:12.343592    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-qx7v5" podStartSLOduration=8.045156723 podStartE2EDuration="37.343574781s" totalImagesPullingTime="29.298418058s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 02:52:35 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:53:12.326825114 +0000 UTC m=+37.707909324" watchObservedRunningTime="2026-09-14 02:53:12.343574781 +0000 UTC m=+37.724658979"
Sep 14 02:53:12.429867 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:12.429832    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:12.616605 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:12.616568    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:12.767758 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:12.767712    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:12.900167 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:12.900080    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:13.028972 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:13.028923    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:13.092329 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:13.092299    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8lf5z"
Sep 14 02:53:13.092467 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:13.092301    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kq5nf"
Sep 14 02:53:13.092467 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:53:13.092449    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8lf5z" podUID="7029d1df-1b4b-40f6-95e1-621b70070086"
Sep 14 02:53:13.092591 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:13.092306    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhkkf"
Sep 14 02:53:13.092591 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:53:13.092509    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kq5nf" podUID="8804d811-79f4-4612-bf40-38c08a04d2c8"
Sep 14 02:53:13.092591 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:53:13.092574    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhkkf" podUID="18063d29-58fa-4599-8e9c-93c829fa0fb7"
Sep 14 02:53:13.219169 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:13.219096    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:13.409999 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:13.409970    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:13.500076 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:13.499990    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:13.830341 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:13.830307    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:13.887140 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:13.887112    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:13.960420 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:13.960385    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:14.060240 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:14.060210    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:14.225663 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:14.225577    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:14.417041 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:14.417006    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:14.633667 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:14.633621    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:14.825078 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:14.825026    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:15.093934 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:15.093908    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8lf5z"
Sep 14 02:53:15.094294 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:15.094003    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kq5nf"
Sep 14 02:53:15.094294 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:53:15.094036    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8lf5z" podUID="7029d1df-1b4b-40f6-95e1-621b70070086"
Sep 14 02:53:15.094294 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:15.094045    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhkkf"
Sep 14 02:53:15.094294 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:53:15.094108    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kq5nf" podUID="8804d811-79f4-4612-bf40-38c08a04d2c8"
Sep 14 02:53:15.094294 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:53:15.094198    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhkkf" podUID="18063d29-58fa-4599-8e9c-93c829fa0fb7"
Sep 14 02:53:15.219110 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:15.219077    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:15.617420 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:15.617359    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:16.014990 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:16.014908    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:16.413687 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:16.413655    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:16.618305 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:16.618276    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:17.016801 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:17.016762    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:17.092281 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:17.092247    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhkkf"
Sep 14 02:53:17.092491 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:17.092247    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kq5nf"
Sep 14 02:53:17.092491 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:53:17.092465    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kq5nf" podUID="8804d811-79f4-4612-bf40-38c08a04d2c8"
Sep 14 02:53:17.092491 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:53:17.092357    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhkkf" podUID="18063d29-58fa-4599-8e9c-93c829fa0fb7"
Sep 14 02:53:17.092594 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:17.092514    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8lf5z"
Sep 14 02:53:17.092642 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:53:17.092622    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8lf5z" podUID="7029d1df-1b4b-40f6-95e1-621b70070086"
Sep 14 02:53:17.137526 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:17.137491    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:17.620987 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:17.620954    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:17.800905 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:17.800867    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:17.863417 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:17.863385    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:18.013968 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:18.013883    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:19.091634 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:19.091598    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kq5nf"
Sep 14 02:53:19.092062 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:19.091598    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhkkf"
Sep 14 02:53:19.092062 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:53:19.091735    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kq5nf" podUID="8804d811-79f4-4612-bf40-38c08a04d2c8"
Sep 14 02:53:19.092062 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:19.091608    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8lf5z"
Sep 14 02:53:19.092062 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:53:19.091800    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhkkf" podUID="18063d29-58fa-4599-8e9c-93c829fa0fb7"
Sep 14 02:53:19.092062 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:53:19.091860    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8lf5z" podUID="7029d1df-1b4b-40f6-95e1-621b70070086"
Sep 14 02:53:21.092117 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:21.092085    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhkkf"
Sep 14 02:53:21.092637 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:21.092092    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8lf5z"
Sep 14 02:53:21.092637 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:53:21.092189    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhkkf" podUID="18063d29-58fa-4599-8e9c-93c829fa0fb7"
Sep 14 02:53:21.092637 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:53:21.092291    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8lf5z" podUID="7029d1df-1b4b-40f6-95e1-621b70070086"
Sep 14 02:53:21.092637 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:21.092299    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kq5nf"
Sep 14 02:53:21.092637 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:53:21.092379    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kq5nf" podUID="8804d811-79f4-4612-bf40-38c08a04d2c8"
Sep 14 02:53:23.091965 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:23.091933    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kq5nf"
Sep 14 02:53:23.092371 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:23.091941    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhkkf"
Sep 14 02:53:23.092371 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:53:23.092030    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kq5nf" podUID="8804d811-79f4-4612-bf40-38c08a04d2c8"
Sep 14 02:53:23.092371 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:53:23.092104    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhkkf" podUID="18063d29-58fa-4599-8e9c-93c829fa0fb7"
Sep 14 02:53:23.092371 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:23.092141    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8lf5z"
Sep 14 02:53:23.092371 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:53:23.092266    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8lf5z" podUID="7029d1df-1b4b-40f6-95e1-621b70070086"
Sep 14 02:53:25.092342 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:25.092309    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kq5nf"
Sep 14 02:53:25.092724 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:25.092370    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhkkf"
Sep 14 02:53:25.092724 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:53:25.092387    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kq5nf" podUID="8804d811-79f4-4612-bf40-38c08a04d2c8"
Sep 14 02:53:25.092724 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:53:25.092468    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhkkf" podUID="18063d29-58fa-4599-8e9c-93c829fa0fb7"
Sep 14 02:53:25.092724 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:25.092520    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8lf5z"
Sep 14 02:53:25.092724 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:53:25.092579    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8lf5z" podUID="7029d1df-1b4b-40f6-95e1-621b70070086"
Sep 14 02:53:26.820576 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:26.820534    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:26.994504 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:26.994471    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:27.091625 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:27.091531    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhkkf"
Sep 14 02:53:27.091625 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:27.091547    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8lf5z"
Sep 14 02:53:27.091849 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:53:27.091640    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhkkf" podUID="18063d29-58fa-4599-8e9c-93c829fa0fb7"
Sep 14 02:53:27.091849 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:53:27.091697    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8lf5z" podUID="7029d1df-1b4b-40f6-95e1-621b70070086"
Sep 14 02:53:27.091849 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:27.091714    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kq5nf"
Sep 14 02:53:27.091849 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:53:27.091788    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kq5nf" podUID="8804d811-79f4-4612-bf40-38c08a04d2c8"
Sep 14 02:53:27.312362 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:27.312318    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:29.092083 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:29.092054    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kq5nf"
Sep 14 02:53:29.092589 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:53:29.092142    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kq5nf" podUID="8804d811-79f4-4612-bf40-38c08a04d2c8"
Sep 14 02:53:29.092589 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:29.092065    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8lf5z"
Sep 14 02:53:29.092589 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:29.092167    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhkkf"
Sep 14 02:53:29.092589 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:53:29.092259    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8lf5z" podUID="7029d1df-1b4b-40f6-95e1-621b70070086"
Sep 14 02:53:29.092589 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:53:29.092325    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhkkf" podUID="18063d29-58fa-4599-8e9c-93c829fa0fb7"
Sep 14 02:53:29.868731 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:29.868690    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:29.952169 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:29.952123    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:30.055591 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:30.055553    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:30.158785 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:30.158668    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:30.882891 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:30.882863    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-4.ec2.internal" event="NodeReady"
Sep 14 02:53:30.883047 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:30.882990    2564 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 14 02:53:30.934830 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:30.934801    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-nwtp8"]
Sep 14 02:53:30.936380 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:30.936364    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nwtp8"
Sep 14 02:53:30.938903 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:30.938886    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 14 02:53:30.938971 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:30.938920    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 14 02:53:30.938971 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:30.938888    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gj97m\""
Sep 14 02:53:30.943351 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:30.943332    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jhnxs"]
Sep 14 02:53:30.944791 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:30.944775    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jhnxs"
Sep 14 02:53:30.947280 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:30.947257    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 14 02:53:30.947388 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:30.947368    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 14 02:53:30.948765 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:30.947943    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-gwhfd\""
Sep 14 02:53:30.948765 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:30.947977    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 14 02:53:30.951699 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:30.950597    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nwtp8"]
Sep 14 02:53:30.956006 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:30.955985    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jhnxs"]
Sep 14 02:53:31.040845 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:31.040812    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-d7hms"]
Sep 14 02:53:31.042550 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:31.042536    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-d7hms"
Sep 14 02:53:31.045380 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:31.045358    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 14 02:53:31.045641 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:31.045622    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 14 02:53:31.045820 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:31.045803    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ztclm\""
Sep 14 02:53:31.045928 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:31.045910    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 14 02:53:31.046090 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:31.046077    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 14 02:53:31.059250 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:31.059226    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-d7hms"]
Sep 14 02:53:31.069790 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:31.069763    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:31.091700 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:31.091676    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8lf5z"
Sep 14 02:53:31.091700 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:31.091692    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kq5nf"
Sep 14 02:53:31.091871 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:31.091670    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhkkf"
Sep 14 02:53:31.094583 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:31.094563    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 14 02:53:31.094758 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:31.094743    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hn74x\""
Sep 14 02:53:31.095665 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:31.095646    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 14 02:53:31.095765 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:31.095683    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lx47r\""
Sep 14 02:53:31.095765 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:31.095687    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 14 02:53:31.096736 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:31.096721    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 14 02:53:31.114063 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:31.114046    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4sfk\" (UniqueName: \"kubernetes.io/projected/c9ce7325-f74d-4a0f-8dec-51099aaa1cf3-kube-api-access-s4sfk\") pod \"dns-default-nwtp8\" (UID: \"c9ce7325-f74d-4a0f-8dec-51099aaa1cf3\") " pod="openshift-dns/dns-default-nwtp8"
Sep 14 02:53:31.114139 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:31.114072    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvb87\" (UniqueName: \"kubernetes.io/projected/e30169cd-3ac5-476e-bdae-555456b42f7a-kube-api-access-wvb87\") pod \"ingress-canary-jhnxs\" (UID: \"e30169cd-3ac5-476e-bdae-555456b42f7a\") " pod="openshift-ingress-canary/ingress-canary-jhnxs"
Sep 14 02:53:31.114139 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:31.114087    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c9ce7325-f74d-4a0f-8dec-51099aaa1cf3-metrics-tls\") pod \"dns-default-nwtp8\" (UID: \"c9ce7325-f74d-4a0f-8dec-51099aaa1cf3\") " pod="openshift-dns/dns-default-nwtp8"
Sep 14 02:53:31.114209 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:31.114160    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e30169cd-3ac5-476e-bdae-555456b42f7a-cert\") pod \"ingress-canary-jhnxs\" (UID: \"e30169cd-3ac5-476e-bdae-555456b42f7a\") " pod="openshift-ingress-canary/ingress-canary-jhnxs"
Sep 14 02:53:31.114209 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:31.114182    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c9ce7325-f74d-4a0f-8dec-51099aaa1cf3-config-volume\") pod \"dns-default-nwtp8\" (UID: \"c9ce7325-f74d-4a0f-8dec-51099aaa1cf3\") " pod="openshift-dns/dns-default-nwtp8"
Sep 14 02:53:31.114267 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:31.114207    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9ce7325-f74d-4a0f-8dec-51099aaa1cf3-tmp-dir\") pod \"dns-default-nwtp8\" (UID: \"c9ce7325-f74d-4a0f-8dec-51099aaa1cf3\") " pod="openshift-dns/dns-default-nwtp8"
Sep 14 02:53:31.143568 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:31.143505    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:31.210704 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:31.210665    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:31.215411 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:31.215382    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvb87\" (UniqueName: \"kubernetes.io/projected/e30169cd-3ac5-476e-bdae-555456b42f7a-kube-api-access-wvb87\") pod \"ingress-canary-jhnxs\" (UID: \"e30169cd-3ac5-476e-bdae-555456b42f7a\") " pod="openshift-ingress-canary/ingress-canary-jhnxs"
Sep 14 02:53:31.215491 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:31.215430    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c9ce7325-f74d-4a0f-8dec-51099aaa1cf3-metrics-tls\") pod \"dns-default-nwtp8\" (UID: \"c9ce7325-f74d-4a0f-8dec-51099aaa1cf3\") " pod="openshift-dns/dns-default-nwtp8"
Sep 14 02:53:31.215491 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:31.215483    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/09f4cb2d-a780-460e-aa2d-86eca496c0d1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d7hms\" (UID: \"09f4cb2d-a780-460e-aa2d-86eca496c0d1\") " pod="openshift-insights/insights-runtime-extractor-d7hms"
Sep 14 02:53:31.215555 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:31.215513    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e30169cd-3ac5-476e-bdae-555456b42f7a-cert\") pod \"ingress-canary-jhnxs\" (UID: \"e30169cd-3ac5-476e-bdae-555456b42f7a\") " pod="openshift-ingress-canary/ingress-canary-jhnxs"
Sep 14 02:53:31.215555 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:31.215530    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c9ce7325-f74d-4a0f-8dec-51099aaa1cf3-config-volume\") pod \"dns-default-nwtp8\" (UID: \"c9ce7325-f74d-4a0f-8dec-51099aaa1cf3\") " pod="openshift-dns/dns-default-nwtp8"
Sep 14 02:53:31.215555 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:31.215550    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/09f4cb2d-a780-460e-aa2d-86eca496c0d1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d7hms\" (UID: \"09f4cb2d-a780-460e-aa2d-86eca496c0d1\") " pod="openshift-insights/insights-runtime-extractor-d7hms"
Sep 14 02:53:31.215677 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:31.215578    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9ce7325-f74d-4a0f-8dec-51099aaa1cf3-tmp-dir\") pod \"dns-default-nwtp8\" (UID: \"c9ce7325-f74d-4a0f-8dec-51099aaa1cf3\") " pod="openshift-dns/dns-default-nwtp8"
Sep 14 02:53:31.215677 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:31.215605    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/09f4cb2d-a780-460e-aa2d-86eca496c0d1-crio-socket\") pod \"insights-runtime-extractor-d7hms\" (UID: \"09f4cb2d-a780-460e-aa2d-86eca496c0d1\") " pod="openshift-insights/insights-runtime-extractor-d7hms"
Sep 14 02:53:31.215677 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:31.215654    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzq74\" (UniqueName: \"kubernetes.io/projected/09f4cb2d-a780-460e-aa2d-86eca496c0d1-kube-api-access-nzq74\") pod \"insights-runtime-extractor-d7hms\" (UID: \"09f4cb2d-a780-460e-aa2d-86eca496c0d1\") " pod="openshift-insights/insights-runtime-extractor-d7hms"
Sep 14 02:53:31.215797 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:31.215781    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/09f4cb2d-a780-460e-aa2d-86eca496c0d1-data-volume\") pod \"insights-runtime-extractor-d7hms\" (UID: \"09f4cb2d-a780-460e-aa2d-86eca496c0d1\") " pod="openshift-insights/insights-runtime-extractor-d7hms"
Sep 14 02:53:31.215839 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:31.215815    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4sfk\" (UniqueName: \"kubernetes.io/projected/c9ce7325-f74d-4a0f-8dec-51099aaa1cf3-kube-api-access-s4sfk\") pod \"dns-default-nwtp8\" (UID: \"c9ce7325-f74d-4a0f-8dec-51099aaa1cf3\") " pod="openshift-dns/dns-default-nwtp8"
Sep 14 02:53:31.215922 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:31.215901    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9ce7325-f74d-4a0f-8dec-51099aaa1cf3-tmp-dir\") pod \"dns-default-nwtp8\" (UID: \"c9ce7325-f74d-4a0f-8dec-51099aaa1cf3\") " pod="openshift-dns/dns-default-nwtp8"
Sep 14 02:53:31.216224 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:31.216200    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c9ce7325-f74d-4a0f-8dec-51099aaa1cf3-config-volume\") pod \"dns-default-nwtp8\" (UID: \"c9ce7325-f74d-4a0f-8dec-51099aaa1cf3\") " pod="openshift-dns/dns-default-nwtp8"
Sep 14 02:53:31.219216 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:31.219198    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c9ce7325-f74d-4a0f-8dec-51099aaa1cf3-metrics-tls\") pod \"dns-default-nwtp8\" (UID: \"c9ce7325-f74d-4a0f-8dec-51099aaa1cf3\") " pod="openshift-dns/dns-default-nwtp8"
Sep 14 02:53:31.219356 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:31.219339    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e30169cd-3ac5-476e-bdae-555456b42f7a-cert\") pod \"ingress-canary-jhnxs\" (UID: \"e30169cd-3ac5-476e-bdae-555456b42f7a\") " pod="openshift-ingress-canary/ingress-canary-jhnxs"
Sep 14 02:53:31.223993 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:31.223970    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvb87\" (UniqueName: \"kubernetes.io/projected/e30169cd-3ac5-476e-bdae-555456b42f7a-kube-api-access-wvb87\") pod \"ingress-canary-jhnxs\" (UID: \"e30169cd-3ac5-476e-bdae-555456b42f7a\") " pod="openshift-ingress-canary/ingress-canary-jhnxs"
Sep 14 02:53:31.224133 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:31.224117    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4sfk\" (UniqueName: \"kubernetes.io/projected/c9ce7325-f74d-4a0f-8dec-51099aaa1cf3-kube-api-access-s4sfk\") pod \"dns-default-nwtp8\" (UID: \"c9ce7325-f74d-4a0f-8dec-51099aaa1cf3\") " pod="openshift-dns/dns-default-nwtp8"
Sep 14 02:53:31.246051 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:31.246020    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nwtp8"
Sep 14 02:53:31.254992 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:31.254967    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jhnxs"
Sep 14 02:53:31.312685 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:31.312649    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:31.316713 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:31.316095    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/09f4cb2d-a780-460e-aa2d-86eca496c0d1-data-volume\") pod \"insights-runtime-extractor-d7hms\" (UID: \"09f4cb2d-a780-460e-aa2d-86eca496c0d1\") " pod="openshift-insights/insights-runtime-extractor-d7hms"
Sep 14 02:53:31.316713 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:31.316180    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/09f4cb2d-a780-460e-aa2d-86eca496c0d1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d7hms\" (UID: \"09f4cb2d-a780-460e-aa2d-86eca496c0d1\") " pod="openshift-insights/insights-runtime-extractor-d7hms"
Sep 14 02:53:31.316713 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:31.316222    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/09f4cb2d-a780-460e-aa2d-86eca496c0d1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d7hms\" (UID: \"09f4cb2d-a780-460e-aa2d-86eca496c0d1\") " pod="openshift-insights/insights-runtime-extractor-d7hms"
Sep 14 02:53:31.316713 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:31.316255    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/09f4cb2d-a780-460e-aa2d-86eca496c0d1-crio-socket\") pod \"insights-runtime-extractor-d7hms\" (UID: \"09f4cb2d-a780-460e-aa2d-86eca496c0d1\") " pod="openshift-insights/insights-runtime-extractor-d7hms"
Sep 14 02:53:31.316713 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:31.316290    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzq74\" (UniqueName: \"kubernetes.io/projected/09f4cb2d-a780-460e-aa2d-86eca496c0d1-kube-api-access-nzq74\") pod \"insights-runtime-extractor-d7hms\" (UID: \"09f4cb2d-a780-460e-aa2d-86eca496c0d1\") " pod="openshift-insights/insights-runtime-extractor-d7hms"
Sep 14 02:53:31.317609 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:31.317487    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/09f4cb2d-a780-460e-aa2d-86eca496c0d1-crio-socket\") pod \"insights-runtime-extractor-d7hms\" (UID: \"09f4cb2d-a780-460e-aa2d-86eca496c0d1\") " pod="openshift-insights/insights-runtime-extractor-d7hms"
Sep 14 02:53:31.318067 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:31.317781    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/09f4cb2d-a780-460e-aa2d-86eca496c0d1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d7hms\" (UID: \"09f4cb2d-a780-460e-aa2d-86eca496c0d1\") " pod="openshift-insights/insights-runtime-extractor-d7hms"
Sep 14 02:53:31.318067 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:31.318007    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/09f4cb2d-a780-460e-aa2d-86eca496c0d1-data-volume\") pod \"insights-runtime-extractor-d7hms\" (UID: \"09f4cb2d-a780-460e-aa2d-86eca496c0d1\") " pod="openshift-insights/insights-runtime-extractor-d7hms"
Sep 14 02:53:31.320714 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:31.320651    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/09f4cb2d-a780-460e-aa2d-86eca496c0d1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d7hms\" (UID: \"09f4cb2d-a780-460e-aa2d-86eca496c0d1\") " pod="openshift-insights/insights-runtime-extractor-d7hms"
Sep 14 02:53:31.325108 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:31.325084    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzq74\" (UniqueName: \"kubernetes.io/projected/09f4cb2d-a780-460e-aa2d-86eca496c0d1-kube-api-access-nzq74\") pod \"insights-runtime-extractor-d7hms\" (UID: \"09f4cb2d-a780-460e-aa2d-86eca496c0d1\") " pod="openshift-insights/insights-runtime-extractor-d7hms"
Sep 14 02:53:31.350981 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:31.350961    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-d7hms"
Sep 14 02:53:31.523866 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:31.523838    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jhnxs"]
Sep 14 02:53:31.526174 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:31.526150    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nwtp8"]
Sep 14 02:53:31.527155 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:53:31.527087    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode30169cd_3ac5_476e_bdae_555456b42f7a.slice/crio-04b8abceffeae76debf4df80a482df30089ca9b373685621fc219fb3ba96c403 WatchSource:0}: Error finding container 04b8abceffeae76debf4df80a482df30089ca9b373685621fc219fb3ba96c403: Status 404 returned error can't find the container with id 04b8abceffeae76debf4df80a482df30089ca9b373685621fc219fb3ba96c403
Sep 14 02:53:31.529223 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:53:31.529154    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9ce7325_f74d_4a0f_8dec_51099aaa1cf3.slice/crio-7e0445064b94fd0f77210a1971e187ba5ffab08260dfb791ed4a15f58b7e0d6e WatchSource:0}: Error finding container 7e0445064b94fd0f77210a1971e187ba5ffab08260dfb791ed4a15f58b7e0d6e: Status 404 returned error can't find the container with id 7e0445064b94fd0f77210a1971e187ba5ffab08260dfb791ed4a15f58b7e0d6e
Sep 14 02:53:31.616513 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:31.616482    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-d7hms"]
Sep 14 02:53:31.619135 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:53:31.619109    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09f4cb2d_a780_460e_aa2d_86eca496c0d1.slice/crio-5efbe17a57e862435d0fe0bb51f6d60a4bfa0065c5506046bd43896ba3e7497a WatchSource:0}: Error finding container 5efbe17a57e862435d0fe0bb51f6d60a4bfa0065c5506046bd43896ba3e7497a: Status 404 returned error can't find the container with id 5efbe17a57e862435d0fe0bb51f6d60a4bfa0065c5506046bd43896ba3e7497a
Sep 14 02:53:32.324023 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:32.323907    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d7hms" event={"ID":"09f4cb2d-a780-460e-aa2d-86eca496c0d1","Type":"ContainerStarted","Data":"8b295460aa5e5e9775fd57337733ad3b77bbb38ac9a24285f91fb90eac84ca0d"}
Sep 14 02:53:32.324023 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:32.323981    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d7hms" event={"ID":"09f4cb2d-a780-460e-aa2d-86eca496c0d1","Type":"ContainerStarted","Data":"5efbe17a57e862435d0fe0bb51f6d60a4bfa0065c5506046bd43896ba3e7497a"}
Sep 14 02:53:32.325256 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:32.325229    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jhnxs" event={"ID":"e30169cd-3ac5-476e-bdae-555456b42f7a","Type":"ContainerStarted","Data":"04b8abceffeae76debf4df80a482df30089ca9b373685621fc219fb3ba96c403"}
Sep 14 02:53:32.326639 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:32.326615    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nwtp8" event={"ID":"c9ce7325-f74d-4a0f-8dec-51099aaa1cf3","Type":"ContainerStarted","Data":"7e0445064b94fd0f77210a1971e187ba5ffab08260dfb791ed4a15f58b7e0d6e"}
Sep 14 02:53:33.849729 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:33.849698    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:33.914008 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:33.913974    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:34.236711 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:34.236649    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-66f88cd956-494g9"]
Sep 14 02:53:34.239979 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:34.239951    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66f88cd956-494g9"
Sep 14 02:53:34.242890 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:34.242867    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 14 02:53:34.244602 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:34.244015    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 14 02:53:34.244602 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:34.244048    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 14 02:53:34.244602 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:34.244134    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 14 02:53:34.244602 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:34.244046    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 14 02:53:34.244602 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:34.244264    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-xzm62\""
Sep 14 02:53:34.244602 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:34.244360    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 14 02:53:34.244602 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:34.244443    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 14 02:53:34.249757 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:34.249735    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 14 02:53:34.250958 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:34.250942    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66f88cd956-494g9"]
Sep 14 02:53:34.333513 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:34.333480    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d7hms" event={"ID":"09f4cb2d-a780-460e-aa2d-86eca496c0d1","Type":"ContainerStarted","Data":"f9e995daf93e113d84d160ef80d11f483bc93492b99ec07ab6e4f03701eacb24"}
Sep 14 02:53:34.333830 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:34.333811    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4-console-serving-cert\") pod \"console-66f88cd956-494g9\" (UID: \"91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4\") " pod="openshift-console/console-66f88cd956-494g9"
Sep 14 02:53:34.333921 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:34.333847    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4-oauth-serving-cert\") pod \"console-66f88cd956-494g9\" (UID: \"91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4\") " pod="openshift-console/console-66f88cd956-494g9"
Sep 14 02:53:34.333981 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:34.333959    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4-console-config\") pod \"console-66f88cd956-494g9\" (UID: \"91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4\") " pod="openshift-console/console-66f88cd956-494g9"
Sep 14 02:53:34.334035 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:34.333993    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4-service-ca\") pod \"console-66f88cd956-494g9\" (UID: \"91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4\") " pod="openshift-console/console-66f88cd956-494g9"
Sep 14 02:53:34.334092 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:34.334043    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4-console-oauth-config\") pod \"console-66f88cd956-494g9\" (UID: \"91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4\") " pod="openshift-console/console-66f88cd956-494g9"
Sep 14 02:53:34.334092 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:34.334073    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4-trusted-ca-bundle\") pod \"console-66f88cd956-494g9\" (UID: \"91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4\") " pod="openshift-console/console-66f88cd956-494g9"
Sep 14 02:53:34.334092 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:34.334088    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfhg8\" (UniqueName: \"kubernetes.io/projected/91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4-kube-api-access-tfhg8\") pod \"console-66f88cd956-494g9\" (UID: \"91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4\") " pod="openshift-console/console-66f88cd956-494g9"
Sep 14 02:53:34.335088 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:34.335057    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jhnxs" event={"ID":"e30169cd-3ac5-476e-bdae-555456b42f7a","Type":"ContainerStarted","Data":"491b6d787ebdaaa0cde677863c7934e3a21dcfbab215bc47b4a589f5fce45234"}
Sep 14 02:53:34.336817 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:34.336797    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nwtp8" event={"ID":"c9ce7325-f74d-4a0f-8dec-51099aaa1cf3","Type":"ContainerStarted","Data":"a7d87cd93ec983a0bf0601487af5d80bc73422ee9bda5b4522d444629c9efadc"}
Sep 14 02:53:34.336941 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:34.336833    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nwtp8" event={"ID":"c9ce7325-f74d-4a0f-8dec-51099aaa1cf3","Type":"ContainerStarted","Data":"40b6cc6b5671dd0581d6c8787ce113c1b6c77e329bd06822ca37070134f53243"}
Sep 14 02:53:34.336941 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:34.336920    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-nwtp8"
Sep 14 02:53:34.352391 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:34.352307    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-jhnxs" podStartSLOduration=1.769942274 podStartE2EDuration="4.352289976s" totalImagesPullingTime="2.582347702s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:53:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:53:34.352184445 +0000 UTC m=+59.733268644" watchObservedRunningTime="2026-09-14 02:53:34.352289976 +0000 UTC m=+59.733374181"
Sep 14 02:53:34.373174 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:34.373136    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-nwtp8" podStartSLOduration=1.798443923 podStartE2EDuration="4.37312179s" totalImagesPullingTime="2.574677867s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:53:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:53:34.372658746 +0000 UTC m=+59.753742945" watchObservedRunningTime="2026-09-14 02:53:34.37312179 +0000 UTC m=+59.754206026"
Sep 14 02:53:34.435177 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:34.435148    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4-console-config\") pod \"console-66f88cd956-494g9\" (UID: \"91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4\") " pod="openshift-console/console-66f88cd956-494g9"
Sep 14 02:53:34.435177 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:34.435179    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4-service-ca\") pod \"console-66f88cd956-494g9\" (UID: \"91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4\") " pod="openshift-console/console-66f88cd956-494g9"
Sep 14 02:53:34.435436 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:34.435206    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4-console-oauth-config\") pod \"console-66f88cd956-494g9\" (UID: \"91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4\") " pod="openshift-console/console-66f88cd956-494g9"
Sep 14 02:53:34.435436 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:34.435224    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4-trusted-ca-bundle\") pod \"console-66f88cd956-494g9\" (UID: \"91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4\") " pod="openshift-console/console-66f88cd956-494g9"
Sep 14 02:53:34.435436 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:34.435240    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfhg8\" (UniqueName: \"kubernetes.io/projected/91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4-kube-api-access-tfhg8\") pod \"console-66f88cd956-494g9\" (UID: \"91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4\") " pod="openshift-console/console-66f88cd956-494g9"
Sep 14 02:53:34.435436 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:34.435261    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4-console-serving-cert\") pod \"console-66f88cd956-494g9\" (UID: \"91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4\") " pod="openshift-console/console-66f88cd956-494g9"
Sep 14 02:53:34.435843 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:34.435798    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4-oauth-serving-cert\") pod \"console-66f88cd956-494g9\" (UID: \"91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4\") " pod="openshift-console/console-66f88cd956-494g9"
Sep 14 02:53:34.436076 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:34.436049    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4-console-config\") pod \"console-66f88cd956-494g9\" (UID: \"91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4\") " pod="openshift-console/console-66f88cd956-494g9"
Sep 14 02:53:34.436567 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:34.436544    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4-service-ca\") pod \"console-66f88cd956-494g9\" (UID: \"91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4\") " pod="openshift-console/console-66f88cd956-494g9"
Sep 14 02:53:34.436692 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:34.436671    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4-oauth-serving-cert\") pod \"console-66f88cd956-494g9\" (UID: \"91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4\") " pod="openshift-console/console-66f88cd956-494g9"
Sep 14 02:53:34.436732 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:34.436708    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4-trusted-ca-bundle\") pod \"console-66f88cd956-494g9\" (UID: \"91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4\") " pod="openshift-console/console-66f88cd956-494g9"
Sep 14 02:53:34.438289 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:34.438267    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4-console-oauth-config\") pod \"console-66f88cd956-494g9\" (UID: \"91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4\") " pod="openshift-console/console-66f88cd956-494g9"
Sep 14 02:53:34.438491 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:34.438469    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4-console-serving-cert\") pod \"console-66f88cd956-494g9\" (UID: \"91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4\") " pod="openshift-console/console-66f88cd956-494g9"
Sep 14 02:53:34.444260 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:34.444242    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfhg8\" (UniqueName: \"kubernetes.io/projected/91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4-kube-api-access-tfhg8\") pod \"console-66f88cd956-494g9\" (UID: \"91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4\") " pod="openshift-console/console-66f88cd956-494g9"
Sep 14 02:53:34.562551 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:34.562520    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66f88cd956-494g9"
Sep 14 02:53:34.829911 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:34.829879    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66f88cd956-494g9"]
Sep 14 02:53:34.833366 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:53:34.833333    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91a8ab63_3f50_4e46_a9c6_0bb59e2d45a4.slice/crio-68235fa15dddd381debdeabc5b3a518a4c47503999f9db852cb808e2f78cc7e9 WatchSource:0}: Error finding container 68235fa15dddd381debdeabc5b3a518a4c47503999f9db852cb808e2f78cc7e9: Status 404 returned error can't find the container with id 68235fa15dddd381debdeabc5b3a518a4c47503999f9db852cb808e2f78cc7e9
Sep 14 02:53:35.264682 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:35.264651    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qm7jd"
Sep 14 02:53:35.340944 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:35.340905    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66f88cd956-494g9" event={"ID":"91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4","Type":"ContainerStarted","Data":"68235fa15dddd381debdeabc5b3a518a4c47503999f9db852cb808e2f78cc7e9"}
Sep 14 02:53:35.365069 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:35.365039    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:35.461719 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:35.461676    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:35.559571 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:35.559518    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:35.623831 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:35.623789    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:36.032840 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:36.032795    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:36.077999 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:36.077966    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-5gzd6"]
Sep 14 02:53:36.081270 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:36.081241    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-5gzd6"
Sep 14 02:53:36.085015 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:36.084989    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 14 02:53:36.085015 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:36.085003    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 14 02:53:36.085176 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:36.085013    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 14 02:53:36.086098 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:36.085913    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 14 02:53:36.086098 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:36.086026    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-b6zfv\""
Sep 14 02:53:36.086238 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:36.086199    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 14 02:53:36.092220 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:36.092184    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-5gzd6"]
Sep 14 02:53:36.115737 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:36.115663    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:36.147925 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:36.147894    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/660447a7-e634-4653-95f6-736d85f5bdd9-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-5gzd6\" (UID: \"660447a7-e634-4653-95f6-736d85f5bdd9\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-5gzd6"
Sep 14 02:53:36.148097 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:36.147933    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/660447a7-e634-4653-95f6-736d85f5bdd9-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-5gzd6\" (UID: \"660447a7-e634-4653-95f6-736d85f5bdd9\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-5gzd6"
Sep 14 02:53:36.148097 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:36.147956    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqdtj\" (UniqueName: \"kubernetes.io/projected/660447a7-e634-4653-95f6-736d85f5bdd9-kube-api-access-bqdtj\") pod \"prometheus-operator-68bcd7f987-5gzd6\" (UID: \"660447a7-e634-4653-95f6-736d85f5bdd9\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-5gzd6"
Sep 14 02:53:36.148097 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:36.148016    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/660447a7-e634-4653-95f6-736d85f5bdd9-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-5gzd6\" (UID: \"660447a7-e634-4653-95f6-736d85f5bdd9\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-5gzd6"
Sep 14 02:53:36.218718 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:36.218666    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:36.248773 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:36.248734    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/660447a7-e634-4653-95f6-736d85f5bdd9-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-5gzd6\" (UID: \"660447a7-e634-4653-95f6-736d85f5bdd9\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-5gzd6"
Sep 14 02:53:36.248773 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:36.248778    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqdtj\" (UniqueName: \"kubernetes.io/projected/660447a7-e634-4653-95f6-736d85f5bdd9-kube-api-access-bqdtj\") pod \"prometheus-operator-68bcd7f987-5gzd6\" (UID: \"660447a7-e634-4653-95f6-736d85f5bdd9\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-5gzd6"
Sep 14 02:53:36.248994 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:36.248815    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/660447a7-e634-4653-95f6-736d85f5bdd9-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-5gzd6\" (UID: \"660447a7-e634-4653-95f6-736d85f5bdd9\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-5gzd6"
Sep 14 02:53:36.248994 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:36.248877    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/660447a7-e634-4653-95f6-736d85f5bdd9-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-5gzd6\" (UID: \"660447a7-e634-4653-95f6-736d85f5bdd9\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-5gzd6"
Sep 14 02:53:36.249542 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:36.249513    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/660447a7-e634-4653-95f6-736d85f5bdd9-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-5gzd6\" (UID: \"660447a7-e634-4653-95f6-736d85f5bdd9\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-5gzd6"
Sep 14 02:53:36.251534 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:36.251508    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/660447a7-e634-4653-95f6-736d85f5bdd9-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-5gzd6\" (UID: \"660447a7-e634-4653-95f6-736d85f5bdd9\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-5gzd6"
Sep 14 02:53:36.251652 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:36.251590    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/660447a7-e634-4653-95f6-736d85f5bdd9-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-5gzd6\" (UID: \"660447a7-e634-4653-95f6-736d85f5bdd9\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-5gzd6"
Sep 14 02:53:36.259674 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:36.259647    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqdtj\" (UniqueName: \"kubernetes.io/projected/660447a7-e634-4653-95f6-736d85f5bdd9-kube-api-access-bqdtj\") pod \"prometheus-operator-68bcd7f987-5gzd6\" (UID: \"660447a7-e634-4653-95f6-736d85f5bdd9\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-5gzd6"
Sep 14 02:53:36.347791 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:36.347702    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d7hms" event={"ID":"09f4cb2d-a780-460e-aa2d-86eca496c0d1","Type":"ContainerStarted","Data":"59772e46b851117b755af36d5047ae63bb5ef29f76234433cc94053a8e4bdeac"}
Sep 14 02:53:36.367200 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:36.367143    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-d7hms" podStartSLOduration=1.661684513 podStartE2EDuration="5.367126171s" totalImagesPullingTime="3.705441658s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 02:53:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:53:36.366644923 +0000 UTC m=+61.747729123" watchObservedRunningTime="2026-09-14 02:53:36.367126171 +0000 UTC m=+61.748210369"
Sep 14 02:53:36.392711 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:36.392674    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-5gzd6"
Sep 14 02:53:36.640791 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:36.640704    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:36.671025 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:36.670995    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-5gzd6"]
Sep 14 02:53:36.674044 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:53:36.674019    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod660447a7_e634_4653_95f6_736d85f5bdd9.slice/crio-957fa73da34c5d6f5277a7c12974c1ae41d28c919122d73acae77f1a32194708 WatchSource:0}: Error finding container 957fa73da34c5d6f5277a7c12974c1ae41d28c919122d73acae77f1a32194708: Status 404 returned error can't find the container with id 957fa73da34c5d6f5277a7c12974c1ae41d28c919122d73acae77f1a32194708
Sep 14 02:53:36.742277 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:36.742228    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:37.355011 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:37.354955    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-5gzd6" event={"ID":"660447a7-e634-4653-95f6-736d85f5bdd9","Type":"ContainerStarted","Data":"957fa73da34c5d6f5277a7c12974c1ae41d28c919122d73acae77f1a32194708"}
Sep 14 02:53:38.360633 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:38.360595    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66f88cd956-494g9" event={"ID":"91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4","Type":"ContainerStarted","Data":"303a1b144e71b6b11ce609578ac6d11b07603db9be51b4c6de2a703ca5201522"}
Sep 14 02:53:38.727728 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:38.727642    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:39.365472 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:39.365430    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-5gzd6" event={"ID":"660447a7-e634-4653-95f6-736d85f5bdd9","Type":"ContainerStarted","Data":"1679a87379077b31ab64b664478e78166ae86c71ca57021559837b1a01650f8e"}
Sep 14 02:53:39.365941 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:39.365486    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-5gzd6" event={"ID":"660447a7-e634-4653-95f6-736d85f5bdd9","Type":"ContainerStarted","Data":"227ab8fc320fb999b81fcb4593b2c795893dc72c033517eb3fcc50dca906cd5b"}
Sep 14 02:53:39.382376 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:39.382328    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-66f88cd956-494g9" podStartSLOduration=2.463106513 podStartE2EDuration="5.382313902s" totalImagesPullingTime="2.919207389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:53:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:53:38.379563272 +0000 UTC m=+63.760647471" watchObservedRunningTime="2026-09-14 02:53:39.382313902 +0000 UTC m=+64.763398102"
Sep 14 02:53:39.382537 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:39.382434    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-5gzd6" podStartSLOduration=1.567619704 podStartE2EDuration="3.382429976s" totalImagesPullingTime="1.814810272s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:53:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:53:39.381849592 +0000 UTC m=+64.762933790" watchObservedRunningTime="2026-09-14 02:53:39.382429976 +0000 UTC m=+64.763514175"
Sep 14 02:53:40.447972 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:40.447933    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:40.548646 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:40.548608    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:40.785298 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:40.785267    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/18063d29-58fa-4599-8e9c-93c829fa0fb7-original-pull-secret\") pod \"global-pull-secret-syncer-zhkkf\" (UID: \"18063d29-58fa-4599-8e9c-93c829fa0fb7\") " pod="kube-system/global-pull-secret-syncer-zhkkf"
Sep 14 02:53:40.787990 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:40.787971    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 14 02:53:40.798007 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:40.797983    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/18063d29-58fa-4599-8e9c-93c829fa0fb7-original-pull-secret\") pod \"global-pull-secret-syncer-zhkkf\" (UID: \"18063d29-58fa-4599-8e9c-93c829fa0fb7\") " pod="kube-system/global-pull-secret-syncer-zhkkf"
Sep 14 02:53:40.885909 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:40.885866    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7029d1df-1b4b-40f6-95e1-621b70070086-metrics-certs\") pod \"network-metrics-daemon-8lf5z\" (UID: \"7029d1df-1b4b-40f6-95e1-621b70070086\") " pod="openshift-multus/network-metrics-daemon-8lf5z"
Sep 14 02:53:40.888775 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:40.888757    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 14 02:53:40.898919 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:40.898895    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7029d1df-1b4b-40f6-95e1-621b70070086-metrics-certs\") pod \"network-metrics-daemon-8lf5z\" (UID: \"7029d1df-1b4b-40f6-95e1-621b70070086\") " pod="openshift-multus/network-metrics-daemon-8lf5z"
Sep 14 02:53:40.986633 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:40.986591    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn8sw\" (UniqueName: \"kubernetes.io/projected/8804d811-79f4-4612-bf40-38c08a04d2c8-kube-api-access-tn8sw\") pod \"network-check-target-kq5nf\" (UID: \"8804d811-79f4-4612-bf40-38c08a04d2c8\") " pod="openshift-network-diagnostics/network-check-target-kq5nf"
Sep 14 02:53:40.989571 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:40.989550    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 14 02:53:40.999799 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:40.999776    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 14 02:53:41.003106 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.003086    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lx47r\""
Sep 14 02:53:41.009628 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.009610    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn8sw\" (UniqueName: \"kubernetes.io/projected/8804d811-79f4-4612-bf40-38c08a04d2c8-kube-api-access-tn8sw\") pod \"network-check-target-kq5nf\" (UID: \"8804d811-79f4-4612-bf40-38c08a04d2c8\") " pod="openshift-network-diagnostics/network-check-target-kq5nf"
Sep 14 02:53:41.011378 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.011359    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8lf5z"
Sep 14 02:53:41.011432 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.011376    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhkkf"
Sep 14 02:53:41.223563 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.223530    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:41.289966 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.289940    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zhkkf"]
Sep 14 02:53:41.292235 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:53:41.292212    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18063d29_58fa_4599_8e9c_93c829fa0fb7.slice/crio-0320065c518bb4802a46645f1e267049e77cd76257be76e8c3bd475944850f22 WatchSource:0}: Error finding container 0320065c518bb4802a46645f1e267049e77cd76257be76e8c3bd475944850f22: Status 404 returned error can't find the container with id 0320065c518bb4802a46645f1e267049e77cd76257be76e8c3bd475944850f22
Sep 14 02:53:41.293844 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.293822    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8lf5z"]
Sep 14 02:53:41.296874 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:53:41.296846    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7029d1df_1b4b_40f6_95e1_621b70070086.slice/crio-ef4c4eb9a22955f2eb51a8a9904c8e9a68a290ee35d581257cd738780122ce9c WatchSource:0}: Error finding container ef4c4eb9a22955f2eb51a8a9904c8e9a68a290ee35d581257cd738780122ce9c: Status 404 returned error can't find the container with id ef4c4eb9a22955f2eb51a8a9904c8e9a68a290ee35d581257cd738780122ce9c
Sep 14 02:53:41.309543 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.309522    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hn74x\""
Sep 14 02:53:41.317238 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.317184    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kq5nf"
Sep 14 02:53:41.338099 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.338067    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:41.376546 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.376516    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8lf5z" event={"ID":"7029d1df-1b4b-40f6-95e1-621b70070086","Type":"ContainerStarted","Data":"ef4c4eb9a22955f2eb51a8a9904c8e9a68a290ee35d581257cd738780122ce9c"}
Sep 14 02:53:41.377625 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.377603    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zhkkf" event={"ID":"18063d29-58fa-4599-8e9c-93c829fa0fb7","Type":"ContainerStarted","Data":"0320065c518bb4802a46645f1e267049e77cd76257be76e8c3bd475944850f22"}
Sep 14 02:53:41.441210 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.441177    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:41.462057 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.462032    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-qqz7z"]
Sep 14 02:53:41.464206 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.464191    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-qqz7z"
Sep 14 02:53:41.466718 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.466700    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 14 02:53:41.466807 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.466703    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 14 02:53:41.466807 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.466726    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-f2xnj\""
Sep 14 02:53:41.477677 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.477655    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-qqz7z"]
Sep 14 02:53:41.478567 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.478551    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-rrskf"]
Sep 14 02:53:41.480351 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.480335    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rrskf"
Sep 14 02:53:41.482687 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.482668    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 14 02:53:41.482770 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.482696    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 14 02:53:41.482835 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.482767    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-27h6r\""
Sep 14 02:53:41.483729 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.483713    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 14 02:53:41.490424 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.490392    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6102f536-8eb3-4666-91cd-7781a4cca236-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-qqz7z\" (UID: \"6102f536-8eb3-4666-91cd-7781a4cca236\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-qqz7z"
Sep 14 02:53:41.490489 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.490431    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6102f536-8eb3-4666-91cd-7781a4cca236-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-qqz7z\" (UID: \"6102f536-8eb3-4666-91cd-7781a4cca236\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-qqz7z"
Sep 14 02:53:41.490531 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.490481    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z2kf\" (UniqueName: \"kubernetes.io/projected/6102f536-8eb3-4666-91cd-7781a4cca236-kube-api-access-4z2kf\") pod \"openshift-state-metrics-67dcf8c979-qqz7z\" (UID: \"6102f536-8eb3-4666-91cd-7781a4cca236\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-qqz7z"
Sep 14 02:53:41.490565 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.490546    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6102f536-8eb3-4666-91cd-7781a4cca236-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-qqz7z\" (UID: \"6102f536-8eb3-4666-91cd-7781a4cca236\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-qqz7z"
Sep 14 02:53:41.498540 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.498522    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-rrskf"]
Sep 14 02:53:41.514499 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.514475    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-tlvfd"]
Sep 14 02:53:41.516337 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.516322    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tlvfd"
Sep 14 02:53:41.523524 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.523501    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 14 02:53:41.523611 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.523578    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 14 02:53:41.523665 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.523634    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xb6b2\""
Sep 14 02:53:41.523713 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.523639    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 14 02:53:41.543522 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.543497    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:41.586673 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.586600    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kq5nf"]
Sep 14 02:53:41.589125 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:53:41.589102    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8804d811_79f4_4612_bf40_38c08a04d2c8.slice/crio-deeb0add4fd8c6c35f51c774cce9931ea4119d02396bb24181ff143e60e8c379 WatchSource:0}: Error finding container deeb0add4fd8c6c35f51c774cce9931ea4119d02396bb24181ff143e60e8c379: Status 404 returned error can't find the container with id deeb0add4fd8c6c35f51c774cce9931ea4119d02396bb24181ff143e60e8c379
Sep 14 02:53:41.590968 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.590915    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z2kf\" (UniqueName: \"kubernetes.io/projected/6102f536-8eb3-4666-91cd-7781a4cca236-kube-api-access-4z2kf\") pod \"openshift-state-metrics-67dcf8c979-qqz7z\" (UID: \"6102f536-8eb3-4666-91cd-7781a4cca236\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-qqz7z"
Sep 14 02:53:41.590968 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.590943    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/68ff2110-81b2-469c-ba95-dab13090e6a8-node-exporter-tls\") pod \"node-exporter-tlvfd\" (UID: \"68ff2110-81b2-469c-ba95-dab13090e6a8\") " pod="openshift-monitoring/node-exporter-tlvfd"
Sep 14 02:53:41.591063 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.590971    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/09b1dd75-0309-48b0-a9dd-2a997d1444c4-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-rrskf\" (UID: \"09b1dd75-0309-48b0-a9dd-2a997d1444c4\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rrskf"
Sep 14 02:53:41.591063 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.590997    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/68ff2110-81b2-469c-ba95-dab13090e6a8-node-exporter-accelerators-collector-config\") pod \"node-exporter-tlvfd\" (UID: \"68ff2110-81b2-469c-ba95-dab13090e6a8\") " pod="openshift-monitoring/node-exporter-tlvfd"
Sep 14 02:53:41.591063 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.591019    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/09b1dd75-0309-48b0-a9dd-2a997d1444c4-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-rrskf\" (UID: \"09b1dd75-0309-48b0-a9dd-2a997d1444c4\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rrskf"
Sep 14 02:53:41.591063 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.591043    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/68ff2110-81b2-469c-ba95-dab13090e6a8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tlvfd\" (UID: \"68ff2110-81b2-469c-ba95-dab13090e6a8\") " pod="openshift-monitoring/node-exporter-tlvfd"
Sep 14 02:53:41.591286 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.591142    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/68ff2110-81b2-469c-ba95-dab13090e6a8-node-exporter-wtmp\") pod \"node-exporter-tlvfd\" (UID: \"68ff2110-81b2-469c-ba95-dab13090e6a8\") " pod="openshift-monitoring/node-exporter-tlvfd"
Sep 14 02:53:41.591286 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.591205    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/09b1dd75-0309-48b0-a9dd-2a997d1444c4-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-rrskf\" (UID: \"09b1dd75-0309-48b0-a9dd-2a997d1444c4\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rrskf"
Sep 14 02:53:41.591286 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.591230    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9grk4\" (UniqueName: \"kubernetes.io/projected/09b1dd75-0309-48b0-a9dd-2a997d1444c4-kube-api-access-9grk4\") pod \"kube-state-metrics-59fbdd77d-rrskf\" (UID: \"09b1dd75-0309-48b0-a9dd-2a997d1444c4\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rrskf"
Sep 14 02:53:41.591286 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.591269    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6102f536-8eb3-4666-91cd-7781a4cca236-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-qqz7z\" (UID: \"6102f536-8eb3-4666-91cd-7781a4cca236\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-qqz7z"
Sep 14 02:53:41.591434 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.591291    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88p96\" (UniqueName: \"kubernetes.io/projected/68ff2110-81b2-469c-ba95-dab13090e6a8-kube-api-access-88p96\") pod \"node-exporter-tlvfd\" (UID: \"68ff2110-81b2-469c-ba95-dab13090e6a8\") " pod="openshift-monitoring/node-exporter-tlvfd"
Sep 14 02:53:41.591434 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.591309    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/68ff2110-81b2-469c-ba95-dab13090e6a8-metrics-client-ca\") pod \"node-exporter-tlvfd\" (UID: \"68ff2110-81b2-469c-ba95-dab13090e6a8\") " pod="openshift-monitoring/node-exporter-tlvfd"
Sep 14 02:53:41.591434 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.591325    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/68ff2110-81b2-469c-ba95-dab13090e6a8-node-exporter-textfile\") pod \"node-exporter-tlvfd\" (UID: \"68ff2110-81b2-469c-ba95-dab13090e6a8\") " pod="openshift-monitoring/node-exporter-tlvfd"
Sep 14 02:53:41.591434 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.591356    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/68ff2110-81b2-469c-ba95-dab13090e6a8-sys\") pod \"node-exporter-tlvfd\" (UID: \"68ff2110-81b2-469c-ba95-dab13090e6a8\") " pod="openshift-monitoring/node-exporter-tlvfd"
Sep 14 02:53:41.591434 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.591383    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/09b1dd75-0309-48b0-a9dd-2a997d1444c4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-rrskf\" (UID: \"09b1dd75-0309-48b0-a9dd-2a997d1444c4\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rrskf"
Sep 14 02:53:41.591434 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.591425    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6102f536-8eb3-4666-91cd-7781a4cca236-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-qqz7z\" (UID: \"6102f536-8eb3-4666-91cd-7781a4cca236\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-qqz7z"
Sep 14 02:53:41.591691 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.591451    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/09b1dd75-0309-48b0-a9dd-2a997d1444c4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-rrskf\" (UID: \"09b1dd75-0309-48b0-a9dd-2a997d1444c4\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rrskf"
Sep 14 02:53:41.591691 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.591555    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6102f536-8eb3-4666-91cd-7781a4cca236-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-qqz7z\" (UID: \"6102f536-8eb3-4666-91cd-7781a4cca236\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-qqz7z"
Sep 14 02:53:41.591691 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.591577    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/68ff2110-81b2-469c-ba95-dab13090e6a8-root\") pod \"node-exporter-tlvfd\" (UID: \"68ff2110-81b2-469c-ba95-dab13090e6a8\") " pod="openshift-monitoring/node-exporter-tlvfd"
Sep 14 02:53:41.591691 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:53:41.591686    2564 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 14 02:53:41.591873 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:53:41.591738    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6102f536-8eb3-4666-91cd-7781a4cca236-openshift-state-metrics-tls podName:6102f536-8eb3-4666-91cd-7781a4cca236 nodeName:}" failed. No retries permitted until 2026-09-14 02:53:42.091723064 +0000 UTC m=+67.472807243 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/6102f536-8eb3-4666-91cd-7781a4cca236-openshift-state-metrics-tls") pod "openshift-state-metrics-67dcf8c979-qqz7z" (UID: "6102f536-8eb3-4666-91cd-7781a4cca236") : secret "openshift-state-metrics-tls" not found
Sep 14 02:53:41.592043 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.591912    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6102f536-8eb3-4666-91cd-7781a4cca236-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-qqz7z\" (UID: \"6102f536-8eb3-4666-91cd-7781a4cca236\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-qqz7z"
Sep 14 02:53:41.593900 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.593872    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6102f536-8eb3-4666-91cd-7781a4cca236-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-qqz7z\" (UID: \"6102f536-8eb3-4666-91cd-7781a4cca236\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-qqz7z"
Sep 14 02:53:41.600103 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.600079    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z2kf\" (UniqueName: \"kubernetes.io/projected/6102f536-8eb3-4666-91cd-7781a4cca236-kube-api-access-4z2kf\") pod \"openshift-state-metrics-67dcf8c979-qqz7z\" (UID: \"6102f536-8eb3-4666-91cd-7781a4cca236\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-qqz7z"
Sep 14 02:53:41.692355 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.692311    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88p96\" (UniqueName: \"kubernetes.io/projected/68ff2110-81b2-469c-ba95-dab13090e6a8-kube-api-access-88p96\") pod \"node-exporter-tlvfd\" (UID: \"68ff2110-81b2-469c-ba95-dab13090e6a8\") " pod="openshift-monitoring/node-exporter-tlvfd"
Sep 14 02:53:41.692355 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.692361    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/68ff2110-81b2-469c-ba95-dab13090e6a8-metrics-client-ca\") pod \"node-exporter-tlvfd\" (UID: \"68ff2110-81b2-469c-ba95-dab13090e6a8\") " pod="openshift-monitoring/node-exporter-tlvfd"
Sep 14 02:53:41.692638 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.692384    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/68ff2110-81b2-469c-ba95-dab13090e6a8-node-exporter-textfile\") pod \"node-exporter-tlvfd\" (UID: \"68ff2110-81b2-469c-ba95-dab13090e6a8\") " pod="openshift-monitoring/node-exporter-tlvfd"
Sep 14 02:53:41.692638 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.692436    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/68ff2110-81b2-469c-ba95-dab13090e6a8-sys\") pod \"node-exporter-tlvfd\" (UID: \"68ff2110-81b2-469c-ba95-dab13090e6a8\") " pod="openshift-monitoring/node-exporter-tlvfd"
Sep 14 02:53:41.692638 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.692463    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/09b1dd75-0309-48b0-a9dd-2a997d1444c4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-rrskf\" (UID: \"09b1dd75-0309-48b0-a9dd-2a997d1444c4\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rrskf"
Sep 14 02:53:41.692638 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.692491    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/09b1dd75-0309-48b0-a9dd-2a997d1444c4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-rrskf\" (UID: \"09b1dd75-0309-48b0-a9dd-2a997d1444c4\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rrskf"
Sep 14 02:53:41.692638 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.692530    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/68ff2110-81b2-469c-ba95-dab13090e6a8-root\") pod \"node-exporter-tlvfd\" (UID: \"68ff2110-81b2-469c-ba95-dab13090e6a8\") " pod="openshift-monitoring/node-exporter-tlvfd"
Sep 14 02:53:41.692638 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.692539    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/68ff2110-81b2-469c-ba95-dab13090e6a8-sys\") pod \"node-exporter-tlvfd\" (UID: \"68ff2110-81b2-469c-ba95-dab13090e6a8\") " pod="openshift-monitoring/node-exporter-tlvfd"
Sep 14 02:53:41.692638 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.692564    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/68ff2110-81b2-469c-ba95-dab13090e6a8-node-exporter-tls\") pod \"node-exporter-tlvfd\" (UID: \"68ff2110-81b2-469c-ba95-dab13090e6a8\") " pod="openshift-monitoring/node-exporter-tlvfd"
Sep 14 02:53:41.692638 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.692595    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/09b1dd75-0309-48b0-a9dd-2a997d1444c4-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-rrskf\" (UID: \"09b1dd75-0309-48b0-a9dd-2a997d1444c4\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rrskf"
Sep 14 02:53:41.692638 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.692626    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/68ff2110-81b2-469c-ba95-dab13090e6a8-node-exporter-accelerators-collector-config\") pod \"node-exporter-tlvfd\" (UID: \"68ff2110-81b2-469c-ba95-dab13090e6a8\") " pod="openshift-monitoring/node-exporter-tlvfd"
Sep 14 02:53:41.692638 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.692625    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/68ff2110-81b2-469c-ba95-dab13090e6a8-root\") pod \"node-exporter-tlvfd\" (UID: \"68ff2110-81b2-469c-ba95-dab13090e6a8\") " pod="openshift-monitoring/node-exporter-tlvfd"
Sep 14 02:53:41.693120 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.692658    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/09b1dd75-0309-48b0-a9dd-2a997d1444c4-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-rrskf\" (UID: \"09b1dd75-0309-48b0-a9dd-2a997d1444c4\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rrskf"
Sep 14 02:53:41.693120 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.692691    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/68ff2110-81b2-469c-ba95-dab13090e6a8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tlvfd\" (UID: \"68ff2110-81b2-469c-ba95-dab13090e6a8\") " pod="openshift-monitoring/node-exporter-tlvfd"
Sep 14 02:53:41.693120 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:53:41.692712    2564 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 14 02:53:41.693120 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.692716    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/68ff2110-81b2-469c-ba95-dab13090e6a8-node-exporter-wtmp\") pod \"node-exporter-tlvfd\" (UID: \"68ff2110-81b2-469c-ba95-dab13090e6a8\") " pod="openshift-monitoring/node-exporter-tlvfd"
Sep 14 02:53:41.693120 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:53:41.692732    2564 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 14 02:53:41.693120 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.692747    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/09b1dd75-0309-48b0-a9dd-2a997d1444c4-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-rrskf\" (UID: \"09b1dd75-0309-48b0-a9dd-2a997d1444c4\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rrskf"
Sep 14 02:53:41.693120 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:53:41.692770    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68ff2110-81b2-469c-ba95-dab13090e6a8-node-exporter-tls podName:68ff2110-81b2-469c-ba95-dab13090e6a8 nodeName:}" failed. No retries permitted until 2026-09-14 02:53:42.192750363 +0000 UTC m=+67.573834552 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/68ff2110-81b2-469c-ba95-dab13090e6a8-node-exporter-tls") pod "node-exporter-tlvfd" (UID: "68ff2110-81b2-469c-ba95-dab13090e6a8") : secret "node-exporter-tls" not found
Sep 14 02:53:41.693120 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:53:41.692973    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09b1dd75-0309-48b0-a9dd-2a997d1444c4-kube-state-metrics-tls podName:09b1dd75-0309-48b0-a9dd-2a997d1444c4 nodeName:}" failed. No retries permitted until 2026-09-14 02:53:42.192956088 +0000 UTC m=+67.574040269 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/09b1dd75-0309-48b0-a9dd-2a997d1444c4-kube-state-metrics-tls") pod "kube-state-metrics-59fbdd77d-rrskf" (UID: "09b1dd75-0309-48b0-a9dd-2a997d1444c4") : secret "kube-state-metrics-tls" not found
Sep 14 02:53:41.693120 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.693011    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9grk4\" (UniqueName: \"kubernetes.io/projected/09b1dd75-0309-48b0-a9dd-2a997d1444c4-kube-api-access-9grk4\") pod \"kube-state-metrics-59fbdd77d-rrskf\" (UID: \"09b1dd75-0309-48b0-a9dd-2a997d1444c4\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rrskf"
Sep 14 02:53:41.693120 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.693080    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/68ff2110-81b2-469c-ba95-dab13090e6a8-metrics-client-ca\") pod \"node-exporter-tlvfd\" (UID: \"68ff2110-81b2-469c-ba95-dab13090e6a8\") " pod="openshift-monitoring/node-exporter-tlvfd"
Sep 14 02:53:41.693120 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.693106    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/09b1dd75-0309-48b0-a9dd-2a997d1444c4-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-rrskf\" (UID: \"09b1dd75-0309-48b0-a9dd-2a997d1444c4\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rrskf"
Sep 14 02:53:41.693623 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.693209    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/68ff2110-81b2-469c-ba95-dab13090e6a8-node-exporter-wtmp\") pod \"node-exporter-tlvfd\" (UID: \"68ff2110-81b2-469c-ba95-dab13090e6a8\") " pod="openshift-monitoring/node-exporter-tlvfd"
Sep 14 02:53:41.693623 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.693456    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/09b1dd75-0309-48b0-a9dd-2a997d1444c4-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-rrskf\" (UID: \"09b1dd75-0309-48b0-a9dd-2a997d1444c4\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rrskf"
Sep 14 02:53:41.693623 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.693471    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/09b1dd75-0309-48b0-a9dd-2a997d1444c4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-rrskf\" (UID: \"09b1dd75-0309-48b0-a9dd-2a997d1444c4\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rrskf"
Sep 14 02:53:41.693623 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.693516    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/68ff2110-81b2-469c-ba95-dab13090e6a8-node-exporter-textfile\") pod \"node-exporter-tlvfd\" (UID: \"68ff2110-81b2-469c-ba95-dab13090e6a8\") " pod="openshift-monitoring/node-exporter-tlvfd"
Sep 14 02:53:41.693623 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.693596    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/68ff2110-81b2-469c-ba95-dab13090e6a8-node-exporter-accelerators-collector-config\") pod \"node-exporter-tlvfd\" (UID: \"68ff2110-81b2-469c-ba95-dab13090e6a8\") " pod="openshift-monitoring/node-exporter-tlvfd"
Sep 14 02:53:41.694981 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.694956    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/68ff2110-81b2-469c-ba95-dab13090e6a8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tlvfd\" (UID: \"68ff2110-81b2-469c-ba95-dab13090e6a8\") " pod="openshift-monitoring/node-exporter-tlvfd"
Sep 14 02:53:41.695078 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.695037    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/09b1dd75-0309-48b0-a9dd-2a997d1444c4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-rrskf\" (UID: \"09b1dd75-0309-48b0-a9dd-2a997d1444c4\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rrskf"
Sep 14 02:53:41.701855 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.701831    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88p96\" (UniqueName: \"kubernetes.io/projected/68ff2110-81b2-469c-ba95-dab13090e6a8-kube-api-access-88p96\") pod \"node-exporter-tlvfd\" (UID: \"68ff2110-81b2-469c-ba95-dab13090e6a8\") " pod="openshift-monitoring/node-exporter-tlvfd"
Sep 14 02:53:41.701946 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.701861    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9grk4\" (UniqueName: \"kubernetes.io/projected/09b1dd75-0309-48b0-a9dd-2a997d1444c4-kube-api-access-9grk4\") pod \"kube-state-metrics-59fbdd77d-rrskf\" (UID: \"09b1dd75-0309-48b0-a9dd-2a997d1444c4\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rrskf"
Sep 14 02:53:41.772575 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.772538    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:41.878466 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.878366    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:41.981146 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:41.981117    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:42.078748 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:42.078710    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:42.095933 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:42.095905    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6102f536-8eb3-4666-91cd-7781a4cca236-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-qqz7z\" (UID: \"6102f536-8eb3-4666-91cd-7781a4cca236\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-qqz7z"
Sep 14 02:53:42.098845 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:42.098814    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6102f536-8eb3-4666-91cd-7781a4cca236-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-qqz7z\" (UID: \"6102f536-8eb3-4666-91cd-7781a4cca236\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-qqz7z"
Sep 14 02:53:42.197180 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:42.197089    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/68ff2110-81b2-469c-ba95-dab13090e6a8-node-exporter-tls\") pod \"node-exporter-tlvfd\" (UID: \"68ff2110-81b2-469c-ba95-dab13090e6a8\") " pod="openshift-monitoring/node-exporter-tlvfd"
Sep 14 02:53:42.197180 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:42.197153    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/09b1dd75-0309-48b0-a9dd-2a997d1444c4-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-rrskf\" (UID: \"09b1dd75-0309-48b0-a9dd-2a997d1444c4\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rrskf"
Sep 14 02:53:42.200034 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:42.199994    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/68ff2110-81b2-469c-ba95-dab13090e6a8-node-exporter-tls\") pod \"node-exporter-tlvfd\" (UID: \"68ff2110-81b2-469c-ba95-dab13090e6a8\") " pod="openshift-monitoring/node-exporter-tlvfd"
Sep 14 02:53:42.200499 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:42.200474    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/09b1dd75-0309-48b0-a9dd-2a997d1444c4-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-rrskf\" (UID: \"09b1dd75-0309-48b0-a9dd-2a997d1444c4\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rrskf"
Sep 14 02:53:42.373004 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:42.372977    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-qqz7z"
Sep 14 02:53:42.384715 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:42.384684    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8lf5z" event={"ID":"7029d1df-1b4b-40f6-95e1-621b70070086","Type":"ContainerStarted","Data":"6eb72ab3ab02973c70502e99761d2fc72a2d44f72aad58466e048e6c406c2dcc"}
Sep 14 02:53:42.386882 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:42.386812    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kq5nf" event={"ID":"8804d811-79f4-4612-bf40-38c08a04d2c8","Type":"ContainerStarted","Data":"deeb0add4fd8c6c35f51c774cce9931ea4119d02396bb24181ff143e60e8c379"}
Sep 14 02:53:42.388455 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:42.388435    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rrskf"
Sep 14 02:53:42.424977 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:42.424735    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tlvfd"
Sep 14 02:53:42.468501 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:42.468466    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:42.556281 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:42.556243    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 14 02:53:42.563379 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:42.562716    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:53:42.569118 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:42.568422    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 14 02:53:42.569118 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:42.568489    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-gkbx9\""
Sep 14 02:53:42.569118 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:42.568490    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 14 02:53:42.569118 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:42.568744    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 14 02:53:42.569118 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:42.568970    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 14 02:53:42.569487 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:42.569266    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 14 02:53:42.569544 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:42.569515    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 14 02:53:42.569623 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:42.569560    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 14 02:53:42.571246 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:42.569809    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 14 02:53:42.571246 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:42.570162    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:42.571598 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:42.571455    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 14 02:53:42.581950 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:42.581930    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 14 02:53:42.600504 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:42.600475    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5960d625-c84e-4102-b89f-c81fd92b3660-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5960d625-c84e-4102-b89f-c81fd92b3660\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:53:42.600504 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:42.600514    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5960d625-c84e-4102-b89f-c81fd92b3660-config-out\") pod \"alertmanager-main-0\" (UID: \"5960d625-c84e-4102-b89f-c81fd92b3660\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:53:42.600724 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:42.600558    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5960d625-c84e-4102-b89f-c81fd92b3660-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5960d625-c84e-4102-b89f-c81fd92b3660\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:53:42.600724 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:42.600588    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5960d625-c84e-4102-b89f-c81fd92b3660-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5960d625-c84e-4102-b89f-c81fd92b3660\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:53:42.600724 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:42.600644    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5960d625-c84e-4102-b89f-c81fd92b3660-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5960d625-c84e-4102-b89f-c81fd92b3660\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:53:42.600724 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:42.600673    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5960d625-c84e-4102-b89f-c81fd92b3660-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5960d625-c84e-4102-b89f-c81fd92b3660\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:53:42.600724 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:42.600710    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5960d625-c84e-4102-b89f-c81fd92b3660-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5960d625-c84e-4102-b89f-c81fd92b3660\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:53:42.600959 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:42.600735    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5960d625-c84e-4102-b89f-c81fd92b3660-config-volume\") pod \"alertmanager-main-0\" (UID: \"5960d625-c84e-4102-b89f-c81fd92b3660\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:53:42.600959 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:42.600789    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtdrl\" (UniqueName: \"kubernetes.io/projected/5960d625-c84e-4102-b89f-c81fd92b3660-kube-api-access-xtdrl\") pod \"alertmanager-main-0\" (UID: \"5960d625-c84e-4102-b89f-c81fd92b3660\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:53:42.600959 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:42.600822    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5960d625-c84e-4102-b89f-c81fd92b3660-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5960d625-c84e-4102-b89f-c81fd92b3660\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:53:42.600959 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:42.600844    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5960d625-c84e-4102-b89f-c81fd92b3660-web-config\") pod \"alertmanager-main-0\" (UID: \"5960d625-c84e-4102-b89f-c81fd92b3660\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:53:42.600959 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:42.600873    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5960d625-c84e-4102-b89f-c81fd92b3660-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5960d625-c84e-4102-b89f-c81fd92b3660\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:53:42.600959 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:42.600905    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5960d625-c84e-4102-b89f-c81fd92b3660-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5960d625-c84e-4102-b89f-c81fd92b3660\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:53:42.671982 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:42.671899    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-qqz7z"]
Sep 14 02:53:42.672618 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:42.672590    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:42.674353 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:53:42.674324    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6102f536_8eb3_4666_91cd_7781a4cca236.slice/crio-5fb452fef94c5df0975b83506cbd4b85c1bd55853df0297fc09b18419beaf153 WatchSource:0}: Error finding container 5fb452fef94c5df0975b83506cbd4b85c1bd55853df0297fc09b18419beaf153: Status 404 returned error can't find the container with id 5fb452fef94c5df0975b83506cbd4b85c1bd55853df0297fc09b18419beaf153
Sep 14 02:53:42.701504 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:42.701480    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5960d625-c84e-4102-b89f-c81fd92b3660-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5960d625-c84e-4102-b89f-c81fd92b3660\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:53:42.701602 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:42.701518    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5960d625-c84e-4102-b89f-c81fd92b3660-config-volume\") pod \"alertmanager-main-0\" (UID: \"5960d625-c84e-4102-b89f-c81fd92b3660\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:53:42.701602 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:42.701545    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtdrl\" (UniqueName: \"kubernetes.io/projected/5960d625-c84e-4102-b89f-c81fd92b3660-kube-api-access-xtdrl\") pod \"alertmanager-main-0\" (UID: \"5960d625-c84e-4102-b89f-c81fd92b3660\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:53:42.701602 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:42.701578    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5960d625-c84e-4102-b89f-c81fd92b3660-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5960d625-c84e-4102-b89f-c81fd92b3660\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:53:42.701773 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:42.701600    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5960d625-c84e-4102-b89f-c81fd92b3660-web-config\") pod \"alertmanager-main-0\" (UID: \"5960d625-c84e-4102-b89f-c81fd92b3660\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:53:42.701773 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:42.701627    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5960d625-c84e-4102-b89f-c81fd92b3660-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5960d625-c84e-4102-b89f-c81fd92b3660\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:53:42.701773 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:42.701662    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5960d625-c84e-4102-b89f-c81fd92b3660-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5960d625-c84e-4102-b89f-c81fd92b3660\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:53:42.701773 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:42.701707    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5960d625-c84e-4102-b89f-c81fd92b3660-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5960d625-c84e-4102-b89f-c81fd92b3660\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:53:42.701773 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:42.701731    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5960d625-c84e-4102-b89f-c81fd92b3660-config-out\") pod \"alertmanager-main-0\" (UID: \"5960d625-c84e-4102-b89f-c81fd92b3660\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:53:42.702036 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:42.701771    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5960d625-c84e-4102-b89f-c81fd92b3660-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5960d625-c84e-4102-b89f-c81fd92b3660\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:53:42.702036 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:42.701808    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5960d625-c84e-4102-b89f-c81fd92b3660-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5960d625-c84e-4102-b89f-c81fd92b3660\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:53:42.702036 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:42.701843    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5960d625-c84e-4102-b89f-c81fd92b3660-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5960d625-c84e-4102-b89f-c81fd92b3660\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:53:42.702036 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:42.701872    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5960d625-c84e-4102-b89f-c81fd92b3660-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5960d625-c84e-4102-b89f-c81fd92b3660\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:53:42.702912 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:42.702628    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5960d625-c84e-4102-b89f-c81fd92b3660-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5960d625-c84e-4102-b89f-c81fd92b3660\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:53:42.702912 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:42.702665    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5960d625-c84e-4102-b89f-c81fd92b3660-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5960d625-c84e-4102-b89f-c81fd92b3660\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:53:42.703419 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:53:42.703150    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5960d625-c84e-4102-b89f-c81fd92b3660-alertmanager-trusted-ca-bundle podName:5960d625-c84e-4102-b89f-c81fd92b3660 nodeName:}" failed. No retries permitted until 2026-09-14 02:53:43.203121013 +0000 UTC m=+68.584205217 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/5960d625-c84e-4102-b89f-c81fd92b3660-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "5960d625-c84e-4102-b89f-c81fd92b3660") : configmap references non-existent config key: ca-bundle.crt
Sep 14 02:53:42.708164 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:42.706251    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-rrskf"]
Sep 14 02:53:42.708164 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:42.706595    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5960d625-c84e-4102-b89f-c81fd92b3660-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5960d625-c84e-4102-b89f-c81fd92b3660\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:53:42.708164 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:42.707959    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5960d625-c84e-4102-b89f-c81fd92b3660-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5960d625-c84e-4102-b89f-c81fd92b3660\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:53:42.710703 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:42.710681    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5960d625-c84e-4102-b89f-c81fd92b3660-config-out\") pod \"alertmanager-main-0\" (UID: \"5960d625-c84e-4102-b89f-c81fd92b3660\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:53:42.711250 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:42.711228    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5960d625-c84e-4102-b89f-c81fd92b3660-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5960d625-c84e-4102-b89f-c81fd92b3660\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:53:42.711964 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:42.711921    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5960d625-c84e-4102-b89f-c81fd92b3660-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5960d625-c84e-4102-b89f-c81fd92b3660\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:53:42.712989 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:42.712969    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5960d625-c84e-4102-b89f-c81fd92b3660-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5960d625-c84e-4102-b89f-c81fd92b3660\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:53:42.713328 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:42.713302    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5960d625-c84e-4102-b89f-c81fd92b3660-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5960d625-c84e-4102-b89f-c81fd92b3660\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:53:42.713714 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:42.713690    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5960d625-c84e-4102-b89f-c81fd92b3660-web-config\") pod \"alertmanager-main-0\" (UID: \"5960d625-c84e-4102-b89f-c81fd92b3660\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:53:42.713826 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:42.713757    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5960d625-c84e-4102-b89f-c81fd92b3660-config-volume\") pod \"alertmanager-main-0\" (UID: \"5960d625-c84e-4102-b89f-c81fd92b3660\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:53:42.715874 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:53:42.715844    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09b1dd75_0309_48b0_a9dd_2a997d1444c4.slice/crio-e1621ff925c78b01e59634860ddcc84515ce954ebaf1712ee8be604165f6a34a WatchSource:0}: Error finding container e1621ff925c78b01e59634860ddcc84515ce954ebaf1712ee8be604165f6a34a: Status 404 returned error can't find the container with id e1621ff925c78b01e59634860ddcc84515ce954ebaf1712ee8be604165f6a34a
Sep 14 02:53:42.721352 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:42.721332    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtdrl\" (UniqueName: \"kubernetes.io/projected/5960d625-c84e-4102-b89f-c81fd92b3660-kube-api-access-xtdrl\") pod \"alertmanager-main-0\" (UID: \"5960d625-c84e-4102-b89f-c81fd92b3660\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:53:42.870210 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:42.870119    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:43.206197 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:43.206155    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5960d625-c84e-4102-b89f-c81fd92b3660-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5960d625-c84e-4102-b89f-c81fd92b3660\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:53:43.207078 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:43.207055    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5960d625-c84e-4102-b89f-c81fd92b3660-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5960d625-c84e-4102-b89f-c81fd92b3660\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:53:43.399303 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:43.399208    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tlvfd" event={"ID":"68ff2110-81b2-469c-ba95-dab13090e6a8","Type":"ContainerStarted","Data":"6dc4c88026aa2b461e34234090d89e187c85165c13b8c72658684c67b37fc329"}
Sep 14 02:53:43.404392 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:43.403699    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8lf5z" event={"ID":"7029d1df-1b4b-40f6-95e1-621b70070086","Type":"ContainerStarted","Data":"ecb6a40e51167d7bce21e796163aeed6861244bcfd681489dd6f1aca807b5f9e"}
Sep 14 02:53:43.406091 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:43.406013    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rrskf" event={"ID":"09b1dd75-0309-48b0-a9dd-2a997d1444c4","Type":"ContainerStarted","Data":"e1621ff925c78b01e59634860ddcc84515ce954ebaf1712ee8be604165f6a34a"}
Sep 14 02:53:43.410455 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:43.410316    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-qqz7z" event={"ID":"6102f536-8eb3-4666-91cd-7781a4cca236","Type":"ContainerStarted","Data":"fcb6f975486158aac4b1b1472c7d029a5ef8c6c1aefdfc97f5573dbf6b6db4d5"}
Sep 14 02:53:43.410455 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:43.410369    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-qqz7z" event={"ID":"6102f536-8eb3-4666-91cd-7781a4cca236","Type":"ContainerStarted","Data":"753c3ca088eee71cc275b5ef4bc93b1525acd46c17ebfc9add79a56b9da56f6f"}
Sep 14 02:53:43.410455 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:43.410397    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-qqz7z" event={"ID":"6102f536-8eb3-4666-91cd-7781a4cca236","Type":"ContainerStarted","Data":"5fb452fef94c5df0975b83506cbd4b85c1bd55853df0297fc09b18419beaf153"}
Sep 14 02:53:43.477170 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:43.477094    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:53:43.517450 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:43.517421    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:43.590500 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:43.589996    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-8lf5z" podStartSLOduration=67.612416267 podStartE2EDuration="1m8.589972347s" totalImagesPullingTime="977.55608ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:52:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:53:43.432526004 +0000 UTC m=+68.813610203" watchObservedRunningTime="2026-09-14 02:53:43.589972347 +0000 UTC m=+68.971056547"
Sep 14 02:53:43.590500 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:43.590318    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-76b45698db-7zh9r"]
Sep 14 02:53:43.604980 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:43.603680    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-76b45698db-7zh9r"
Sep 14 02:53:43.615740 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:43.615720    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 14 02:53:43.616460 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:43.615930    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 14 02:53:43.616460 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:43.615936    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 14 02:53:43.617017 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:43.616727    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:43.617017 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:43.616734    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-nb6r8\""
Sep 14 02:53:43.617017 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:43.616782    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 14 02:53:43.618399 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:43.618166    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-6ltjpqijd5iml\""
Sep 14 02:53:43.618399 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:43.618369    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 14 02:53:43.620154 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:43.620132    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-76b45698db-7zh9r"]
Sep 14 02:53:43.712540 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:43.712429    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:43.717453 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:43.717422    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/cc3d5c64-3f98-4267-9469-27eb14252478-secret-thanos-querier-tls\") pod \"thanos-querier-76b45698db-7zh9r\" (UID: \"cc3d5c64-3f98-4267-9469-27eb14252478\") " pod="openshift-monitoring/thanos-querier-76b45698db-7zh9r"
Sep 14 02:53:43.717600 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:43.717492    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cc3d5c64-3f98-4267-9469-27eb14252478-secret-grpc-tls\") pod \"thanos-querier-76b45698db-7zh9r\" (UID: \"cc3d5c64-3f98-4267-9469-27eb14252478\") " pod="openshift-monitoring/thanos-querier-76b45698db-7zh9r"
Sep 14 02:53:43.717600 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:43.717533    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/cc3d5c64-3f98-4267-9469-27eb14252478-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-76b45698db-7zh9r\" (UID: \"cc3d5c64-3f98-4267-9469-27eb14252478\") " pod="openshift-monitoring/thanos-querier-76b45698db-7zh9r"
Sep 14 02:53:43.717600 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:43.717568    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtxld\" (UniqueName: \"kubernetes.io/projected/cc3d5c64-3f98-4267-9469-27eb14252478-kube-api-access-gtxld\") pod \"thanos-querier-76b45698db-7zh9r\" (UID: \"cc3d5c64-3f98-4267-9469-27eb14252478\") " pod="openshift-monitoring/thanos-querier-76b45698db-7zh9r"
Sep 14 02:53:43.717759 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:43.717604    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cc3d5c64-3f98-4267-9469-27eb14252478-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-76b45698db-7zh9r\" (UID: \"cc3d5c64-3f98-4267-9469-27eb14252478\") " pod="openshift-monitoring/thanos-querier-76b45698db-7zh9r"
Sep 14 02:53:43.717759 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:43.717649    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cc3d5c64-3f98-4267-9469-27eb14252478-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-76b45698db-7zh9r\" (UID: \"cc3d5c64-3f98-4267-9469-27eb14252478\") " pod="openshift-monitoring/thanos-querier-76b45698db-7zh9r"
Sep 14 02:53:43.717759 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:43.717683    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc3d5c64-3f98-4267-9469-27eb14252478-metrics-client-ca\") pod \"thanos-querier-76b45698db-7zh9r\" (UID: \"cc3d5c64-3f98-4267-9469-27eb14252478\") " pod="openshift-monitoring/thanos-querier-76b45698db-7zh9r"
Sep 14 02:53:43.717759 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:43.717740    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/cc3d5c64-3f98-4267-9469-27eb14252478-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-76b45698db-7zh9r\" (UID: \"cc3d5c64-3f98-4267-9469-27eb14252478\") " pod="openshift-monitoring/thanos-querier-76b45698db-7zh9r"
Sep 14 02:53:43.809056 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:43.808956    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 14 02:53:43.818278 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:43.818246    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cc3d5c64-3f98-4267-9469-27eb14252478-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-76b45698db-7zh9r\" (UID: \"cc3d5c64-3f98-4267-9469-27eb14252478\") " pod="openshift-monitoring/thanos-querier-76b45698db-7zh9r"
Sep 14 02:53:43.818392 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:43.818316    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cc3d5c64-3f98-4267-9469-27eb14252478-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-76b45698db-7zh9r\" (UID: \"cc3d5c64-3f98-4267-9469-27eb14252478\") " pod="openshift-monitoring/thanos-querier-76b45698db-7zh9r"
Sep 14 02:53:43.818392 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:43.818378    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc3d5c64-3f98-4267-9469-27eb14252478-metrics-client-ca\") pod \"thanos-querier-76b45698db-7zh9r\" (UID: \"cc3d5c64-3f98-4267-9469-27eb14252478\") " pod="openshift-monitoring/thanos-querier-76b45698db-7zh9r"
Sep 14 02:53:43.818524 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:43.818428    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/cc3d5c64-3f98-4267-9469-27eb14252478-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-76b45698db-7zh9r\" (UID: \"cc3d5c64-3f98-4267-9469-27eb14252478\") " pod="openshift-monitoring/thanos-querier-76b45698db-7zh9r"
Sep 14 02:53:43.818524 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:43.818469    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/cc3d5c64-3f98-4267-9469-27eb14252478-secret-thanos-querier-tls\") pod \"thanos-querier-76b45698db-7zh9r\" (UID: \"cc3d5c64-3f98-4267-9469-27eb14252478\") " pod="openshift-monitoring/thanos-querier-76b45698db-7zh9r"
Sep 14 02:53:43.818635 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:43.818525    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cc3d5c64-3f98-4267-9469-27eb14252478-secret-grpc-tls\") pod \"thanos-querier-76b45698db-7zh9r\" (UID: \"cc3d5c64-3f98-4267-9469-27eb14252478\") " pod="openshift-monitoring/thanos-querier-76b45698db-7zh9r"
Sep 14 02:53:43.818635 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:43.818566    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/cc3d5c64-3f98-4267-9469-27eb14252478-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-76b45698db-7zh9r\" (UID: \"cc3d5c64-3f98-4267-9469-27eb14252478\") " pod="openshift-monitoring/thanos-querier-76b45698db-7zh9r"
Sep 14 02:53:43.818635 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:43.818597    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtxld\" (UniqueName: \"kubernetes.io/projected/cc3d5c64-3f98-4267-9469-27eb14252478-kube-api-access-gtxld\") pod \"thanos-querier-76b45698db-7zh9r\" (UID: \"cc3d5c64-3f98-4267-9469-27eb14252478\") " pod="openshift-monitoring/thanos-querier-76b45698db-7zh9r"
Sep 14 02:53:43.819946 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:43.819895    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc3d5c64-3f98-4267-9469-27eb14252478-metrics-client-ca\") pod \"thanos-querier-76b45698db-7zh9r\" (UID: \"cc3d5c64-3f98-4267-9469-27eb14252478\") " pod="openshift-monitoring/thanos-querier-76b45698db-7zh9r"
Sep 14 02:53:43.822426 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:43.822289    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cc3d5c64-3f98-4267-9469-27eb14252478-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-76b45698db-7zh9r\" (UID: \"cc3d5c64-3f98-4267-9469-27eb14252478\") " pod="openshift-monitoring/thanos-querier-76b45698db-7zh9r"
Sep 14 02:53:43.823485 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:43.823458    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/cc3d5c64-3f98-4267-9469-27eb14252478-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-76b45698db-7zh9r\" (UID: \"cc3d5c64-3f98-4267-9469-27eb14252478\") " pod="openshift-monitoring/thanos-querier-76b45698db-7zh9r"
Sep 14 02:53:43.823576 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:43.823487    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cc3d5c64-3f98-4267-9469-27eb14252478-secret-grpc-tls\") pod \"thanos-querier-76b45698db-7zh9r\" (UID: \"cc3d5c64-3f98-4267-9469-27eb14252478\") " pod="openshift-monitoring/thanos-querier-76b45698db-7zh9r"
Sep 14 02:53:43.824290 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:43.824249    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cc3d5c64-3f98-4267-9469-27eb14252478-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-76b45698db-7zh9r\" (UID: \"cc3d5c64-3f98-4267-9469-27eb14252478\") " pod="openshift-monitoring/thanos-querier-76b45698db-7zh9r"
Sep 14 02:53:43.824497 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:43.824392    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/cc3d5c64-3f98-4267-9469-27eb14252478-secret-thanos-querier-tls\") pod \"thanos-querier-76b45698db-7zh9r\" (UID: \"cc3d5c64-3f98-4267-9469-27eb14252478\") " pod="openshift-monitoring/thanos-querier-76b45698db-7zh9r"
Sep 14 02:53:43.827132 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:43.827107    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/cc3d5c64-3f98-4267-9469-27eb14252478-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-76b45698db-7zh9r\" (UID: \"cc3d5c64-3f98-4267-9469-27eb14252478\") " pod="openshift-monitoring/thanos-querier-76b45698db-7zh9r"
Sep 14 02:53:43.837789 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:43.837747    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtxld\" (UniqueName: \"kubernetes.io/projected/cc3d5c64-3f98-4267-9469-27eb14252478-kube-api-access-gtxld\") pod \"thanos-querier-76b45698db-7zh9r\" (UID: \"cc3d5c64-3f98-4267-9469-27eb14252478\") " pod="openshift-monitoring/thanos-querier-76b45698db-7zh9r"
Sep 14 02:53:43.927326 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:43.927293    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-76b45698db-7zh9r"
Sep 14 02:53:43.987914 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:53:43.987838    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5960d625_c84e_4102_b89f_c81fd92b3660.slice/crio-5754c20ca2aa95033f8330f50555b9fde17af2fc8c3a98c0968e5fefbc1ebacf WatchSource:0}: Error finding container 5754c20ca2aa95033f8330f50555b9fde17af2fc8c3a98c0968e5fefbc1ebacf: Status 404 returned error can't find the container with id 5754c20ca2aa95033f8330f50555b9fde17af2fc8c3a98c0968e5fefbc1ebacf
Sep 14 02:53:44.277337 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:44.277230    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-76b45698db-7zh9r"]
Sep 14 02:53:44.345449 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:44.343956    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-nwtp8"
Sep 14 02:53:44.417887 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:44.417850    2564 generic.go:356] "Generic (PLEG): container finished" podID="68ff2110-81b2-469c-ba95-dab13090e6a8" containerID="df98c02b0b60df381347dc8f8f088d979d4d2fe3e4a0c4198f17c2055dd12585" exitCode=0
Sep 14 02:53:44.418078 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:44.417956    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tlvfd" event={"ID":"68ff2110-81b2-469c-ba95-dab13090e6a8","Type":"ContainerDied","Data":"df98c02b0b60df381347dc8f8f088d979d4d2fe3e4a0c4198f17c2055dd12585"}
Sep 14 02:53:44.420007 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:44.419909    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5960d625-c84e-4102-b89f-c81fd92b3660","Type":"ContainerStarted","Data":"5754c20ca2aa95033f8330f50555b9fde17af2fc8c3a98c0968e5fefbc1ebacf"}
Sep 14 02:53:44.424831 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:44.424396    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-qqz7z" event={"ID":"6102f536-8eb3-4666-91cd-7781a4cca236","Type":"ContainerStarted","Data":"cafdd0a944ad892adfceae663f3de2c6682557141a1e1a967894381a7dadee2a"}
Sep 14 02:53:44.471118 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:44.470452    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-qqz7z" podStartSLOduration=2.236453506 podStartE2EDuration="3.470431746s" totalImagesPullingTime="1.23397824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:53:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:53:44.469105378 +0000 UTC m=+69.850189591" watchObservedRunningTime="2026-09-14 02:53:44.470431746 +0000 UTC m=+69.851515944"
Sep 14 02:53:44.471118 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:44.470962    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:44.551593 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:44.551445    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:44.562804 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:44.562774    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-66f88cd956-494g9"
Sep 14 02:53:44.562952 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:44.562858    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-66f88cd956-494g9"
Sep 14 02:53:44.567676 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:44.567656    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-66f88cd956-494g9"
Sep 14 02:53:44.570777 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:53:44.570755    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc3d5c64_3f98_4267_9469_27eb14252478.slice/crio-b4309ed601472838147ce6af7f88630a65dd96b2c235868442692bf47647af0f WatchSource:0}: Error finding container b4309ed601472838147ce6af7f88630a65dd96b2c235868442692bf47647af0f: Status 404 returned error can't find the container with id b4309ed601472838147ce6af7f88630a65dd96b2c235868442692bf47647af0f
Sep 14 02:53:45.121687 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:45.121652    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:45.429000 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:45.428921    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76b45698db-7zh9r" event={"ID":"cc3d5c64-3f98-4267-9469-27eb14252478","Type":"ContainerStarted","Data":"b4309ed601472838147ce6af7f88630a65dd96b2c235868442692bf47647af0f"}
Sep 14 02:53:45.433511 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:45.433486    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-66f88cd956-494g9"
Sep 14 02:53:45.626174 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:45.626135    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:45.732315 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:45.732220    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:45.834662 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:45.834630    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:45.990652 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:45.990565    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7f54f5c747-7g79s"]
Sep 14 02:53:45.994021 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:45.993996    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7f54f5c747-7g79s"
Sep 14 02:53:45.996670 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:45.996644    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 14 02:53:45.996814 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:45.996791    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 14 02:53:45.996876 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:45.996839    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 14 02:53:45.997601 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:45.997582    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 14 02:53:45.997821 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:45.997776    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-8efh868e8hkd3\""
Sep 14 02:53:45.997934 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:45.997914    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-vm842\""
Sep 14 02:53:46.005792 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:46.005765    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7f54f5c747-7g79s"]
Sep 14 02:53:46.038304 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:46.038265    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bffd2f8e-b7bb-49f5-b563-9c1c1af69a67-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7f54f5c747-7g79s\" (UID: \"bffd2f8e-b7bb-49f5-b563-9c1c1af69a67\") " pod="openshift-monitoring/metrics-server-7f54f5c747-7g79s"
Sep 14 02:53:46.038480 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:46.038308    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpptt\" (UniqueName: \"kubernetes.io/projected/bffd2f8e-b7bb-49f5-b563-9c1c1af69a67-kube-api-access-qpptt\") pod \"metrics-server-7f54f5c747-7g79s\" (UID: \"bffd2f8e-b7bb-49f5-b563-9c1c1af69a67\") " pod="openshift-monitoring/metrics-server-7f54f5c747-7g79s"
Sep 14 02:53:46.038480 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:46.038349    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/bffd2f8e-b7bb-49f5-b563-9c1c1af69a67-client-ca-bundle\") pod \"metrics-server-7f54f5c747-7g79s\" (UID: \"bffd2f8e-b7bb-49f5-b563-9c1c1af69a67\") " pod="openshift-monitoring/metrics-server-7f54f5c747-7g79s"
Sep 14 02:53:46.038480 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:46.038428    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/bffd2f8e-b7bb-49f5-b563-9c1c1af69a67-metrics-server-audit-profiles\") pod \"metrics-server-7f54f5c747-7g79s\" (UID: \"bffd2f8e-b7bb-49f5-b563-9c1c1af69a67\") " pod="openshift-monitoring/metrics-server-7f54f5c747-7g79s"
Sep 14 02:53:46.038480 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:46.038475    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/bffd2f8e-b7bb-49f5-b563-9c1c1af69a67-secret-metrics-server-client-certs\") pod \"metrics-server-7f54f5c747-7g79s\" (UID: \"bffd2f8e-b7bb-49f5-b563-9c1c1af69a67\") " pod="openshift-monitoring/metrics-server-7f54f5c747-7g79s"
Sep 14 02:53:46.038707 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:46.038518    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/bffd2f8e-b7bb-49f5-b563-9c1c1af69a67-secret-metrics-server-tls\") pod \"metrics-server-7f54f5c747-7g79s\" (UID: \"bffd2f8e-b7bb-49f5-b563-9c1c1af69a67\") " pod="openshift-monitoring/metrics-server-7f54f5c747-7g79s"
Sep 14 02:53:46.038707 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:46.038544    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/bffd2f8e-b7bb-49f5-b563-9c1c1af69a67-audit-log\") pod \"metrics-server-7f54f5c747-7g79s\" (UID: \"bffd2f8e-b7bb-49f5-b563-9c1c1af69a67\") " pod="openshift-monitoring/metrics-server-7f54f5c747-7g79s"
Sep 14 02:53:46.139700 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:46.139669    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/bffd2f8e-b7bb-49f5-b563-9c1c1af69a67-metrics-server-audit-profiles\") pod \"metrics-server-7f54f5c747-7g79s\" (UID: \"bffd2f8e-b7bb-49f5-b563-9c1c1af69a67\") " pod="openshift-monitoring/metrics-server-7f54f5c747-7g79s"
Sep 14 02:53:46.139859 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:46.139710    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/bffd2f8e-b7bb-49f5-b563-9c1c1af69a67-secret-metrics-server-client-certs\") pod \"metrics-server-7f54f5c747-7g79s\" (UID: \"bffd2f8e-b7bb-49f5-b563-9c1c1af69a67\") " pod="openshift-monitoring/metrics-server-7f54f5c747-7g79s"
Sep 14 02:53:46.139859 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:46.139746    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/bffd2f8e-b7bb-49f5-b563-9c1c1af69a67-secret-metrics-server-tls\") pod \"metrics-server-7f54f5c747-7g79s\" (UID: \"bffd2f8e-b7bb-49f5-b563-9c1c1af69a67\") " pod="openshift-monitoring/metrics-server-7f54f5c747-7g79s"
Sep 14 02:53:46.139859 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:46.139773    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/bffd2f8e-b7bb-49f5-b563-9c1c1af69a67-audit-log\") pod \"metrics-server-7f54f5c747-7g79s\" (UID: \"bffd2f8e-b7bb-49f5-b563-9c1c1af69a67\") " pod="openshift-monitoring/metrics-server-7f54f5c747-7g79s"
Sep 14 02:53:46.139859 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:46.139806    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bffd2f8e-b7bb-49f5-b563-9c1c1af69a67-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7f54f5c747-7g79s\" (UID: \"bffd2f8e-b7bb-49f5-b563-9c1c1af69a67\") " pod="openshift-monitoring/metrics-server-7f54f5c747-7g79s"
Sep 14 02:53:46.139859 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:46.139824    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpptt\" (UniqueName: \"kubernetes.io/projected/bffd2f8e-b7bb-49f5-b563-9c1c1af69a67-kube-api-access-qpptt\") pod \"metrics-server-7f54f5c747-7g79s\" (UID: \"bffd2f8e-b7bb-49f5-b563-9c1c1af69a67\") " pod="openshift-monitoring/metrics-server-7f54f5c747-7g79s"
Sep 14 02:53:46.140110 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:46.139880    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/bffd2f8e-b7bb-49f5-b563-9c1c1af69a67-client-ca-bundle\") pod \"metrics-server-7f54f5c747-7g79s\" (UID: \"bffd2f8e-b7bb-49f5-b563-9c1c1af69a67\") " pod="openshift-monitoring/metrics-server-7f54f5c747-7g79s"
Sep 14 02:53:46.140386 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:46.140337    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/bffd2f8e-b7bb-49f5-b563-9c1c1af69a67-audit-log\") pod \"metrics-server-7f54f5c747-7g79s\" (UID: \"bffd2f8e-b7bb-49f5-b563-9c1c1af69a67\") " pod="openshift-monitoring/metrics-server-7f54f5c747-7g79s"
Sep 14 02:53:46.140709 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:46.140652    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bffd2f8e-b7bb-49f5-b563-9c1c1af69a67-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7f54f5c747-7g79s\" (UID: \"bffd2f8e-b7bb-49f5-b563-9c1c1af69a67\") " pod="openshift-monitoring/metrics-server-7f54f5c747-7g79s"
Sep 14 02:53:46.140872 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:46.140846    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/bffd2f8e-b7bb-49f5-b563-9c1c1af69a67-metrics-server-audit-profiles\") pod \"metrics-server-7f54f5c747-7g79s\" (UID: \"bffd2f8e-b7bb-49f5-b563-9c1c1af69a67\") " pod="openshift-monitoring/metrics-server-7f54f5c747-7g79s"
Sep 14 02:53:46.142837 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:46.142815    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/bffd2f8e-b7bb-49f5-b563-9c1c1af69a67-client-ca-bundle\") pod \"metrics-server-7f54f5c747-7g79s\" (UID: \"bffd2f8e-b7bb-49f5-b563-9c1c1af69a67\") " pod="openshift-monitoring/metrics-server-7f54f5c747-7g79s"
Sep 14 02:53:46.142935 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:46.142845    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/bffd2f8e-b7bb-49f5-b563-9c1c1af69a67-secret-metrics-server-tls\") pod \"metrics-server-7f54f5c747-7g79s\" (UID: \"bffd2f8e-b7bb-49f5-b563-9c1c1af69a67\") " pod="openshift-monitoring/metrics-server-7f54f5c747-7g79s"
Sep 14 02:53:46.142935 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:46.142878    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/bffd2f8e-b7bb-49f5-b563-9c1c1af69a67-secret-metrics-server-client-certs\") pod \"metrics-server-7f54f5c747-7g79s\" (UID: \"bffd2f8e-b7bb-49f5-b563-9c1c1af69a67\") " pod="openshift-monitoring/metrics-server-7f54f5c747-7g79s"
Sep 14 02:53:46.148323 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:46.148303    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpptt\" (UniqueName: \"kubernetes.io/projected/bffd2f8e-b7bb-49f5-b563-9c1c1af69a67-kube-api-access-qpptt\") pod \"metrics-server-7f54f5c747-7g79s\" (UID: \"bffd2f8e-b7bb-49f5-b563-9c1c1af69a67\") " pod="openshift-monitoring/metrics-server-7f54f5c747-7g79s"
Sep 14 02:53:46.258379 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:46.258295    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-znr76"]
Sep 14 02:53:46.261720 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:46.261678    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:46.264075 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:46.263288    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-znr76"
Sep 14 02:53:46.266305 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:46.266283    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-sqhh8\""
Sep 14 02:53:46.266605 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:46.266585    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 14 02:53:46.270529 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:46.270478    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-znr76"]
Sep 14 02:53:46.304982 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:46.304958    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7f54f5c747-7g79s"
Sep 14 02:53:46.341524 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:46.341491    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/349c747c-2054-4d9a-9d6c-9bda627193fd-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-znr76\" (UID: \"349c747c-2054-4d9a-9d6c-9bda627193fd\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-znr76"
Sep 14 02:53:46.341682 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:46.341586    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrfh9\" (UniqueName: \"kubernetes.io/projected/349c747c-2054-4d9a-9d6c-9bda627193fd-kube-api-access-jrfh9\") pod \"monitoring-plugin-5d85778b9d-znr76\" (UID: \"349c747c-2054-4d9a-9d6c-9bda627193fd\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-znr76"
Sep 14 02:53:46.374932 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:46.374837    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:46.439749 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:46.439697    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:46.442715 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:46.442690    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/349c747c-2054-4d9a-9d6c-9bda627193fd-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-znr76\" (UID: \"349c747c-2054-4d9a-9d6c-9bda627193fd\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-znr76"
Sep 14 02:53:46.442806 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:46.442746    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrfh9\" (UniqueName: \"kubernetes.io/projected/349c747c-2054-4d9a-9d6c-9bda627193fd-kube-api-access-jrfh9\") pod \"monitoring-plugin-5d85778b9d-znr76\" (UID: \"349c747c-2054-4d9a-9d6c-9bda627193fd\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-znr76"
Sep 14 02:53:46.442840 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:53:46.442820    2564 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 14 02:53:46.442902 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:53:46.442891    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/349c747c-2054-4d9a-9d6c-9bda627193fd-monitoring-plugin-cert podName:349c747c-2054-4d9a-9d6c-9bda627193fd nodeName:}" failed. No retries permitted until 2026-09-14 02:53:46.942873158 +0000 UTC m=+72.323957337 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/349c747c-2054-4d9a-9d6c-9bda627193fd-monitoring-plugin-cert") pod "monitoring-plugin-5d85778b9d-znr76" (UID: "349c747c-2054-4d9a-9d6c-9bda627193fd") : secret "monitoring-plugin-cert" not found
Sep 14 02:53:46.454266 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:46.454235    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrfh9\" (UniqueName: \"kubernetes.io/projected/349c747c-2054-4d9a-9d6c-9bda627193fd-kube-api-access-jrfh9\") pod \"monitoring-plugin-5d85778b9d-znr76\" (UID: \"349c747c-2054-4d9a-9d6c-9bda627193fd\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-znr76"
Sep 14 02:53:46.587647 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:46.587605    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:46.947668 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:46.947591    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/349c747c-2054-4d9a-9d6c-9bda627193fd-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-znr76\" (UID: \"349c747c-2054-4d9a-9d6c-9bda627193fd\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-znr76"
Sep 14 02:53:46.950642 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:46.950612    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/349c747c-2054-4d9a-9d6c-9bda627193fd-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-znr76\" (UID: \"349c747c-2054-4d9a-9d6c-9bda627193fd\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-znr76"
Sep 14 02:53:47.175454 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:47.175415    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-znr76"
Sep 14 02:53:47.361325 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:47.361255    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-cd9dc9dbb-256pz"]
Sep 14 02:53:47.370784 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:47.368988    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cd9dc9dbb-256pz"
Sep 14 02:53:47.381251 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:47.381223    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cd9dc9dbb-256pz"]
Sep 14 02:53:47.448482 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:47.448075    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tlvfd" event={"ID":"68ff2110-81b2-469c-ba95-dab13090e6a8","Type":"ContainerStarted","Data":"fef6182aa118a0e2d2f22873b6b81232bc9154492f88b43749c3601dd94031ca"}
Sep 14 02:53:47.451904 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:47.451869    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b-oauth-serving-cert\") pod \"console-cd9dc9dbb-256pz\" (UID: \"3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b\") " pod="openshift-console/console-cd9dc9dbb-256pz"
Sep 14 02:53:47.452076 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:47.451996    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4th7l\" (UniqueName: \"kubernetes.io/projected/3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b-kube-api-access-4th7l\") pod \"console-cd9dc9dbb-256pz\" (UID: \"3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b\") " pod="openshift-console/console-cd9dc9dbb-256pz"
Sep 14 02:53:47.452076 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:47.452047    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b-console-oauth-config\") pod \"console-cd9dc9dbb-256pz\" (UID: \"3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b\") " pod="openshift-console/console-cd9dc9dbb-256pz"
Sep 14 02:53:47.452183 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:47.452119    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b-service-ca\") pod \"console-cd9dc9dbb-256pz\" (UID: \"3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b\") " pod="openshift-console/console-cd9dc9dbb-256pz"
Sep 14 02:53:47.452183 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:47.452149    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b-console-serving-cert\") pod \"console-cd9dc9dbb-256pz\" (UID: \"3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b\") " pod="openshift-console/console-cd9dc9dbb-256pz"
Sep 14 02:53:47.452183 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:47.452177    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b-console-config\") pod \"console-cd9dc9dbb-256pz\" (UID: \"3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b\") " pod="openshift-console/console-cd9dc9dbb-256pz"
Sep 14 02:53:47.452330 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:47.452200    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b-trusted-ca-bundle\") pod \"console-cd9dc9dbb-256pz\" (UID: \"3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b\") " pod="openshift-console/console-cd9dc9dbb-256pz"
Sep 14 02:53:47.553294 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:47.553266    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b-service-ca\") pod \"console-cd9dc9dbb-256pz\" (UID: \"3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b\") " pod="openshift-console/console-cd9dc9dbb-256pz"
Sep 14 02:53:47.553393 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:47.553339    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b-console-serving-cert\") pod \"console-cd9dc9dbb-256pz\" (UID: \"3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b\") " pod="openshift-console/console-cd9dc9dbb-256pz"
Sep 14 02:53:47.553393 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:47.553373    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b-console-config\") pod \"console-cd9dc9dbb-256pz\" (UID: \"3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b\") " pod="openshift-console/console-cd9dc9dbb-256pz"
Sep 14 02:53:47.553556 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:47.553415    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b-trusted-ca-bundle\") pod \"console-cd9dc9dbb-256pz\" (UID: \"3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b\") " pod="openshift-console/console-cd9dc9dbb-256pz"
Sep 14 02:53:47.553556 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:47.553468    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b-oauth-serving-cert\") pod \"console-cd9dc9dbb-256pz\" (UID: \"3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b\") " pod="openshift-console/console-cd9dc9dbb-256pz"
Sep 14 02:53:47.553663 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:47.553555    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4th7l\" (UniqueName: \"kubernetes.io/projected/3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b-kube-api-access-4th7l\") pod \"console-cd9dc9dbb-256pz\" (UID: \"3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b\") " pod="openshift-console/console-cd9dc9dbb-256pz"
Sep 14 02:53:47.553663 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:47.553621    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b-console-oauth-config\") pod \"console-cd9dc9dbb-256pz\" (UID: \"3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b\") " pod="openshift-console/console-cd9dc9dbb-256pz"
Sep 14 02:53:47.556898 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:47.556491    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b-oauth-serving-cert\") pod \"console-cd9dc9dbb-256pz\" (UID: \"3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b\") " pod="openshift-console/console-cd9dc9dbb-256pz"
Sep 14 02:53:47.557745 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:47.557717    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b-trusted-ca-bundle\") pod \"console-cd9dc9dbb-256pz\" (UID: \"3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b\") " pod="openshift-console/console-cd9dc9dbb-256pz"
Sep 14 02:53:47.558315 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:47.558289    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b-console-config\") pod \"console-cd9dc9dbb-256pz\" (UID: \"3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b\") " pod="openshift-console/console-cd9dc9dbb-256pz"
Sep 14 02:53:47.558785 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:47.558748    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b-service-ca\") pod \"console-cd9dc9dbb-256pz\" (UID: \"3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b\") " pod="openshift-console/console-cd9dc9dbb-256pz"
Sep 14 02:53:47.561025 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:47.560985    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b-console-serving-cert\") pod \"console-cd9dc9dbb-256pz\" (UID: \"3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b\") " pod="openshift-console/console-cd9dc9dbb-256pz"
Sep 14 02:53:47.567938 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:47.567889    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b-console-oauth-config\") pod \"console-cd9dc9dbb-256pz\" (UID: \"3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b\") " pod="openshift-console/console-cd9dc9dbb-256pz"
Sep 14 02:53:47.572636 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:47.572591    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4th7l\" (UniqueName: \"kubernetes.io/projected/3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b-kube-api-access-4th7l\") pod \"console-cd9dc9dbb-256pz\" (UID: \"3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b\") " pod="openshift-console/console-cd9dc9dbb-256pz"
Sep 14 02:53:47.611794 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:47.611759    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-znr76"]
Sep 14 02:53:47.614542 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:47.613752    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7f54f5c747-7g79s"]
Sep 14 02:53:47.627558 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:53:47.623834    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod349c747c_2054_4d9a_9d6c_9bda627193fd.slice/crio-38925a7e5497305e78700233b5361109b1b33bf6f9e95f86c453ee422c1f2e0a WatchSource:0}: Error finding container 38925a7e5497305e78700233b5361109b1b33bf6f9e95f86c453ee422c1f2e0a: Status 404 returned error can't find the container with id 38925a7e5497305e78700233b5361109b1b33bf6f9e95f86c453ee422c1f2e0a
Sep 14 02:53:47.627558 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:53:47.624103    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbffd2f8e_b7bb_49f5_b563_9c1c1af69a67.slice/crio-6e8eef1a12b11fbc29eef3b4c19d0039ade312da5e8fafbea9e4e5ba5e258d1e WatchSource:0}: Error finding container 6e8eef1a12b11fbc29eef3b4c19d0039ade312da5e8fafbea9e4e5ba5e258d1e: Status 404 returned error can't find the container with id 6e8eef1a12b11fbc29eef3b4c19d0039ade312da5e8fafbea9e4e5ba5e258d1e
Sep 14 02:53:47.636937 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:47.635863    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:47.714598 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:47.714246    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cd9dc9dbb-256pz"
Sep 14 02:53:47.736891 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:47.736695    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:47.814748 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:47.814182    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 14 02:53:47.820073 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:47.820047    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:53:47.824159 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:47.824126    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 14 02:53:47.824256 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:47.824172    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 14 02:53:47.824256 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:47.824205    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 14 02:53:47.824339 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:47.824132    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9l9d1jhjomc4c\""
Sep 14 02:53:47.824555 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:47.824537    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 14 02:53:47.825084 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:47.825065    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:47.825473 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:47.825327    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-rdxjb\""
Sep 14 02:53:47.825473 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:47.825363    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 14 02:53:47.825473 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:47.825381    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 14 02:53:47.825776 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:47.825756    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 14 02:53:47.826492 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:47.826468    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 14 02:53:47.827061 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:47.827037    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 14 02:53:47.827267 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:47.827199    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 14 02:53:47.827381 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:47.827354    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 14 02:53:47.827879 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:47.827466    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 14 02:53:47.839459 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:47.837913    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 14 02:53:47.839459 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:47.838988    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 14 02:53:47.844083 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:47.844061    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 14 02:53:47.962446 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:47.962389    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/603095db-c9ac-44a7-b91a-c8f7ebde9397-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:53:47.962446 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:47.962447    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/603095db-c9ac-44a7-b91a-c8f7ebde9397-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:53:47.962884 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:47.962479    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/603095db-c9ac-44a7-b91a-c8f7ebde9397-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:53:47.962884 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:47.962512    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/603095db-c9ac-44a7-b91a-c8f7ebde9397-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:53:47.962884 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:47.962546    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/603095db-c9ac-44a7-b91a-c8f7ebde9397-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:53:47.962884 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:47.962581    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/603095db-c9ac-44a7-b91a-c8f7ebde9397-config-out\") pod \"prometheus-k8s-0\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:53:47.962884 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:47.962603    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/603095db-c9ac-44a7-b91a-c8f7ebde9397-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:53:47.962884 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:47.962635    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/603095db-c9ac-44a7-b91a-c8f7ebde9397-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:53:47.962884 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:47.962671    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/603095db-c9ac-44a7-b91a-c8f7ebde9397-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:53:47.962884 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:47.962694    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/603095db-c9ac-44a7-b91a-c8f7ebde9397-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:53:47.962884 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:47.962733    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/603095db-c9ac-44a7-b91a-c8f7ebde9397-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:53:47.962884 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:47.962761    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/603095db-c9ac-44a7-b91a-c8f7ebde9397-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:53:47.962884 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:47.962814    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/603095db-c9ac-44a7-b91a-c8f7ebde9397-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:53:47.962884 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:47.962840    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/603095db-c9ac-44a7-b91a-c8f7ebde9397-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:53:47.962884 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:47.962864    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbk6c\" (UniqueName: \"kubernetes.io/projected/603095db-c9ac-44a7-b91a-c8f7ebde9397-kube-api-access-zbk6c\") pod \"prometheus-k8s-0\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:53:47.962884 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:47.962888    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/603095db-c9ac-44a7-b91a-c8f7ebde9397-config\") pod \"prometheus-k8s-0\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:53:47.963384 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:47.962913    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/603095db-c9ac-44a7-b91a-c8f7ebde9397-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:53:47.963384 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:47.962946    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/603095db-c9ac-44a7-b91a-c8f7ebde9397-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:53:47.963384 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:47.962970    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/603095db-c9ac-44a7-b91a-c8f7ebde9397-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:53:47.963384 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:47.962995    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/603095db-c9ac-44a7-b91a-c8f7ebde9397-web-config\") pod \"prometheus-k8s-0\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:53:48.016768 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:48.016607    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cd9dc9dbb-256pz"]
Sep 14 02:53:48.020123 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:53:48.020094    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3acdfdd4_0913_4d4d_bd8e_fdc448e5ff7b.slice/crio-1ee8edb2b2051afe71650b0b3394b07427f793e9fd28547bc949e3e7541aeff1 WatchSource:0}: Error finding container 1ee8edb2b2051afe71650b0b3394b07427f793e9fd28547bc949e3e7541aeff1: Status 404 returned error can't find the container with id 1ee8edb2b2051afe71650b0b3394b07427f793e9fd28547bc949e3e7541aeff1
Sep 14 02:53:48.064486 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:48.063438    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/603095db-c9ac-44a7-b91a-c8f7ebde9397-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:53:48.064486 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:48.063485    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/603095db-c9ac-44a7-b91a-c8f7ebde9397-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:53:48.064486 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:48.063517    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/603095db-c9ac-44a7-b91a-c8f7ebde9397-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:53:48.064486 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:48.063550    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/603095db-c9ac-44a7-b91a-c8f7ebde9397-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:53:48.064486 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:48.063581    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/603095db-c9ac-44a7-b91a-c8f7ebde9397-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:53:48.064486 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:48.063614    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/603095db-c9ac-44a7-b91a-c8f7ebde9397-config-out\") pod \"prometheus-k8s-0\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:53:48.064486 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:48.063638    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/603095db-c9ac-44a7-b91a-c8f7ebde9397-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:53:48.064486 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:48.063665    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/603095db-c9ac-44a7-b91a-c8f7ebde9397-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:53:48.064486 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:48.063697    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/603095db-c9ac-44a7-b91a-c8f7ebde9397-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:53:48.064486 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:48.063724    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/603095db-c9ac-44a7-b91a-c8f7ebde9397-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:53:48.064486 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:48.063760    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/603095db-c9ac-44a7-b91a-c8f7ebde9397-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:53:48.064486 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:48.063787    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/603095db-c9ac-44a7-b91a-c8f7ebde9397-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:53:48.064486 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:48.063839    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/603095db-c9ac-44a7-b91a-c8f7ebde9397-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:53:48.064486 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:48.063861    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/603095db-c9ac-44a7-b91a-c8f7ebde9397-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:53:48.064486 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:48.063887    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbk6c\" (UniqueName: \"kubernetes.io/projected/603095db-c9ac-44a7-b91a-c8f7ebde9397-kube-api-access-zbk6c\") pod \"prometheus-k8s-0\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:53:48.064486 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:48.063912    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/603095db-c9ac-44a7-b91a-c8f7ebde9397-config\") pod \"prometheus-k8s-0\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:53:48.064486 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:48.063936    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/603095db-c9ac-44a7-b91a-c8f7ebde9397-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:53:48.065466 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:48.063970    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/603095db-c9ac-44a7-b91a-c8f7ebde9397-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:53:48.065466 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:48.063994    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/603095db-c9ac-44a7-b91a-c8f7ebde9397-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:53:48.065466 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:48.064019    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/603095db-c9ac-44a7-b91a-c8f7ebde9397-web-config\") pod \"prometheus-k8s-0\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:53:48.072177 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:48.065660    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/603095db-c9ac-44a7-b91a-c8f7ebde9397-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:53:48.072177 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:48.066938    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/603095db-c9ac-44a7-b91a-c8f7ebde9397-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:53:48.072177 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:48.067528    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/603095db-c9ac-44a7-b91a-c8f7ebde9397-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:53:48.072177 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:48.067969    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/603095db-c9ac-44a7-b91a-c8f7ebde9397-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:53:48.072177 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:48.069135    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/603095db-c9ac-44a7-b91a-c8f7ebde9397-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:53:48.073338 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:48.072776    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/603095db-c9ac-44a7-b91a-c8f7ebde9397-web-config\") pod \"prometheus-k8s-0\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:53:48.073338 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:48.073116    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/603095db-c9ac-44a7-b91a-c8f7ebde9397-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:53:48.074840 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:48.074420    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/603095db-c9ac-44a7-b91a-c8f7ebde9397-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:53:48.074840 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:48.074616    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/603095db-c9ac-44a7-b91a-c8f7ebde9397-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:53:48.076020 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:48.075893    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/603095db-c9ac-44a7-b91a-c8f7ebde9397-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:53:48.077539 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:48.077392    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/603095db-c9ac-44a7-b91a-c8f7ebde9397-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:53:48.079582 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:48.079540    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/603095db-c9ac-44a7-b91a-c8f7ebde9397-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:53:48.081971 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:48.081838    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/603095db-c9ac-44a7-b91a-c8f7ebde9397-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:53:48.081971 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:48.081903    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/603095db-c9ac-44a7-b91a-c8f7ebde9397-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:53:48.082117 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:48.082010    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/603095db-c9ac-44a7-b91a-c8f7ebde9397-config-out\") pod \"prometheus-k8s-0\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:53:48.082117 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:48.082067    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/603095db-c9ac-44a7-b91a-c8f7ebde9397-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:53:48.082618 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:48.082523    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/603095db-c9ac-44a7-b91a-c8f7ebde9397-config\") pod \"prometheus-k8s-0\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:53:48.083010 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:48.082963    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/603095db-c9ac-44a7-b91a-c8f7ebde9397-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:53:48.083371 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:48.083177    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/603095db-c9ac-44a7-b91a-c8f7ebde9397-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:53:48.084682 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:48.084342    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbk6c\" (UniqueName: \"kubernetes.io/projected/603095db-c9ac-44a7-b91a-c8f7ebde9397-kube-api-access-zbk6c\") pod \"prometheus-k8s-0\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:53:48.149185 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:48.146748    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:53:48.460230 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:48.460132    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tlvfd" event={"ID":"68ff2110-81b2-469c-ba95-dab13090e6a8","Type":"ContainerStarted","Data":"72ddbc79fc379cead298971ccf16784e8a1e5ded06bfc93ae1713484ce9a52d1"}
Sep 14 02:53:48.462544 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:48.462469    2564 generic.go:356] "Generic (PLEG): container finished" podID="5960d625-c84e-4102-b89f-c81fd92b3660" containerID="7c7fd69687459ad5a3b83b18c525508626853e58062825dfb74b8f5b4963fa30" exitCode=0
Sep 14 02:53:48.462544 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:48.462535    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5960d625-c84e-4102-b89f-c81fd92b3660","Type":"ContainerDied","Data":"7c7fd69687459ad5a3b83b18c525508626853e58062825dfb74b8f5b4963fa30"}
Sep 14 02:53:48.468100 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:48.468069    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 14 02:53:48.470623 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:48.470387    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rrskf" event={"ID":"09b1dd75-0309-48b0-a9dd-2a997d1444c4","Type":"ContainerStarted","Data":"8ec25dc8faa161c265f1a9b70bab11b31b45cf3a69b8a617fe5a62f8aef8f058"}
Sep 14 02:53:48.470623 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:48.470459    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rrskf" event={"ID":"09b1dd75-0309-48b0-a9dd-2a997d1444c4","Type":"ContainerStarted","Data":"8823ebf883f5f274a9317252d3b198d90a3d2f4a3fda51553340b5df6433a89d"}
Sep 14 02:53:48.470623 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:48.470487    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rrskf" event={"ID":"09b1dd75-0309-48b0-a9dd-2a997d1444c4","Type":"ContainerStarted","Data":"2a53b5a0f0a250dac9224efa0c7be5f54c53aeca32ad217a9c4e300eb17ff523"}
Sep 14 02:53:48.472096 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:48.472071    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-znr76" event={"ID":"349c747c-2054-4d9a-9d6c-9bda627193fd","Type":"ContainerStarted","Data":"38925a7e5497305e78700233b5361109b1b33bf6f9e95f86c453ee422c1f2e0a"}
Sep 14 02:53:48.474701 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:48.474677    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7f54f5c747-7g79s" event={"ID":"bffd2f8e-b7bb-49f5-b563-9c1c1af69a67","Type":"ContainerStarted","Data":"6e8eef1a12b11fbc29eef3b4c19d0039ade312da5e8fafbea9e4e5ba5e258d1e"}
Sep 14 02:53:48.476715 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:48.476667    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zhkkf" event={"ID":"18063d29-58fa-4599-8e9c-93c829fa0fb7","Type":"ContainerStarted","Data":"eb48964c984a35dbaf8abf29f8807875892bc0f8e6abd95b429c1ec95639f415"}
Sep 14 02:53:48.481744 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:48.481326    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kq5nf" event={"ID":"8804d811-79f4-4612-bf40-38c08a04d2c8","Type":"ContainerStarted","Data":"6c20ed27a49500060f5c70e3a17cfe9e591595241d7ea2676f2357791cf7a9a0"}
Sep 14 02:53:48.481744 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:48.481542    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-kq5nf"
Sep 14 02:53:48.483895 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:48.483879    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cd9dc9dbb-256pz" event={"ID":"3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b","Type":"ContainerStarted","Data":"191b491e45f3e873b27b6f81346908c8b8908a90364b8f6def907ad8975bdad7"}
Sep 14 02:53:48.484489 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:48.484463    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cd9dc9dbb-256pz" event={"ID":"3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b","Type":"ContainerStarted","Data":"1ee8edb2b2051afe71650b0b3394b07427f793e9fd28547bc949e3e7541aeff1"}
Sep 14 02:53:48.492999 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:48.492953    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-tlvfd" podStartSLOduration=6.36593745 podStartE2EDuration="7.492937459s" totalImagesPullingTime="1.041877535s" totalInitContainerRuntime="85.122474ms" isStatefulPod=true podCreationTimestamp="2026-09-14 02:53:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:53:48.49104596 +0000 UTC m=+73.872130169" watchObservedRunningTime="2026-09-14 02:53:48.492937459 +0000 UTC m=+73.874021660"
Sep 14 02:53:48.547754 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:48.546779    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-kq5nf" podStartSLOduration=67.830626876 podStartE2EDuration="1m13.546757051s" totalImagesPullingTime="5.716130175s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:52:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:53:48.52327507 +0000 UTC m=+73.904359272" watchObservedRunningTime="2026-09-14 02:53:48.546757051 +0000 UTC m=+73.927841253"
Sep 14 02:53:48.561571 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:48.561516    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-cd9dc9dbb-256pz" podStartSLOduration=1.5614927170000001 podStartE2EDuration="1.561492717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:53:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:53:48.545946413 +0000 UTC m=+73.927030612" watchObservedRunningTime="2026-09-14 02:53:48.561492717 +0000 UTC m=+73.942576919"
Sep 14 02:53:48.562321 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:48.561981    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-zhkkf" podStartSLOduration=67.552760513 podStartE2EDuration="1m13.561967952s" totalImagesPullingTime="6.009207439s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 02:52:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:53:48.560390902 +0000 UTC m=+73.941475102" watchObservedRunningTime="2026-09-14 02:53:48.561967952 +0000 UTC m=+73.943052153"
Sep 14 02:53:48.611324 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:48.610137    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rrskf" podStartSLOduration=3.023389526 podStartE2EDuration="7.610113202s" totalImagesPullingTime="4.586723676s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:53:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:53:48.581813858 +0000 UTC m=+73.962898058" watchObservedRunningTime="2026-09-14 02:53:48.610113202 +0000 UTC m=+73.991197402"
Sep 14 02:53:49.790593 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:53:49.790555    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod603095db_c9ac_44a7_b91a_c8f7ebde9397.slice/crio-6229d8ba8797f5c0a945e5cc423fb3f639e9f652200fc2875650cc251aa6f1fe WatchSource:0}: Error finding container 6229d8ba8797f5c0a945e5cc423fb3f639e9f652200fc2875650cc251aa6f1fe: Status 404 returned error can't find the container with id 6229d8ba8797f5c0a945e5cc423fb3f639e9f652200fc2875650cc251aa6f1fe
Sep 14 02:53:49.944417 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:49.944353    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:50.004661 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:50.003975    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:50.501675 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:50.501597    2564 generic.go:356] "Generic (PLEG): container finished" podID="603095db-c9ac-44a7-b91a-c8f7ebde9397" containerID="35879a0a0769cfc374379b4d93cdab1b2d9bc1d95d51317d090cce1fda427d38" exitCode=0
Sep 14 02:53:50.501948 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:50.501908    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"603095db-c9ac-44a7-b91a-c8f7ebde9397","Type":"ContainerDied","Data":"35879a0a0769cfc374379b4d93cdab1b2d9bc1d95d51317d090cce1fda427d38"}
Sep 14 02:53:50.503458 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:50.503428    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"603095db-c9ac-44a7-b91a-c8f7ebde9397","Type":"ContainerStarted","Data":"6229d8ba8797f5c0a945e5cc423fb3f639e9f652200fc2875650cc251aa6f1fe"}
Sep 14 02:53:50.508078 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:50.508050    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76b45698db-7zh9r" event={"ID":"cc3d5c64-3f98-4267-9469-27eb14252478","Type":"ContainerStarted","Data":"5980fc2e3b857225a3998fbc7358a2044d63f37e850f04e7e4e016dc762f88e7"}
Sep 14 02:53:50.508177 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:50.508119    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76b45698db-7zh9r" event={"ID":"cc3d5c64-3f98-4267-9469-27eb14252478","Type":"ContainerStarted","Data":"ed8f5366ea7bb42c1a19c5ff506c0dcbe65a6894cecdaf4078d2281d634504d4"}
Sep 14 02:53:50.508177 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:50.508149    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76b45698db-7zh9r" event={"ID":"cc3d5c64-3f98-4267-9469-27eb14252478","Type":"ContainerStarted","Data":"46bd855e7a11ff79b2809e26745b89d51d04de20cd66e60eacb4ce4735ff44cb"}
Sep 14 02:53:50.510143 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:50.510117    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-znr76" event={"ID":"349c747c-2054-4d9a-9d6c-9bda627193fd","Type":"ContainerStarted","Data":"313894c7fa3326a2711149a03f178d9c744d15026fca0acc3fad4b43718c0550"}
Sep 14 02:53:50.510375 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:50.510329    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-znr76"
Sep 14 02:53:50.512934 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:50.512470    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7f54f5c747-7g79s" event={"ID":"bffd2f8e-b7bb-49f5-b563-9c1c1af69a67","Type":"ContainerStarted","Data":"dcb0e40c6d9817139ea59761ec9a50cf24735c75326cf9d9c2d6651a8d4da920"}
Sep 14 02:53:50.521511 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:50.521464    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-znr76"
Sep 14 02:53:50.553425 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:50.550551    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-znr76" podStartSLOduration=2.286697766 podStartE2EDuration="4.550529312s" totalImagesPullingTime="2.263831546s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:53:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:53:50.549953815 +0000 UTC m=+75.931038014" watchObservedRunningTime="2026-09-14 02:53:50.550529312 +0000 UTC m=+75.931613511"
Sep 14 02:53:50.568954 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:50.568339    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7f54f5c747-7g79s" podStartSLOduration=3.316629526 podStartE2EDuration="5.568316085s" totalImagesPullingTime="2.251686559s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:53:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:53:50.568159847 +0000 UTC m=+75.949244048" watchObservedRunningTime="2026-09-14 02:53:50.568316085 +0000 UTC m=+75.949400286"
Sep 14 02:53:51.524541 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:51.524511    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5960d625-c84e-4102-b89f-c81fd92b3660","Type":"ContainerStarted","Data":"1afd700924134b81e5c6e0713323c4b6af790388dd2e4dcc5033bfb810b1b3a5"}
Sep 14 02:53:51.524895 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:51.524626    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5960d625-c84e-4102-b89f-c81fd92b3660","Type":"ContainerStarted","Data":"e13cdd9189147c51f3db58945df63ba9d48612c79c087523e2263a1b4c6afe51"}
Sep 14 02:53:51.524895 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:51.524665    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5960d625-c84e-4102-b89f-c81fd92b3660","Type":"ContainerStarted","Data":"9464805b9e81b9870a42b302dec0581d430e70191988504479a1cf2fcaf73a41"}
Sep 14 02:53:51.524895 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:51.524704    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5960d625-c84e-4102-b89f-c81fd92b3660","Type":"ContainerStarted","Data":"dda9129a9368827e8a495a538eb3ffd6e210ccf439618b2de4a77404cf4ae4f6"}
Sep 14 02:53:51.524895 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:51.524742    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5960d625-c84e-4102-b89f-c81fd92b3660","Type":"ContainerStarted","Data":"a07fdab59d76ce07237fe50f28b9a0d6e2cf0cab22f86914abcd9258671f1657"}
Sep 14 02:53:51.524895 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:51.524775    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5960d625-c84e-4102-b89f-c81fd92b3660","Type":"ContainerStarted","Data":"1a1a5de3dff20bb36e14563c750686eeaecb59acd263563732e8f733f5faee4b"}
Sep 14 02:53:51.528229 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:51.528121    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76b45698db-7zh9r" event={"ID":"cc3d5c64-3f98-4267-9469-27eb14252478","Type":"ContainerStarted","Data":"14fd0fd2e3d2ff63cf7c3982c467de949354417775a060ba6e9fce39ee21a004"}
Sep 14 02:53:51.555615 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:51.555574    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.548960855 podStartE2EDuration="9.555556645s" totalImagesPullingTime="5.981953801s" totalInitContainerRuntime="24.641989ms" isStatefulPod=false podCreationTimestamp="2026-09-14 02:53:42 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:53:51.55466624 +0000 UTC m=+76.935750451" watchObservedRunningTime="2026-09-14 02:53:51.555556645 +0000 UTC m=+76.936640867"
Sep 14 02:53:52.110329 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:52.110280    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:52.211661 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:52.211561    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:52.343642 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:52.343610    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:52.444416 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:52.444313    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:52.538303 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:52.538265    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76b45698db-7zh9r" event={"ID":"cc3d5c64-3f98-4267-9469-27eb14252478","Type":"ContainerStarted","Data":"6ad0ee469ea6755060e8fc0fda51cbee54a5eb0ba9e8584cb85a15eff1f8af3e"}
Sep 14 02:53:52.538779 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:52.538345    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76b45698db-7zh9r" event={"ID":"cc3d5c64-3f98-4267-9469-27eb14252478","Type":"ContainerStarted","Data":"11db26a4f43a6778f94b0290dddf55dea43cafdbb47fee3dfbbb08413b364422"}
Sep 14 02:53:52.548934 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:52.548846    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:52.576871 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:52.576806    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-76b45698db-7zh9r" podStartSLOduration=3.073087339 podStartE2EDuration="9.576784221s" totalImagesPullingTime="6.503696882s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:53:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:53:52.5767455 +0000 UTC m=+77.957829701" watchObservedRunningTime="2026-09-14 02:53:52.576784221 +0000 UTC m=+77.957868421"
Sep 14 02:53:53.542557 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:53.542529    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-76b45698db-7zh9r"
Sep 14 02:53:54.549887 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:54.549854    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"603095db-c9ac-44a7-b91a-c8f7ebde9397","Type":"ContainerStarted","Data":"64ee82c7d3547678c022e1726d07c12e9ff159cedfedd39f914cefd522fad67f"}
Sep 14 02:53:54.550303 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:54.549922    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"603095db-c9ac-44a7-b91a-c8f7ebde9397","Type":"ContainerStarted","Data":"34d5b4c42bd7f2ed7ddd63b21861378c290a610fe6fd5186632daef4db9edd81"}
Sep 14 02:53:54.550303 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:54.549960    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"603095db-c9ac-44a7-b91a-c8f7ebde9397","Type":"ContainerStarted","Data":"01104db36bf308971798d8562e27c24d014803f40a52576922181a5634f7ad1e"}
Sep 14 02:53:54.550303 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:54.549989    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"603095db-c9ac-44a7-b91a-c8f7ebde9397","Type":"ContainerStarted","Data":"fdff46c1b857b5255645de42ef2c7c4e76db2a945f92d70903a18d7fdfb86187"}
Sep 14 02:53:54.550303 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:54.550016    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"603095db-c9ac-44a7-b91a-c8f7ebde9397","Type":"ContainerStarted","Data":"53be009b3e5bbced63e16b80179bfef9c84204e00d481429971cc9be6ca7f11d"}
Sep 14 02:53:54.550303 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:54.550042    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"603095db-c9ac-44a7-b91a-c8f7ebde9397","Type":"ContainerStarted","Data":"a29a0eb7517dcc7c19389cb1cd97fb7a0debd5e5672f0383296fb7255a5531f6"}
Sep 14 02:53:54.610002 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:54.609941    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=4.54846959 podStartE2EDuration="7.609912193s" totalImagesPullingTime="3.037536319s" totalInitContainerRuntime="23.906284ms" isStatefulPod=false podCreationTimestamp="2026-09-14 02:53:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:53:54.609442284 +0000 UTC m=+79.990526484" watchObservedRunningTime="2026-09-14 02:53:54.609912193 +0000 UTC m=+79.990996393"
Sep 14 02:53:54.772687 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:54.772650    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:54.984782 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:54.984708    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:55.143265 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:55.143223    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:55.471230 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:55.471197    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:55.576448 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:55.576413    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:57.715249 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:57.715127    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-cd9dc9dbb-256pz"
Sep 14 02:53:57.715249 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:57.715185    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-cd9dc9dbb-256pz"
Sep 14 02:53:57.720210 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:57.720186    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-cd9dc9dbb-256pz"
Sep 14 02:53:58.148158 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:58.148129    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:53:58.570613 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:58.570584    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-cd9dc9dbb-256pz"
Sep 14 02:53:58.626570 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:58.626537    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66f88cd956-494g9"]
Sep 14 02:53:58.757886 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:58.757851    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:58.856223 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:58.856137    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:58.971371 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:58.971333    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:59.076808 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:59.076756    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:59.172940 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:59.172843    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:53:59.556104 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:53:59.556080    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-76b45698db-7zh9r"
Sep 14 02:54:02.799124 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:54:02.799088    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:54:02.869582 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:54:02.869536    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:54:06.305772 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:54:06.305737    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7f54f5c747-7g79s"
Sep 14 02:54:06.306229 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:54:06.305808    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7f54f5c747-7g79s"
Sep 14 02:54:17.163169 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:54:17.163134    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:54:17.804342 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:54:17.804304    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:54:17.865543 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:54:17.865502    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:54:19.492653 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:54:19.492624    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-kq5nf"
Sep 14 02:54:23.645791 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:54:23.645721    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-66f88cd956-494g9" podUID="91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4" containerName="console" containerID="cri-o://303a1b144e71b6b11ce609578ac6d11b07603db9be51b4c6de2a703ca5201522" gracePeriod=15
Sep 14 02:54:23.940595 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:54:23.940567    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66f88cd956-494g9"
Sep 14 02:54:23.989370 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:54:23.989339    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tfhg8\" (UniqueName: \"kubernetes.io/projected/91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4-kube-api-access-tfhg8\") pod \"91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4\" (UID: \"91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4\") "
Sep 14 02:54:23.989524 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:54:23.989375    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4-console-oauth-config\") pod \"91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4\" (UID: \"91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4\") "
Sep 14 02:54:23.989524 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:54:23.989400    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4-service-ca\") pod \"91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4\" (UID: \"91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4\") "
Sep 14 02:54:23.989524 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:54:23.989457    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4-oauth-serving-cert\") pod \"91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4\" (UID: \"91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4\") "
Sep 14 02:54:23.989524 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:54:23.989487    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4-trusted-ca-bundle\") pod \"91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4\" (UID: \"91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4\") "
Sep 14 02:54:23.989524 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:54:23.989511    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4-console-config\") pod \"91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4\" (UID: \"91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4\") "
Sep 14 02:54:23.989770 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:54:23.989656    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4-console-serving-cert\") pod \"91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4\" (UID: \"91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4\") "
Sep 14 02:54:23.989886 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:54:23.989859    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4-service-ca" (OuterVolumeSpecName: "service-ca") pod "91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4" (UID: "91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 02:54:23.989959 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:54:23.989892    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4" (UID: "91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 02:54:23.989959 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:54:23.989905    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4-console-config" (OuterVolumeSpecName: "console-config") pod "91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4" (UID: "91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 02:54:23.989959 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:54:23.989912    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4" (UID: "91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 02:54:23.990116 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:54:23.990012    2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4-service-ca\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 02:54:23.990116 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:54:23.990026    2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4-oauth-serving-cert\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 02:54:23.990116 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:54:23.990034    2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4-trusted-ca-bundle\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 02:54:23.990116 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:54:23.990046    2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4-console-config\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 02:54:23.991641 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:54:23.991612    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4-kube-api-access-tfhg8" (OuterVolumeSpecName: "kube-api-access-tfhg8") pod "91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4" (UID: "91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4"). InnerVolumeSpecName "kube-api-access-tfhg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 02:54:23.991716 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:54:23.991670    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4" (UID: "91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 02:54:23.991829 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:54:23.991808    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4" (UID: "91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 02:54:24.091328 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:54:24.091294    2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4-console-serving-cert\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 02:54:24.091328 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:54:24.091322    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfhg8\" (UniqueName: \"kubernetes.io/projected/91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4-kube-api-access-tfhg8\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 02:54:24.091328 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:54:24.091332    2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4-console-oauth-config\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 02:54:24.671143 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:54:24.671108    2564 generic.go:356] "Generic (PLEG): container finished" podID="91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4" containerID="303a1b144e71b6b11ce609578ac6d11b07603db9be51b4c6de2a703ca5201522" exitCode=0
Sep 14 02:54:24.671554 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:54:24.671157    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66f88cd956-494g9"
Sep 14 02:54:24.671554 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:54:24.671189    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66f88cd956-494g9" event={"ID":"91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4","Type":"ContainerDied","Data":"303a1b144e71b6b11ce609578ac6d11b07603db9be51b4c6de2a703ca5201522"}
Sep 14 02:54:24.671554 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:54:24.671237    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66f88cd956-494g9" event={"ID":"91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4","Type":"ContainerDied","Data":"68235fa15dddd381debdeabc5b3a518a4c47503999f9db852cb808e2f78cc7e9"}
Sep 14 02:54:24.671554 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:54:24.671266    2564 scope.go:117] "RemoveContainer" containerID="303a1b144e71b6b11ce609578ac6d11b07603db9be51b4c6de2a703ca5201522"
Sep 14 02:54:24.680070 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:54:24.680056    2564 scope.go:117] "RemoveContainer" containerID="303a1b144e71b6b11ce609578ac6d11b07603db9be51b4c6de2a703ca5201522"
Sep 14 02:54:24.680327 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:54:24.680309    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"303a1b144e71b6b11ce609578ac6d11b07603db9be51b4c6de2a703ca5201522\": container with ID starting with 303a1b144e71b6b11ce609578ac6d11b07603db9be51b4c6de2a703ca5201522 not found: ID does not exist" containerID="303a1b144e71b6b11ce609578ac6d11b07603db9be51b4c6de2a703ca5201522"
Sep 14 02:54:24.680385 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:54:24.680335    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"303a1b144e71b6b11ce609578ac6d11b07603db9be51b4c6de2a703ca5201522"} err="failed to get container status \"303a1b144e71b6b11ce609578ac6d11b07603db9be51b4c6de2a703ca5201522\": rpc error: code = NotFound desc = could not find container \"303a1b144e71b6b11ce609578ac6d11b07603db9be51b4c6de2a703ca5201522\": container with ID starting with 303a1b144e71b6b11ce609578ac6d11b07603db9be51b4c6de2a703ca5201522 not found: ID does not exist"
Sep 14 02:54:24.696520 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:54:24.696499    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66f88cd956-494g9"]
Sep 14 02:54:24.706134 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:54:24.706112    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-66f88cd956-494g9"]
Sep 14 02:54:25.098313 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:54:25.098280    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4" path="/var/lib/kubelet/pods/91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4/volumes"
Sep 14 02:54:26.310633 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:54:26.310608    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7f54f5c747-7g79s"
Sep 14 02:54:26.314362 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:54:26.314339    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7f54f5c747-7g79s"
Sep 14 02:54:48.147826 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:54:48.147794    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:54:48.166449 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:54:48.166425    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:54:48.780626 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:54:48.780593    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:55:00.791584 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:00.791546    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:55:01.234961 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:01.234870    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:55:01.857276 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:01.857240    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 14 02:55:01.857683 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:01.857526    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5960d625-c84e-4102-b89f-c81fd92b3660" containerName="alertmanager" containerID="cri-o://1a1a5de3dff20bb36e14563c750686eeaecb59acd263563732e8f733f5faee4b" gracePeriod=120
Sep 14 02:55:01.857683 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:01.857590    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5960d625-c84e-4102-b89f-c81fd92b3660" containerName="kube-rbac-proxy-metric" containerID="cri-o://e13cdd9189147c51f3db58945df63ba9d48612c79c087523e2263a1b4c6afe51" gracePeriod=120
Sep 14 02:55:01.857683 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:01.857657    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5960d625-c84e-4102-b89f-c81fd92b3660" containerName="kube-rbac-proxy-web" containerID="cri-o://dda9129a9368827e8a495a538eb3ffd6e210ccf439618b2de4a77404cf4ae4f6" gracePeriod=120
Sep 14 02:55:01.857854 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:01.857654    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5960d625-c84e-4102-b89f-c81fd92b3660" containerName="kube-rbac-proxy" containerID="cri-o://9464805b9e81b9870a42b302dec0581d430e70191988504479a1cf2fcaf73a41" gracePeriod=120
Sep 14 02:55:01.857854 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:01.857678    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5960d625-c84e-4102-b89f-c81fd92b3660" containerName="prom-label-proxy" containerID="cri-o://1afd700924134b81e5c6e0713323c4b6af790388dd2e4dcc5033bfb810b1b3a5" gracePeriod=120
Sep 14 02:55:01.857854 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:01.857698    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5960d625-c84e-4102-b89f-c81fd92b3660" containerName="config-reloader" containerID="cri-o://a07fdab59d76ce07237fe50f28b9a0d6e2cf0cab22f86914abcd9258671f1657" gracePeriod=120
Sep 14 02:55:02.825238 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:02.825207    2564 generic.go:356] "Generic (PLEG): container finished" podID="5960d625-c84e-4102-b89f-c81fd92b3660" containerID="1afd700924134b81e5c6e0713323c4b6af790388dd2e4dcc5033bfb810b1b3a5" exitCode=0
Sep 14 02:55:02.825238 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:02.825230    2564 generic.go:356] "Generic (PLEG): container finished" podID="5960d625-c84e-4102-b89f-c81fd92b3660" containerID="e13cdd9189147c51f3db58945df63ba9d48612c79c087523e2263a1b4c6afe51" exitCode=0
Sep 14 02:55:02.825238 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:02.825238    2564 generic.go:356] "Generic (PLEG): container finished" podID="5960d625-c84e-4102-b89f-c81fd92b3660" containerID="9464805b9e81b9870a42b302dec0581d430e70191988504479a1cf2fcaf73a41" exitCode=0
Sep 14 02:55:02.825238 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:02.825244    2564 generic.go:356] "Generic (PLEG): container finished" podID="5960d625-c84e-4102-b89f-c81fd92b3660" containerID="a07fdab59d76ce07237fe50f28b9a0d6e2cf0cab22f86914abcd9258671f1657" exitCode=0
Sep 14 02:55:02.825238 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:02.825249    2564 generic.go:356] "Generic (PLEG): container finished" podID="5960d625-c84e-4102-b89f-c81fd92b3660" containerID="1a1a5de3dff20bb36e14563c750686eeaecb59acd263563732e8f733f5faee4b" exitCode=0
Sep 14 02:55:02.825540 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:02.825279    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5960d625-c84e-4102-b89f-c81fd92b3660","Type":"ContainerDied","Data":"1afd700924134b81e5c6e0713323c4b6af790388dd2e4dcc5033bfb810b1b3a5"}
Sep 14 02:55:02.825540 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:02.825370    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5960d625-c84e-4102-b89f-c81fd92b3660","Type":"ContainerDied","Data":"e13cdd9189147c51f3db58945df63ba9d48612c79c087523e2263a1b4c6afe51"}
Sep 14 02:55:02.825540 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:02.825399    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5960d625-c84e-4102-b89f-c81fd92b3660","Type":"ContainerDied","Data":"9464805b9e81b9870a42b302dec0581d430e70191988504479a1cf2fcaf73a41"}
Sep 14 02:55:02.825540 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:02.825437    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5960d625-c84e-4102-b89f-c81fd92b3660","Type":"ContainerDied","Data":"a07fdab59d76ce07237fe50f28b9a0d6e2cf0cab22f86914abcd9258671f1657"}
Sep 14 02:55:02.825540 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:02.825459    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5960d625-c84e-4102-b89f-c81fd92b3660","Type":"ContainerDied","Data":"1a1a5de3dff20bb36e14563c750686eeaecb59acd263563732e8f733f5faee4b"}
Sep 14 02:55:03.105414 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.105377    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:55:03.216177 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.216142    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5960d625-c84e-4102-b89f-c81fd92b3660-secret-alertmanager-kube-rbac-proxy-metric\") pod \"5960d625-c84e-4102-b89f-c81fd92b3660\" (UID: \"5960d625-c84e-4102-b89f-c81fd92b3660\") "
Sep 14 02:55:03.216346 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.216202    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5960d625-c84e-4102-b89f-c81fd92b3660-secret-alertmanager-kube-rbac-proxy-web\") pod \"5960d625-c84e-4102-b89f-c81fd92b3660\" (UID: \"5960d625-c84e-4102-b89f-c81fd92b3660\") "
Sep 14 02:55:03.216411 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.216362    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5960d625-c84e-4102-b89f-c81fd92b3660-alertmanager-trusted-ca-bundle\") pod \"5960d625-c84e-4102-b89f-c81fd92b3660\" (UID: \"5960d625-c84e-4102-b89f-c81fd92b3660\") "
Sep 14 02:55:03.216472 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.216396    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5960d625-c84e-4102-b89f-c81fd92b3660-secret-alertmanager-kube-rbac-proxy\") pod \"5960d625-c84e-4102-b89f-c81fd92b3660\" (UID: \"5960d625-c84e-4102-b89f-c81fd92b3660\") "
Sep 14 02:55:03.216472 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.216436    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5960d625-c84e-4102-b89f-c81fd92b3660-alertmanager-main-db\") pod \"5960d625-c84e-4102-b89f-c81fd92b3660\" (UID: \"5960d625-c84e-4102-b89f-c81fd92b3660\") "
Sep 14 02:55:03.216575 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.216478    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5960d625-c84e-4102-b89f-c81fd92b3660-web-config\") pod \"5960d625-c84e-4102-b89f-c81fd92b3660\" (UID: \"5960d625-c84e-4102-b89f-c81fd92b3660\") "
Sep 14 02:55:03.216575 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.216508    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5960d625-c84e-4102-b89f-c81fd92b3660-metrics-client-ca\") pod \"5960d625-c84e-4102-b89f-c81fd92b3660\" (UID: \"5960d625-c84e-4102-b89f-c81fd92b3660\") "
Sep 14 02:55:03.216575 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.216532    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5960d625-c84e-4102-b89f-c81fd92b3660-cluster-tls-config\") pod \"5960d625-c84e-4102-b89f-c81fd92b3660\" (UID: \"5960d625-c84e-4102-b89f-c81fd92b3660\") "
Sep 14 02:55:03.216779 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.216596    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5960d625-c84e-4102-b89f-c81fd92b3660-config-volume\") pod \"5960d625-c84e-4102-b89f-c81fd92b3660\" (UID: \"5960d625-c84e-4102-b89f-c81fd92b3660\") "
Sep 14 02:55:03.216779 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.216641    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5960d625-c84e-4102-b89f-c81fd92b3660-tls-assets\") pod \"5960d625-c84e-4102-b89f-c81fd92b3660\" (UID: \"5960d625-c84e-4102-b89f-c81fd92b3660\") "
Sep 14 02:55:03.216779 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.216666    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5960d625-c84e-4102-b89f-c81fd92b3660-config-out\") pod \"5960d625-c84e-4102-b89f-c81fd92b3660\" (UID: \"5960d625-c84e-4102-b89f-c81fd92b3660\") "
Sep 14 02:55:03.216779 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.216694    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5960d625-c84e-4102-b89f-c81fd92b3660-secret-alertmanager-main-tls\") pod \"5960d625-c84e-4102-b89f-c81fd92b3660\" (UID: \"5960d625-c84e-4102-b89f-c81fd92b3660\") "
Sep 14 02:55:03.216779 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.216726    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xtdrl\" (UniqueName: \"kubernetes.io/projected/5960d625-c84e-4102-b89f-c81fd92b3660-kube-api-access-xtdrl\") pod \"5960d625-c84e-4102-b89f-c81fd92b3660\" (UID: \"5960d625-c84e-4102-b89f-c81fd92b3660\") "
Sep 14 02:55:03.216779 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.216748    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5960d625-c84e-4102-b89f-c81fd92b3660-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "5960d625-c84e-4102-b89f-c81fd92b3660" (UID: "5960d625-c84e-4102-b89f-c81fd92b3660"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 02:55:03.217086 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.216996    2564 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5960d625-c84e-4102-b89f-c81fd92b3660-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 02:55:03.217811 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.217545    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5960d625-c84e-4102-b89f-c81fd92b3660-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "5960d625-c84e-4102-b89f-c81fd92b3660" (UID: "5960d625-c84e-4102-b89f-c81fd92b3660"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 02:55:03.217811 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.217644    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5960d625-c84e-4102-b89f-c81fd92b3660-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "5960d625-c84e-4102-b89f-c81fd92b3660" (UID: "5960d625-c84e-4102-b89f-c81fd92b3660"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 02:55:03.219436 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.219376    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5960d625-c84e-4102-b89f-c81fd92b3660-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "5960d625-c84e-4102-b89f-c81fd92b3660" (UID: "5960d625-c84e-4102-b89f-c81fd92b3660"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 02:55:03.220728 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.220656    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5960d625-c84e-4102-b89f-c81fd92b3660-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "5960d625-c84e-4102-b89f-c81fd92b3660" (UID: "5960d625-c84e-4102-b89f-c81fd92b3660"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 02:55:03.220997 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.220904    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5960d625-c84e-4102-b89f-c81fd92b3660-config-out" (OuterVolumeSpecName: "config-out") pod "5960d625-c84e-4102-b89f-c81fd92b3660" (UID: "5960d625-c84e-4102-b89f-c81fd92b3660"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 02:55:03.221218 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.221177    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5960d625-c84e-4102-b89f-c81fd92b3660-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "5960d625-c84e-4102-b89f-c81fd92b3660" (UID: "5960d625-c84e-4102-b89f-c81fd92b3660"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 02:55:03.221313 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.221294    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5960d625-c84e-4102-b89f-c81fd92b3660-config-volume" (OuterVolumeSpecName: "config-volume") pod "5960d625-c84e-4102-b89f-c81fd92b3660" (UID: "5960d625-c84e-4102-b89f-c81fd92b3660"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 02:55:03.221516 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.221490    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5960d625-c84e-4102-b89f-c81fd92b3660-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "5960d625-c84e-4102-b89f-c81fd92b3660" (UID: "5960d625-c84e-4102-b89f-c81fd92b3660"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 02:55:03.221600 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.221580    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5960d625-c84e-4102-b89f-c81fd92b3660-kube-api-access-xtdrl" (OuterVolumeSpecName: "kube-api-access-xtdrl") pod "5960d625-c84e-4102-b89f-c81fd92b3660" (UID: "5960d625-c84e-4102-b89f-c81fd92b3660"). InnerVolumeSpecName "kube-api-access-xtdrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 02:55:03.221712 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.221689    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5960d625-c84e-4102-b89f-c81fd92b3660-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "5960d625-c84e-4102-b89f-c81fd92b3660" (UID: "5960d625-c84e-4102-b89f-c81fd92b3660"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 02:55:03.224695 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.224676    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5960d625-c84e-4102-b89f-c81fd92b3660-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "5960d625-c84e-4102-b89f-c81fd92b3660" (UID: "5960d625-c84e-4102-b89f-c81fd92b3660"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 02:55:03.231626 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.231592    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5960d625-c84e-4102-b89f-c81fd92b3660-web-config" (OuterVolumeSpecName: "web-config") pod "5960d625-c84e-4102-b89f-c81fd92b3660" (UID: "5960d625-c84e-4102-b89f-c81fd92b3660"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 02:55:03.318237 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.318188    2564 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5960d625-c84e-4102-b89f-c81fd92b3660-tls-assets\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 02:55:03.318237 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.318227    2564 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5960d625-c84e-4102-b89f-c81fd92b3660-config-out\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 02:55:03.318237 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.318240    2564 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5960d625-c84e-4102-b89f-c81fd92b3660-secret-alertmanager-main-tls\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 02:55:03.318534 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.318254    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtdrl\" (UniqueName: \"kubernetes.io/projected/5960d625-c84e-4102-b89f-c81fd92b3660-kube-api-access-xtdrl\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 02:55:03.318534 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.318268    2564 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5960d625-c84e-4102-b89f-c81fd92b3660-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 02:55:03.318534 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.318282    2564 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5960d625-c84e-4102-b89f-c81fd92b3660-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 02:55:03.318534 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.318294    2564 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5960d625-c84e-4102-b89f-c81fd92b3660-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 02:55:03.318534 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.318305    2564 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5960d625-c84e-4102-b89f-c81fd92b3660-alertmanager-main-db\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 02:55:03.318534 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.318343    2564 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5960d625-c84e-4102-b89f-c81fd92b3660-web-config\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 02:55:03.318534 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.318355    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5960d625-c84e-4102-b89f-c81fd92b3660-metrics-client-ca\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 02:55:03.318534 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.318366    2564 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5960d625-c84e-4102-b89f-c81fd92b3660-cluster-tls-config\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 02:55:03.318534 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.318379    2564 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5960d625-c84e-4102-b89f-c81fd92b3660-config-volume\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 02:55:03.832989 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.832960    2564 generic.go:356] "Generic (PLEG): container finished" podID="5960d625-c84e-4102-b89f-c81fd92b3660" containerID="dda9129a9368827e8a495a538eb3ffd6e210ccf439618b2de4a77404cf4ae4f6" exitCode=0
Sep 14 02:55:03.833167 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.833049    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:55:03.833167 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.833049    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5960d625-c84e-4102-b89f-c81fd92b3660","Type":"ContainerDied","Data":"dda9129a9368827e8a495a538eb3ffd6e210ccf439618b2de4a77404cf4ae4f6"}
Sep 14 02:55:03.833237 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.833182    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5960d625-c84e-4102-b89f-c81fd92b3660","Type":"ContainerDied","Data":"5754c20ca2aa95033f8330f50555b9fde17af2fc8c3a98c0968e5fefbc1ebacf"}
Sep 14 02:55:03.833237 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.833215    2564 scope.go:117] "RemoveContainer" containerID="1afd700924134b81e5c6e0713323c4b6af790388dd2e4dcc5033bfb810b1b3a5"
Sep 14 02:55:03.842024 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.842002    2564 scope.go:117] "RemoveContainer" containerID="e13cdd9189147c51f3db58945df63ba9d48612c79c087523e2263a1b4c6afe51"
Sep 14 02:55:03.849384 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.849365    2564 scope.go:117] "RemoveContainer" containerID="9464805b9e81b9870a42b302dec0581d430e70191988504479a1cf2fcaf73a41"
Sep 14 02:55:03.857363 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.857344    2564 scope.go:117] "RemoveContainer" containerID="dda9129a9368827e8a495a538eb3ffd6e210ccf439618b2de4a77404cf4ae4f6"
Sep 14 02:55:03.857893 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.857866    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 14 02:55:03.863931 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.863908    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 14 02:55:03.865632 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.865608    2564 scope.go:117] "RemoveContainer" containerID="a07fdab59d76ce07237fe50f28b9a0d6e2cf0cab22f86914abcd9258671f1657"
Sep 14 02:55:03.872852 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.872834    2564 scope.go:117] "RemoveContainer" containerID="1a1a5de3dff20bb36e14563c750686eeaecb59acd263563732e8f733f5faee4b"
Sep 14 02:55:03.880051 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.880035    2564 scope.go:117] "RemoveContainer" containerID="7c7fd69687459ad5a3b83b18c525508626853e58062825dfb74b8f5b4963fa30"
Sep 14 02:55:03.887773 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.887753    2564 scope.go:117] "RemoveContainer" containerID="1afd700924134b81e5c6e0713323c4b6af790388dd2e4dcc5033bfb810b1b3a5"
Sep 14 02:55:03.888063 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:55:03.888045    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1afd700924134b81e5c6e0713323c4b6af790388dd2e4dcc5033bfb810b1b3a5\": container with ID starting with 1afd700924134b81e5c6e0713323c4b6af790388dd2e4dcc5033bfb810b1b3a5 not found: ID does not exist" containerID="1afd700924134b81e5c6e0713323c4b6af790388dd2e4dcc5033bfb810b1b3a5"
Sep 14 02:55:03.888104 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.888072    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1afd700924134b81e5c6e0713323c4b6af790388dd2e4dcc5033bfb810b1b3a5"} err="failed to get container status \"1afd700924134b81e5c6e0713323c4b6af790388dd2e4dcc5033bfb810b1b3a5\": rpc error: code = NotFound desc = could not find container \"1afd700924134b81e5c6e0713323c4b6af790388dd2e4dcc5033bfb810b1b3a5\": container with ID starting with 1afd700924134b81e5c6e0713323c4b6af790388dd2e4dcc5033bfb810b1b3a5 not found: ID does not exist"
Sep 14 02:55:03.888104 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.888092    2564 scope.go:117] "RemoveContainer" containerID="e13cdd9189147c51f3db58945df63ba9d48612c79c087523e2263a1b4c6afe51"
Sep 14 02:55:03.888362 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:55:03.888342    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e13cdd9189147c51f3db58945df63ba9d48612c79c087523e2263a1b4c6afe51\": container with ID starting with e13cdd9189147c51f3db58945df63ba9d48612c79c087523e2263a1b4c6afe51 not found: ID does not exist" containerID="e13cdd9189147c51f3db58945df63ba9d48612c79c087523e2263a1b4c6afe51"
Sep 14 02:55:03.888435 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.888372    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e13cdd9189147c51f3db58945df63ba9d48612c79c087523e2263a1b4c6afe51"} err="failed to get container status \"e13cdd9189147c51f3db58945df63ba9d48612c79c087523e2263a1b4c6afe51\": rpc error: code = NotFound desc = could not find container \"e13cdd9189147c51f3db58945df63ba9d48612c79c087523e2263a1b4c6afe51\": container with ID starting with e13cdd9189147c51f3db58945df63ba9d48612c79c087523e2263a1b4c6afe51 not found: ID does not exist"
Sep 14 02:55:03.888435 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.888388    2564 scope.go:117] "RemoveContainer" containerID="9464805b9e81b9870a42b302dec0581d430e70191988504479a1cf2fcaf73a41"
Sep 14 02:55:03.888673 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:55:03.888658    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9464805b9e81b9870a42b302dec0581d430e70191988504479a1cf2fcaf73a41\": container with ID starting with 9464805b9e81b9870a42b302dec0581d430e70191988504479a1cf2fcaf73a41 not found: ID does not exist" containerID="9464805b9e81b9870a42b302dec0581d430e70191988504479a1cf2fcaf73a41"
Sep 14 02:55:03.888710 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.888676    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9464805b9e81b9870a42b302dec0581d430e70191988504479a1cf2fcaf73a41"} err="failed to get container status \"9464805b9e81b9870a42b302dec0581d430e70191988504479a1cf2fcaf73a41\": rpc error: code = NotFound desc = could not find container \"9464805b9e81b9870a42b302dec0581d430e70191988504479a1cf2fcaf73a41\": container with ID starting with 9464805b9e81b9870a42b302dec0581d430e70191988504479a1cf2fcaf73a41 not found: ID does not exist"
Sep 14 02:55:03.888710 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.888689    2564 scope.go:117] "RemoveContainer" containerID="dda9129a9368827e8a495a538eb3ffd6e210ccf439618b2de4a77404cf4ae4f6"
Sep 14 02:55:03.888921 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:55:03.888905    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dda9129a9368827e8a495a538eb3ffd6e210ccf439618b2de4a77404cf4ae4f6\": container with ID starting with dda9129a9368827e8a495a538eb3ffd6e210ccf439618b2de4a77404cf4ae4f6 not found: ID does not exist" containerID="dda9129a9368827e8a495a538eb3ffd6e210ccf439618b2de4a77404cf4ae4f6"
Sep 14 02:55:03.888964 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.888925    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dda9129a9368827e8a495a538eb3ffd6e210ccf439618b2de4a77404cf4ae4f6"} err="failed to get container status \"dda9129a9368827e8a495a538eb3ffd6e210ccf439618b2de4a77404cf4ae4f6\": rpc error: code = NotFound desc = could not find container \"dda9129a9368827e8a495a538eb3ffd6e210ccf439618b2de4a77404cf4ae4f6\": container with ID starting with dda9129a9368827e8a495a538eb3ffd6e210ccf439618b2de4a77404cf4ae4f6 not found: ID does not exist"
Sep 14 02:55:03.888964 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.888937    2564 scope.go:117] "RemoveContainer" containerID="a07fdab59d76ce07237fe50f28b9a0d6e2cf0cab22f86914abcd9258671f1657"
Sep 14 02:55:03.889163 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:55:03.889149    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a07fdab59d76ce07237fe50f28b9a0d6e2cf0cab22f86914abcd9258671f1657\": container with ID starting with a07fdab59d76ce07237fe50f28b9a0d6e2cf0cab22f86914abcd9258671f1657 not found: ID does not exist" containerID="a07fdab59d76ce07237fe50f28b9a0d6e2cf0cab22f86914abcd9258671f1657"
Sep 14 02:55:03.889203 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.889167    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a07fdab59d76ce07237fe50f28b9a0d6e2cf0cab22f86914abcd9258671f1657"} err="failed to get container status \"a07fdab59d76ce07237fe50f28b9a0d6e2cf0cab22f86914abcd9258671f1657\": rpc error: code = NotFound desc = could not find container \"a07fdab59d76ce07237fe50f28b9a0d6e2cf0cab22f86914abcd9258671f1657\": container with ID starting with a07fdab59d76ce07237fe50f28b9a0d6e2cf0cab22f86914abcd9258671f1657 not found: ID does not exist"
Sep 14 02:55:03.889203 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.889180    2564 scope.go:117] "RemoveContainer" containerID="1a1a5de3dff20bb36e14563c750686eeaecb59acd263563732e8f733f5faee4b"
Sep 14 02:55:03.889383 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:55:03.889368    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a1a5de3dff20bb36e14563c750686eeaecb59acd263563732e8f733f5faee4b\": container with ID starting with 1a1a5de3dff20bb36e14563c750686eeaecb59acd263563732e8f733f5faee4b not found: ID does not exist" containerID="1a1a5de3dff20bb36e14563c750686eeaecb59acd263563732e8f733f5faee4b"
Sep 14 02:55:03.889442 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.889390    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a1a5de3dff20bb36e14563c750686eeaecb59acd263563732e8f733f5faee4b"} err="failed to get container status \"1a1a5de3dff20bb36e14563c750686eeaecb59acd263563732e8f733f5faee4b\": rpc error: code = NotFound desc = could not find container \"1a1a5de3dff20bb36e14563c750686eeaecb59acd263563732e8f733f5faee4b\": container with ID starting with 1a1a5de3dff20bb36e14563c750686eeaecb59acd263563732e8f733f5faee4b not found: ID does not exist"
Sep 14 02:55:03.889442 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.889414    2564 scope.go:117] "RemoveContainer" containerID="7c7fd69687459ad5a3b83b18c525508626853e58062825dfb74b8f5b4963fa30"
Sep 14 02:55:03.889618 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:55:03.889604    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c7fd69687459ad5a3b83b18c525508626853e58062825dfb74b8f5b4963fa30\": container with ID starting with 7c7fd69687459ad5a3b83b18c525508626853e58062825dfb74b8f5b4963fa30 not found: ID does not exist" containerID="7c7fd69687459ad5a3b83b18c525508626853e58062825dfb74b8f5b4963fa30"
Sep 14 02:55:03.889660 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.889622    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c7fd69687459ad5a3b83b18c525508626853e58062825dfb74b8f5b4963fa30"} err="failed to get container status \"7c7fd69687459ad5a3b83b18c525508626853e58062825dfb74b8f5b4963fa30\": rpc error: code = NotFound desc = could not find container \"7c7fd69687459ad5a3b83b18c525508626853e58062825dfb74b8f5b4963fa30\": container with ID starting with 7c7fd69687459ad5a3b83b18c525508626853e58062825dfb74b8f5b4963fa30 not found: ID does not exist"
Sep 14 02:55:03.893866 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.893844    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 14 02:55:03.894146 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.894134    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5960d625-c84e-4102-b89f-c81fd92b3660" containerName="kube-rbac-proxy-metric"
Sep 14 02:55:03.894188 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.894149    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5960d625-c84e-4102-b89f-c81fd92b3660" containerName="kube-rbac-proxy-metric"
Sep 14 02:55:03.894188 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.894157    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5960d625-c84e-4102-b89f-c81fd92b3660" containerName="alertmanager"
Sep 14 02:55:03.894188 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.894164    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5960d625-c84e-4102-b89f-c81fd92b3660" containerName="alertmanager"
Sep 14 02:55:03.894188 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.894176    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5960d625-c84e-4102-b89f-c81fd92b3660" containerName="prom-label-proxy"
Sep 14 02:55:03.894188 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.894181    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5960d625-c84e-4102-b89f-c81fd92b3660" containerName="prom-label-proxy"
Sep 14 02:55:03.894335 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.894194    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5960d625-c84e-4102-b89f-c81fd92b3660" containerName="config-reloader"
Sep 14 02:55:03.894335 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.894199    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5960d625-c84e-4102-b89f-c81fd92b3660" containerName="config-reloader"
Sep 14 02:55:03.894335 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.894206    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5960d625-c84e-4102-b89f-c81fd92b3660" containerName="init-config-reloader"
Sep 14 02:55:03.894335 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.894211    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5960d625-c84e-4102-b89f-c81fd92b3660" containerName="init-config-reloader"
Sep 14 02:55:03.894335 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.894218    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5960d625-c84e-4102-b89f-c81fd92b3660" containerName="kube-rbac-proxy-web"
Sep 14 02:55:03.894335 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.894223    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5960d625-c84e-4102-b89f-c81fd92b3660" containerName="kube-rbac-proxy-web"
Sep 14 02:55:03.894335 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.894235    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5960d625-c84e-4102-b89f-c81fd92b3660" containerName="kube-rbac-proxy"
Sep 14 02:55:03.894335 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.894240    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5960d625-c84e-4102-b89f-c81fd92b3660" containerName="kube-rbac-proxy"
Sep 14 02:55:03.894335 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.894246    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4" containerName="console"
Sep 14 02:55:03.894335 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.894251    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4" containerName="console"
Sep 14 02:55:03.894335 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.894293    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5960d625-c84e-4102-b89f-c81fd92b3660" containerName="config-reloader"
Sep 14 02:55:03.894335 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.894300    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5960d625-c84e-4102-b89f-c81fd92b3660" containerName="kube-rbac-proxy"
Sep 14 02:55:03.894335 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.894304    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5960d625-c84e-4102-b89f-c81fd92b3660" containerName="prom-label-proxy"
Sep 14 02:55:03.894335 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.894310    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="91a8ab63-3f50-4e46-a9c6-0bb59e2d45a4" containerName="console"
Sep 14 02:55:03.894335 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.894317    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5960d625-c84e-4102-b89f-c81fd92b3660" containerName="kube-rbac-proxy-web"
Sep 14 02:55:03.894335 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.894321    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5960d625-c84e-4102-b89f-c81fd92b3660" containerName="kube-rbac-proxy-metric"
Sep 14 02:55:03.894335 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.894328    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5960d625-c84e-4102-b89f-c81fd92b3660" containerName="alertmanager"
Sep 14 02:55:03.900110 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.898992    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:55:03.902282 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.902111    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-gkbx9\""
Sep 14 02:55:03.902282 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.902110    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 14 02:55:03.902282 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.902152    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 14 02:55:03.902282 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.902177    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 14 02:55:03.902282 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.902194    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 14 02:55:03.902282 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.902111    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 14 02:55:03.902282 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.902157    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 14 02:55:03.902282 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.902271    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 14 02:55:03.902889 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.902540    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 14 02:55:03.908515 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.908492    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 14 02:55:03.913750 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:03.913729    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 14 02:55:04.023751 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:04.023716    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ee2d96e8-59f2-47f6-af92-13b693d4d7ed-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ee2d96e8-59f2-47f6-af92-13b693d4d7ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:55:04.023751 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:04.023759    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ee2d96e8-59f2-47f6-af92-13b693d4d7ed-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ee2d96e8-59f2-47f6-af92-13b693d4d7ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:55:04.023962 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:04.023787    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ee2d96e8-59f2-47f6-af92-13b693d4d7ed-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ee2d96e8-59f2-47f6-af92-13b693d4d7ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:55:04.023962 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:04.023850    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ee2d96e8-59f2-47f6-af92-13b693d4d7ed-config-volume\") pod \"alertmanager-main-0\" (UID: \"ee2d96e8-59f2-47f6-af92-13b693d4d7ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:55:04.023962 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:04.023900    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee2d96e8-59f2-47f6-af92-13b693d4d7ed-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ee2d96e8-59f2-47f6-af92-13b693d4d7ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:55:04.023962 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:04.023933    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ee2d96e8-59f2-47f6-af92-13b693d4d7ed-config-out\") pod \"alertmanager-main-0\" (UID: \"ee2d96e8-59f2-47f6-af92-13b693d4d7ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:55:04.023962 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:04.023951    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ee2d96e8-59f2-47f6-af92-13b693d4d7ed-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ee2d96e8-59f2-47f6-af92-13b693d4d7ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:55:04.024221 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:04.023973    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ee2d96e8-59f2-47f6-af92-13b693d4d7ed-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ee2d96e8-59f2-47f6-af92-13b693d4d7ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:55:04.024221 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:04.024010    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ee2d96e8-59f2-47f6-af92-13b693d4d7ed-web-config\") pod \"alertmanager-main-0\" (UID: \"ee2d96e8-59f2-47f6-af92-13b693d4d7ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:55:04.024221 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:04.024037    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ee2d96e8-59f2-47f6-af92-13b693d4d7ed-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ee2d96e8-59f2-47f6-af92-13b693d4d7ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:55:04.024221 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:04.024055    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ee2d96e8-59f2-47f6-af92-13b693d4d7ed-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ee2d96e8-59f2-47f6-af92-13b693d4d7ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:55:04.024221 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:04.024112    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ee2d96e8-59f2-47f6-af92-13b693d4d7ed-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ee2d96e8-59f2-47f6-af92-13b693d4d7ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:55:04.024221 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:04.024160    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82nqw\" (UniqueName: \"kubernetes.io/projected/ee2d96e8-59f2-47f6-af92-13b693d4d7ed-kube-api-access-82nqw\") pod \"alertmanager-main-0\" (UID: \"ee2d96e8-59f2-47f6-af92-13b693d4d7ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:55:04.124947 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:04.124851    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ee2d96e8-59f2-47f6-af92-13b693d4d7ed-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ee2d96e8-59f2-47f6-af92-13b693d4d7ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:55:04.124947 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:04.124893    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ee2d96e8-59f2-47f6-af92-13b693d4d7ed-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ee2d96e8-59f2-47f6-af92-13b693d4d7ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:55:04.124947 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:04.124912    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ee2d96e8-59f2-47f6-af92-13b693d4d7ed-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ee2d96e8-59f2-47f6-af92-13b693d4d7ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:55:04.124947 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:04.124938    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82nqw\" (UniqueName: \"kubernetes.io/projected/ee2d96e8-59f2-47f6-af92-13b693d4d7ed-kube-api-access-82nqw\") pod \"alertmanager-main-0\" (UID: \"ee2d96e8-59f2-47f6-af92-13b693d4d7ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:55:04.125589 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:04.124974    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ee2d96e8-59f2-47f6-af92-13b693d4d7ed-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ee2d96e8-59f2-47f6-af92-13b693d4d7ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:55:04.125589 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:04.125008    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ee2d96e8-59f2-47f6-af92-13b693d4d7ed-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ee2d96e8-59f2-47f6-af92-13b693d4d7ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:55:04.125589 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:04.125083    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ee2d96e8-59f2-47f6-af92-13b693d4d7ed-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ee2d96e8-59f2-47f6-af92-13b693d4d7ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:55:04.125589 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:04.125143    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ee2d96e8-59f2-47f6-af92-13b693d4d7ed-config-volume\") pod \"alertmanager-main-0\" (UID: \"ee2d96e8-59f2-47f6-af92-13b693d4d7ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:55:04.125589 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:04.125182    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee2d96e8-59f2-47f6-af92-13b693d4d7ed-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ee2d96e8-59f2-47f6-af92-13b693d4d7ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:55:04.125589 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:04.125228    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ee2d96e8-59f2-47f6-af92-13b693d4d7ed-config-out\") pod \"alertmanager-main-0\" (UID: \"ee2d96e8-59f2-47f6-af92-13b693d4d7ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:55:04.125589 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:04.125257    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ee2d96e8-59f2-47f6-af92-13b693d4d7ed-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ee2d96e8-59f2-47f6-af92-13b693d4d7ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:55:04.125589 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:04.125280    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ee2d96e8-59f2-47f6-af92-13b693d4d7ed-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ee2d96e8-59f2-47f6-af92-13b693d4d7ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:55:04.125589 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:04.125305    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ee2d96e8-59f2-47f6-af92-13b693d4d7ed-web-config\") pod \"alertmanager-main-0\" (UID: \"ee2d96e8-59f2-47f6-af92-13b693d4d7ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:55:04.126074 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:04.125891    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ee2d96e8-59f2-47f6-af92-13b693d4d7ed-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ee2d96e8-59f2-47f6-af92-13b693d4d7ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:55:04.128006 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:04.127982    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ee2d96e8-59f2-47f6-af92-13b693d4d7ed-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ee2d96e8-59f2-47f6-af92-13b693d4d7ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:55:04.128006 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:04.128012    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ee2d96e8-59f2-47f6-af92-13b693d4d7ed-config-out\") pod \"alertmanager-main-0\" (UID: \"ee2d96e8-59f2-47f6-af92-13b693d4d7ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:55:04.128006 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:04.128091    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ee2d96e8-59f2-47f6-af92-13b693d4d7ed-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ee2d96e8-59f2-47f6-af92-13b693d4d7ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:55:04.128006 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:04.128163    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ee2d96e8-59f2-47f6-af92-13b693d4d7ed-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ee2d96e8-59f2-47f6-af92-13b693d4d7ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:55:04.128498 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:04.128343    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ee2d96e8-59f2-47f6-af92-13b693d4d7ed-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ee2d96e8-59f2-47f6-af92-13b693d4d7ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:55:04.128671 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:04.128648    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ee2d96e8-59f2-47f6-af92-13b693d4d7ed-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ee2d96e8-59f2-47f6-af92-13b693d4d7ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:55:04.128671 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:04.128664    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ee2d96e8-59f2-47f6-af92-13b693d4d7ed-config-volume\") pod \"alertmanager-main-0\" (UID: \"ee2d96e8-59f2-47f6-af92-13b693d4d7ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:55:04.128802 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:04.128683    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ee2d96e8-59f2-47f6-af92-13b693d4d7ed-web-config\") pod \"alertmanager-main-0\" (UID: \"ee2d96e8-59f2-47f6-af92-13b693d4d7ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:55:04.128802 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:04.128767    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ee2d96e8-59f2-47f6-af92-13b693d4d7ed-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ee2d96e8-59f2-47f6-af92-13b693d4d7ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:55:04.128877 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:04.128829    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee2d96e8-59f2-47f6-af92-13b693d4d7ed-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ee2d96e8-59f2-47f6-af92-13b693d4d7ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:55:04.130241 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:04.130223    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ee2d96e8-59f2-47f6-af92-13b693d4d7ed-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ee2d96e8-59f2-47f6-af92-13b693d4d7ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:55:04.133950 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:04.133928    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82nqw\" (UniqueName: \"kubernetes.io/projected/ee2d96e8-59f2-47f6-af92-13b693d4d7ed-kube-api-access-82nqw\") pod \"alertmanager-main-0\" (UID: \"ee2d96e8-59f2-47f6-af92-13b693d4d7ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:55:04.210120 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:04.210088    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 14 02:55:04.489847 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:04.489803    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 14 02:55:04.492332 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:55:04.492309    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee2d96e8_59f2_47f6_af92_13b693d4d7ed.slice/crio-cf7023f6e88f0e5e69933516a3f1478ff0b0d5d43c3b1ad77404923b296a3cb4 WatchSource:0}: Error finding container cf7023f6e88f0e5e69933516a3f1478ff0b0d5d43c3b1ad77404923b296a3cb4: Status 404 returned error can't find the container with id cf7023f6e88f0e5e69933516a3f1478ff0b0d5d43c3b1ad77404923b296a3cb4
Sep 14 02:55:04.840351 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:04.840315    2564 generic.go:356] "Generic (PLEG): container finished" podID="ee2d96e8-59f2-47f6-af92-13b693d4d7ed" containerID="3e78ab418d9497a2e1a3831cd7e836fe5c30af8c1092efddc90f9f397f5b353c" exitCode=0
Sep 14 02:55:04.840576 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:04.840397    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ee2d96e8-59f2-47f6-af92-13b693d4d7ed","Type":"ContainerDied","Data":"3e78ab418d9497a2e1a3831cd7e836fe5c30af8c1092efddc90f9f397f5b353c"}
Sep 14 02:55:04.840576 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:04.840523    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ee2d96e8-59f2-47f6-af92-13b693d4d7ed","Type":"ContainerStarted","Data":"cf7023f6e88f0e5e69933516a3f1478ff0b0d5d43c3b1ad77404923b296a3cb4"}
Sep 14 02:55:05.099806 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:05.099778    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5960d625-c84e-4102-b89f-c81fd92b3660" path="/var/lib/kubelet/pods/5960d625-c84e-4102-b89f-c81fd92b3660/volumes"
Sep 14 02:55:05.855703 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:05.855668    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ee2d96e8-59f2-47f6-af92-13b693d4d7ed","Type":"ContainerStarted","Data":"3bf5a489d527668f205880e60a67c5ac20cf006299a3b0eecbfb5bfbfd6e4232"}
Sep 14 02:55:05.856070 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:05.855754    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ee2d96e8-59f2-47f6-af92-13b693d4d7ed","Type":"ContainerStarted","Data":"7232e8a063ed26ffb08cb6ab72c952e239e1fcbfcd25ead45e88a49de7b959f2"}
Sep 14 02:55:05.856070 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:05.855785    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ee2d96e8-59f2-47f6-af92-13b693d4d7ed","Type":"ContainerStarted","Data":"b3a01dfa03aad322ff1cf03debba0e06728fe1fd42413ebb0ce9d30dbff9a19c"}
Sep 14 02:55:05.856070 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:05.855820    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ee2d96e8-59f2-47f6-af92-13b693d4d7ed","Type":"ContainerStarted","Data":"cced573aec11e659375c5346c9ea0ce6577b4231bacd2a463778a50e7c3ff804"}
Sep 14 02:55:05.856070 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:05.855847    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ee2d96e8-59f2-47f6-af92-13b693d4d7ed","Type":"ContainerStarted","Data":"d2f3c15b503ed95e4dfae0d2fe3e9108c04b695211e3b16195cf3d2fdeb3bc3b"}
Sep 14 02:55:05.856070 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:05.855871    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ee2d96e8-59f2-47f6-af92-13b693d4d7ed","Type":"ContainerStarted","Data":"af21801d9240519a171a209776cd19093cc227d2b7cd117210e90d3b40657bc2"}
Sep 14 02:55:06.050385 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.050326    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.035440681 podStartE2EDuration="3.050304529s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.863848ms" isStatefulPod=false podCreationTimestamp="2026-09-14 02:55:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:55:05.917177746 +0000 UTC m=+151.298261960" watchObservedRunningTime="2026-09-14 02:55:06.050304529 +0000 UTC m=+151.431388728"
Sep 14 02:55:06.051181 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.051163    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-df686ffff-z4729"]
Sep 14 02:55:06.054646 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.054631    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-df686ffff-z4729"
Sep 14 02:55:06.058429 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.058394    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-hp6kn\""
Sep 14 02:55:06.058632 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.058618    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 14 02:55:06.059262 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.059244    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 14 02:55:06.059262 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.059260    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 14 02:55:06.059821 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.059804    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 14 02:55:06.060221 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.060200    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 14 02:55:06.065120 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.065100    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 14 02:55:06.080820 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.080778    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-df686ffff-z4729"]
Sep 14 02:55:06.245870 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.245775    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1dffd99b-f463-4251-96de-0b8d7f6d5b71-metrics-client-ca\") pod \"telemeter-client-df686ffff-z4729\" (UID: \"1dffd99b-f463-4251-96de-0b8d7f6d5b71\") " pod="openshift-monitoring/telemeter-client-df686ffff-z4729"
Sep 14 02:55:06.245870 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.245818    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/1dffd99b-f463-4251-96de-0b8d7f6d5b71-telemeter-client-tls\") pod \"telemeter-client-df686ffff-z4729\" (UID: \"1dffd99b-f463-4251-96de-0b8d7f6d5b71\") " pod="openshift-monitoring/telemeter-client-df686ffff-z4729"
Sep 14 02:55:06.246094 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.245874    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/1dffd99b-f463-4251-96de-0b8d7f6d5b71-federate-client-tls\") pod \"telemeter-client-df686ffff-z4729\" (UID: \"1dffd99b-f463-4251-96de-0b8d7f6d5b71\") " pod="openshift-monitoring/telemeter-client-df686ffff-z4729"
Sep 14 02:55:06.246094 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.245925    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1dffd99b-f463-4251-96de-0b8d7f6d5b71-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-df686ffff-z4729\" (UID: \"1dffd99b-f463-4251-96de-0b8d7f6d5b71\") " pod="openshift-monitoring/telemeter-client-df686ffff-z4729"
Sep 14 02:55:06.246094 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.245948    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcfnh\" (UniqueName: \"kubernetes.io/projected/1dffd99b-f463-4251-96de-0b8d7f6d5b71-kube-api-access-rcfnh\") pod \"telemeter-client-df686ffff-z4729\" (UID: \"1dffd99b-f463-4251-96de-0b8d7f6d5b71\") " pod="openshift-monitoring/telemeter-client-df686ffff-z4729"
Sep 14 02:55:06.246094 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.245974    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1dffd99b-f463-4251-96de-0b8d7f6d5b71-serving-certs-ca-bundle\") pod \"telemeter-client-df686ffff-z4729\" (UID: \"1dffd99b-f463-4251-96de-0b8d7f6d5b71\") " pod="openshift-monitoring/telemeter-client-df686ffff-z4729"
Sep 14 02:55:06.246094 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.246042    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/1dffd99b-f463-4251-96de-0b8d7f6d5b71-secret-telemeter-client\") pod \"telemeter-client-df686ffff-z4729\" (UID: \"1dffd99b-f463-4251-96de-0b8d7f6d5b71\") " pod="openshift-monitoring/telemeter-client-df686ffff-z4729"
Sep 14 02:55:06.246094 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.246078    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1dffd99b-f463-4251-96de-0b8d7f6d5b71-telemeter-trusted-ca-bundle\") pod \"telemeter-client-df686ffff-z4729\" (UID: \"1dffd99b-f463-4251-96de-0b8d7f6d5b71\") " pod="openshift-monitoring/telemeter-client-df686ffff-z4729"
Sep 14 02:55:06.303304 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.303264    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 14 02:55:06.303650 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.303621    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="603095db-c9ac-44a7-b91a-c8f7ebde9397" containerName="prometheus" containerID="cri-o://a29a0eb7517dcc7c19389cb1cd97fb7a0debd5e5672f0383296fb7255a5531f6" gracePeriod=600
Sep 14 02:55:06.303749 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.303648    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="603095db-c9ac-44a7-b91a-c8f7ebde9397" containerName="kube-rbac-proxy" containerID="cri-o://34d5b4c42bd7f2ed7ddd63b21861378c290a610fe6fd5186632daef4db9edd81" gracePeriod=600
Sep 14 02:55:06.303749 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.303674    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="603095db-c9ac-44a7-b91a-c8f7ebde9397" containerName="thanos-sidecar" containerID="cri-o://fdff46c1b857b5255645de42ef2c7c4e76db2a945f92d70903a18d7fdfb86187" gracePeriod=600
Sep 14 02:55:06.303749 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.303714    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="603095db-c9ac-44a7-b91a-c8f7ebde9397" containerName="kube-rbac-proxy-thanos" containerID="cri-o://64ee82c7d3547678c022e1726d07c12e9ff159cedfedd39f914cefd522fad67f" gracePeriod=600
Sep 14 02:55:06.303917 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.303721    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="603095db-c9ac-44a7-b91a-c8f7ebde9397" containerName="kube-rbac-proxy-web" containerID="cri-o://01104db36bf308971798d8562e27c24d014803f40a52576922181a5634f7ad1e" gracePeriod=600
Sep 14 02:55:06.303917 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.303685    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="603095db-c9ac-44a7-b91a-c8f7ebde9397" containerName="config-reloader" containerID="cri-o://53be009b3e5bbced63e16b80179bfef9c84204e00d481429971cc9be6ca7f11d" gracePeriod=600
Sep 14 02:55:06.346887 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.346851    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/1dffd99b-f463-4251-96de-0b8d7f6d5b71-telemeter-client-tls\") pod \"telemeter-client-df686ffff-z4729\" (UID: \"1dffd99b-f463-4251-96de-0b8d7f6d5b71\") " pod="openshift-monitoring/telemeter-client-df686ffff-z4729"
Sep 14 02:55:06.346997 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.346909    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/1dffd99b-f463-4251-96de-0b8d7f6d5b71-federate-client-tls\") pod \"telemeter-client-df686ffff-z4729\" (UID: \"1dffd99b-f463-4251-96de-0b8d7f6d5b71\") " pod="openshift-monitoring/telemeter-client-df686ffff-z4729"
Sep 14 02:55:06.346997 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.346948    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1dffd99b-f463-4251-96de-0b8d7f6d5b71-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-df686ffff-z4729\" (UID: \"1dffd99b-f463-4251-96de-0b8d7f6d5b71\") " pod="openshift-monitoring/telemeter-client-df686ffff-z4729"
Sep 14 02:55:06.346997 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.346974    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcfnh\" (UniqueName: \"kubernetes.io/projected/1dffd99b-f463-4251-96de-0b8d7f6d5b71-kube-api-access-rcfnh\") pod \"telemeter-client-df686ffff-z4729\" (UID: \"1dffd99b-f463-4251-96de-0b8d7f6d5b71\") " pod="openshift-monitoring/telemeter-client-df686ffff-z4729"
Sep 14 02:55:06.347170 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.347003    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1dffd99b-f463-4251-96de-0b8d7f6d5b71-serving-certs-ca-bundle\") pod \"telemeter-client-df686ffff-z4729\" (UID: \"1dffd99b-f463-4251-96de-0b8d7f6d5b71\") " pod="openshift-monitoring/telemeter-client-df686ffff-z4729"
Sep 14 02:55:06.347170 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.347070    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/1dffd99b-f463-4251-96de-0b8d7f6d5b71-secret-telemeter-client\") pod \"telemeter-client-df686ffff-z4729\" (UID: \"1dffd99b-f463-4251-96de-0b8d7f6d5b71\") " pod="openshift-monitoring/telemeter-client-df686ffff-z4729"
Sep 14 02:55:06.347170 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.347097    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1dffd99b-f463-4251-96de-0b8d7f6d5b71-telemeter-trusted-ca-bundle\") pod \"telemeter-client-df686ffff-z4729\" (UID: \"1dffd99b-f463-4251-96de-0b8d7f6d5b71\") " pod="openshift-monitoring/telemeter-client-df686ffff-z4729"
Sep 14 02:55:06.347170 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.347153    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1dffd99b-f463-4251-96de-0b8d7f6d5b71-metrics-client-ca\") pod \"telemeter-client-df686ffff-z4729\" (UID: \"1dffd99b-f463-4251-96de-0b8d7f6d5b71\") " pod="openshift-monitoring/telemeter-client-df686ffff-z4729"
Sep 14 02:55:06.347912 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.347885    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1dffd99b-f463-4251-96de-0b8d7f6d5b71-metrics-client-ca\") pod \"telemeter-client-df686ffff-z4729\" (UID: \"1dffd99b-f463-4251-96de-0b8d7f6d5b71\") " pod="openshift-monitoring/telemeter-client-df686ffff-z4729"
Sep 14 02:55:06.351022 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.350231    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1dffd99b-f463-4251-96de-0b8d7f6d5b71-serving-certs-ca-bundle\") pod \"telemeter-client-df686ffff-z4729\" (UID: \"1dffd99b-f463-4251-96de-0b8d7f6d5b71\") " pod="openshift-monitoring/telemeter-client-df686ffff-z4729"
Sep 14 02:55:06.351022 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.350820    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/1dffd99b-f463-4251-96de-0b8d7f6d5b71-telemeter-client-tls\") pod \"telemeter-client-df686ffff-z4729\" (UID: \"1dffd99b-f463-4251-96de-0b8d7f6d5b71\") " pod="openshift-monitoring/telemeter-client-df686ffff-z4729"
Sep 14 02:55:06.351022 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.350923    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1dffd99b-f463-4251-96de-0b8d7f6d5b71-telemeter-trusted-ca-bundle\") pod \"telemeter-client-df686ffff-z4729\" (UID: \"1dffd99b-f463-4251-96de-0b8d7f6d5b71\") " pod="openshift-monitoring/telemeter-client-df686ffff-z4729"
Sep 14 02:55:06.351589 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.351551    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/1dffd99b-f463-4251-96de-0b8d7f6d5b71-federate-client-tls\") pod \"telemeter-client-df686ffff-z4729\" (UID: \"1dffd99b-f463-4251-96de-0b8d7f6d5b71\") " pod="openshift-monitoring/telemeter-client-df686ffff-z4729"
Sep 14 02:55:06.353254 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.353227    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1dffd99b-f463-4251-96de-0b8d7f6d5b71-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-df686ffff-z4729\" (UID: \"1dffd99b-f463-4251-96de-0b8d7f6d5b71\") " pod="openshift-monitoring/telemeter-client-df686ffff-z4729"
Sep 14 02:55:06.353356 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.353274    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/1dffd99b-f463-4251-96de-0b8d7f6d5b71-secret-telemeter-client\") pod \"telemeter-client-df686ffff-z4729\" (UID: \"1dffd99b-f463-4251-96de-0b8d7f6d5b71\") " pod="openshift-monitoring/telemeter-client-df686ffff-z4729"
Sep 14 02:55:06.358172 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.358126    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcfnh\" (UniqueName: \"kubernetes.io/projected/1dffd99b-f463-4251-96de-0b8d7f6d5b71-kube-api-access-rcfnh\") pod \"telemeter-client-df686ffff-z4729\" (UID: \"1dffd99b-f463-4251-96de-0b8d7f6d5b71\") " pod="openshift-monitoring/telemeter-client-df686ffff-z4729"
Sep 14 02:55:06.365241 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.363848    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-df686ffff-z4729"
Sep 14 02:55:06.564920 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.564879    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:55:06.662201 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.662170    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-df686ffff-z4729"]
Sep 14 02:55:06.663958 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:55:06.663921    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dffd99b_f463_4251_96de_0b8d7f6d5b71.slice/crio-26be3c13b6251c09ded63af593cc5ada6dd01f19d0f5942a59117f0323fd58a9 WatchSource:0}: Error finding container 26be3c13b6251c09ded63af593cc5ada6dd01f19d0f5942a59117f0323fd58a9: Status 404 returned error can't find the container with id 26be3c13b6251c09ded63af593cc5ada6dd01f19d0f5942a59117f0323fd58a9
Sep 14 02:55:06.752358 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.752325    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/603095db-c9ac-44a7-b91a-c8f7ebde9397-configmap-metrics-client-ca\") pod \"603095db-c9ac-44a7-b91a-c8f7ebde9397\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") "
Sep 14 02:55:06.752358 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.752359    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/603095db-c9ac-44a7-b91a-c8f7ebde9397-web-config\") pod \"603095db-c9ac-44a7-b91a-c8f7ebde9397\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") "
Sep 14 02:55:06.752609 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.752395    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/603095db-c9ac-44a7-b91a-c8f7ebde9397-configmap-kubelet-serving-ca-bundle\") pod \"603095db-c9ac-44a7-b91a-c8f7ebde9397\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") "
Sep 14 02:55:06.752609 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.752434    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/603095db-c9ac-44a7-b91a-c8f7ebde9397-secret-kube-rbac-proxy\") pod \"603095db-c9ac-44a7-b91a-c8f7ebde9397\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") "
Sep 14 02:55:06.752609 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.752471    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/603095db-c9ac-44a7-b91a-c8f7ebde9397-tls-assets\") pod \"603095db-c9ac-44a7-b91a-c8f7ebde9397\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") "
Sep 14 02:55:06.752609 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.752490    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/603095db-c9ac-44a7-b91a-c8f7ebde9397-prometheus-k8s-db\") pod \"603095db-c9ac-44a7-b91a-c8f7ebde9397\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") "
Sep 14 02:55:06.752609 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.752504    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/603095db-c9ac-44a7-b91a-c8f7ebde9397-thanos-prometheus-http-client-file\") pod \"603095db-c9ac-44a7-b91a-c8f7ebde9397\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") "
Sep 14 02:55:06.752609 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.752533    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/603095db-c9ac-44a7-b91a-c8f7ebde9397-prometheus-k8s-rulefiles-0\") pod \"603095db-c9ac-44a7-b91a-c8f7ebde9397\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") "
Sep 14 02:55:06.752609 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.752569    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/603095db-c9ac-44a7-b91a-c8f7ebde9397-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"603095db-c9ac-44a7-b91a-c8f7ebde9397\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") "
Sep 14 02:55:06.752944 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.752624    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/603095db-c9ac-44a7-b91a-c8f7ebde9397-prometheus-k8s-rulefiles-1\") pod \"603095db-c9ac-44a7-b91a-c8f7ebde9397\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") "
Sep 14 02:55:06.752944 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.752663    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zbk6c\" (UniqueName: \"kubernetes.io/projected/603095db-c9ac-44a7-b91a-c8f7ebde9397-kube-api-access-zbk6c\") pod \"603095db-c9ac-44a7-b91a-c8f7ebde9397\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") "
Sep 14 02:55:06.752944 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.752690    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/603095db-c9ac-44a7-b91a-c8f7ebde9397-config\") pod \"603095db-c9ac-44a7-b91a-c8f7ebde9397\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") "
Sep 14 02:55:06.752944 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.752717    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/603095db-c9ac-44a7-b91a-c8f7ebde9397-configmap-serving-certs-ca-bundle\") pod \"603095db-c9ac-44a7-b91a-c8f7ebde9397\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") "
Sep 14 02:55:06.752944 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.752742    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/603095db-c9ac-44a7-b91a-c8f7ebde9397-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"603095db-c9ac-44a7-b91a-c8f7ebde9397\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") "
Sep 14 02:55:06.752944 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.752745    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/603095db-c9ac-44a7-b91a-c8f7ebde9397-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "603095db-c9ac-44a7-b91a-c8f7ebde9397" (UID: "603095db-c9ac-44a7-b91a-c8f7ebde9397"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 02:55:06.752944 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.752765    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/603095db-c9ac-44a7-b91a-c8f7ebde9397-secret-prometheus-k8s-tls\") pod \"603095db-c9ac-44a7-b91a-c8f7ebde9397\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") "
Sep 14 02:55:06.752944 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.752784    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/603095db-c9ac-44a7-b91a-c8f7ebde9397-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "603095db-c9ac-44a7-b91a-c8f7ebde9397" (UID: "603095db-c9ac-44a7-b91a-c8f7ebde9397"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 02:55:06.752944 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.752822    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/603095db-c9ac-44a7-b91a-c8f7ebde9397-secret-metrics-client-certs\") pod \"603095db-c9ac-44a7-b91a-c8f7ebde9397\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") "
Sep 14 02:55:06.752944 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.752852    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/603095db-c9ac-44a7-b91a-c8f7ebde9397-prometheus-k8s-rulefiles-2\") pod \"603095db-c9ac-44a7-b91a-c8f7ebde9397\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") "
Sep 14 02:55:06.752944 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.752887    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/603095db-c9ac-44a7-b91a-c8f7ebde9397-secret-grpc-tls\") pod \"603095db-c9ac-44a7-b91a-c8f7ebde9397\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") "
Sep 14 02:55:06.752944 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.752921    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/603095db-c9ac-44a7-b91a-c8f7ebde9397-config-out\") pod \"603095db-c9ac-44a7-b91a-c8f7ebde9397\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") "
Sep 14 02:55:06.753541 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.752964    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/603095db-c9ac-44a7-b91a-c8f7ebde9397-prometheus-trusted-ca-bundle\") pod \"603095db-c9ac-44a7-b91a-c8f7ebde9397\" (UID: \"603095db-c9ac-44a7-b91a-c8f7ebde9397\") "
Sep 14 02:55:06.753541 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.753012    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/603095db-c9ac-44a7-b91a-c8f7ebde9397-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "603095db-c9ac-44a7-b91a-c8f7ebde9397" (UID: "603095db-c9ac-44a7-b91a-c8f7ebde9397"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 02:55:06.753541 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.753257    2564 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/603095db-c9ac-44a7-b91a-c8f7ebde9397-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 02:55:06.753541 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.753278    2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/603095db-c9ac-44a7-b91a-c8f7ebde9397-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 02:55:06.753541 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.753294    2564 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/603095db-c9ac-44a7-b91a-c8f7ebde9397-configmap-metrics-client-ca\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 02:55:06.753797 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.753634    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/603095db-c9ac-44a7-b91a-c8f7ebde9397-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "603095db-c9ac-44a7-b91a-c8f7ebde9397" (UID: "603095db-c9ac-44a7-b91a-c8f7ebde9397"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 02:55:06.753797 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.753645    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/603095db-c9ac-44a7-b91a-c8f7ebde9397-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "603095db-c9ac-44a7-b91a-c8f7ebde9397" (UID: "603095db-c9ac-44a7-b91a-c8f7ebde9397"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 02:55:06.755029 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.754733    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/603095db-c9ac-44a7-b91a-c8f7ebde9397-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "603095db-c9ac-44a7-b91a-c8f7ebde9397" (UID: "603095db-c9ac-44a7-b91a-c8f7ebde9397"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 02:55:06.755029 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.755006    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/603095db-c9ac-44a7-b91a-c8f7ebde9397-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "603095db-c9ac-44a7-b91a-c8f7ebde9397" (UID: "603095db-c9ac-44a7-b91a-c8f7ebde9397"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 02:55:06.755771 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.755129    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/603095db-c9ac-44a7-b91a-c8f7ebde9397-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "603095db-c9ac-44a7-b91a-c8f7ebde9397" (UID: "603095db-c9ac-44a7-b91a-c8f7ebde9397"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 02:55:06.756015 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.755971    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/603095db-c9ac-44a7-b91a-c8f7ebde9397-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "603095db-c9ac-44a7-b91a-c8f7ebde9397" (UID: "603095db-c9ac-44a7-b91a-c8f7ebde9397"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 02:55:06.756221 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.756139    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/603095db-c9ac-44a7-b91a-c8f7ebde9397-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "603095db-c9ac-44a7-b91a-c8f7ebde9397" (UID: "603095db-c9ac-44a7-b91a-c8f7ebde9397"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 02:55:06.756221 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.756181    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/603095db-c9ac-44a7-b91a-c8f7ebde9397-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "603095db-c9ac-44a7-b91a-c8f7ebde9397" (UID: "603095db-c9ac-44a7-b91a-c8f7ebde9397"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 02:55:06.756447 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.756393    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/603095db-c9ac-44a7-b91a-c8f7ebde9397-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "603095db-c9ac-44a7-b91a-c8f7ebde9397" (UID: "603095db-c9ac-44a7-b91a-c8f7ebde9397"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 02:55:06.756684 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.756655    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/603095db-c9ac-44a7-b91a-c8f7ebde9397-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "603095db-c9ac-44a7-b91a-c8f7ebde9397" (UID: "603095db-c9ac-44a7-b91a-c8f7ebde9397"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 02:55:06.757067 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.757045    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/603095db-c9ac-44a7-b91a-c8f7ebde9397-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "603095db-c9ac-44a7-b91a-c8f7ebde9397" (UID: "603095db-c9ac-44a7-b91a-c8f7ebde9397"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 02:55:06.757441 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.757386    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/603095db-c9ac-44a7-b91a-c8f7ebde9397-config-out" (OuterVolumeSpecName: "config-out") pod "603095db-c9ac-44a7-b91a-c8f7ebde9397" (UID: "603095db-c9ac-44a7-b91a-c8f7ebde9397"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 02:55:06.757534 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.757507    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/603095db-c9ac-44a7-b91a-c8f7ebde9397-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "603095db-c9ac-44a7-b91a-c8f7ebde9397" (UID: "603095db-c9ac-44a7-b91a-c8f7ebde9397"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 02:55:06.758124 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.758099    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/603095db-c9ac-44a7-b91a-c8f7ebde9397-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "603095db-c9ac-44a7-b91a-c8f7ebde9397" (UID: "603095db-c9ac-44a7-b91a-c8f7ebde9397"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 02:55:06.758174 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.758123    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/603095db-c9ac-44a7-b91a-c8f7ebde9397-config" (OuterVolumeSpecName: "config") pod "603095db-c9ac-44a7-b91a-c8f7ebde9397" (UID: "603095db-c9ac-44a7-b91a-c8f7ebde9397"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 02:55:06.758388 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.758371    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/603095db-c9ac-44a7-b91a-c8f7ebde9397-kube-api-access-zbk6c" (OuterVolumeSpecName: "kube-api-access-zbk6c") pod "603095db-c9ac-44a7-b91a-c8f7ebde9397" (UID: "603095db-c9ac-44a7-b91a-c8f7ebde9397"). InnerVolumeSpecName "kube-api-access-zbk6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 02:55:06.766170 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.766151    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/603095db-c9ac-44a7-b91a-c8f7ebde9397-web-config" (OuterVolumeSpecName: "web-config") pod "603095db-c9ac-44a7-b91a-c8f7ebde9397" (UID: "603095db-c9ac-44a7-b91a-c8f7ebde9397"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 02:55:06.854730 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.854654    2564 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/603095db-c9ac-44a7-b91a-c8f7ebde9397-web-config\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 02:55:06.854730 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.854686    2564 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/603095db-c9ac-44a7-b91a-c8f7ebde9397-secret-kube-rbac-proxy\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 02:55:06.854730 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.854695    2564 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/603095db-c9ac-44a7-b91a-c8f7ebde9397-tls-assets\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 02:55:06.854730 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.854705    2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/603095db-c9ac-44a7-b91a-c8f7ebde9397-prometheus-k8s-db\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 02:55:06.854730 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.854715    2564 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/603095db-c9ac-44a7-b91a-c8f7ebde9397-thanos-prometheus-http-client-file\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 02:55:06.854730 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.854725    2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/603095db-c9ac-44a7-b91a-c8f7ebde9397-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 02:55:06.854730 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.854734    2564 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/603095db-c9ac-44a7-b91a-c8f7ebde9397-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 02:55:06.855035 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.854743    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbk6c\" (UniqueName: \"kubernetes.io/projected/603095db-c9ac-44a7-b91a-c8f7ebde9397-kube-api-access-zbk6c\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 02:55:06.855035 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.854751    2564 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/603095db-c9ac-44a7-b91a-c8f7ebde9397-config\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 02:55:06.855035 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.854760    2564 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/603095db-c9ac-44a7-b91a-c8f7ebde9397-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 02:55:06.855035 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.854772    2564 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/603095db-c9ac-44a7-b91a-c8f7ebde9397-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 02:55:06.855035 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.854781    2564 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/603095db-c9ac-44a7-b91a-c8f7ebde9397-secret-prometheus-k8s-tls\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 02:55:06.855035 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.854788    2564 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/603095db-c9ac-44a7-b91a-c8f7ebde9397-secret-metrics-client-certs\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 02:55:06.855035 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.854797    2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/603095db-c9ac-44a7-b91a-c8f7ebde9397-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 02:55:06.855035 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.854805    2564 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/603095db-c9ac-44a7-b91a-c8f7ebde9397-secret-grpc-tls\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 02:55:06.855035 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.854813    2564 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/603095db-c9ac-44a7-b91a-c8f7ebde9397-config-out\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 02:55:06.855035 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.854820    2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/603095db-c9ac-44a7-b91a-c8f7ebde9397-prometheus-trusted-ca-bundle\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 02:55:06.865217 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.865189    2564 generic.go:356] "Generic (PLEG): container finished" podID="603095db-c9ac-44a7-b91a-c8f7ebde9397" containerID="64ee82c7d3547678c022e1726d07c12e9ff159cedfedd39f914cefd522fad67f" exitCode=0
Sep 14 02:55:06.865217 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.865212    2564 generic.go:356] "Generic (PLEG): container finished" podID="603095db-c9ac-44a7-b91a-c8f7ebde9397" containerID="34d5b4c42bd7f2ed7ddd63b21861378c290a610fe6fd5186632daef4db9edd81" exitCode=0
Sep 14 02:55:06.865217 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.865220    2564 generic.go:356] "Generic (PLEG): container finished" podID="603095db-c9ac-44a7-b91a-c8f7ebde9397" containerID="01104db36bf308971798d8562e27c24d014803f40a52576922181a5634f7ad1e" exitCode=0
Sep 14 02:55:06.865217 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.865225    2564 generic.go:356] "Generic (PLEG): container finished" podID="603095db-c9ac-44a7-b91a-c8f7ebde9397" containerID="fdff46c1b857b5255645de42ef2c7c4e76db2a945f92d70903a18d7fdfb86187" exitCode=0
Sep 14 02:55:06.865824 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.865230    2564 generic.go:356] "Generic (PLEG): container finished" podID="603095db-c9ac-44a7-b91a-c8f7ebde9397" containerID="53be009b3e5bbced63e16b80179bfef9c84204e00d481429971cc9be6ca7f11d" exitCode=0
Sep 14 02:55:06.865824 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.865235    2564 generic.go:356] "Generic (PLEG): container finished" podID="603095db-c9ac-44a7-b91a-c8f7ebde9397" containerID="a29a0eb7517dcc7c19389cb1cd97fb7a0debd5e5672f0383296fb7255a5531f6" exitCode=0
Sep 14 02:55:06.865824 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.865273    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:55:06.865824 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.865281    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"603095db-c9ac-44a7-b91a-c8f7ebde9397","Type":"ContainerDied","Data":"64ee82c7d3547678c022e1726d07c12e9ff159cedfedd39f914cefd522fad67f"}
Sep 14 02:55:06.865824 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.865449    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"603095db-c9ac-44a7-b91a-c8f7ebde9397","Type":"ContainerDied","Data":"34d5b4c42bd7f2ed7ddd63b21861378c290a610fe6fd5186632daef4db9edd81"}
Sep 14 02:55:06.865824 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.865499    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"603095db-c9ac-44a7-b91a-c8f7ebde9397","Type":"ContainerDied","Data":"01104db36bf308971798d8562e27c24d014803f40a52576922181a5634f7ad1e"}
Sep 14 02:55:06.865824 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.865513    2564 scope.go:117] "RemoveContainer" containerID="64ee82c7d3547678c022e1726d07c12e9ff159cedfedd39f914cefd522fad67f"
Sep 14 02:55:06.865824 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.865543    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"603095db-c9ac-44a7-b91a-c8f7ebde9397","Type":"ContainerDied","Data":"fdff46c1b857b5255645de42ef2c7c4e76db2a945f92d70903a18d7fdfb86187"}
Sep 14 02:55:06.865824 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.865586    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"603095db-c9ac-44a7-b91a-c8f7ebde9397","Type":"ContainerDied","Data":"53be009b3e5bbced63e16b80179bfef9c84204e00d481429971cc9be6ca7f11d"}
Sep 14 02:55:06.865824 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.865624    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"603095db-c9ac-44a7-b91a-c8f7ebde9397","Type":"ContainerDied","Data":"a29a0eb7517dcc7c19389cb1cd97fb7a0debd5e5672f0383296fb7255a5531f6"}
Sep 14 02:55:06.865824 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.865660    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"603095db-c9ac-44a7-b91a-c8f7ebde9397","Type":"ContainerDied","Data":"6229d8ba8797f5c0a945e5cc423fb3f639e9f652200fc2875650cc251aa6f1fe"}
Sep 14 02:55:06.866791 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.866769    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-df686ffff-z4729" event={"ID":"1dffd99b-f463-4251-96de-0b8d7f6d5b71","Type":"ContainerStarted","Data":"26be3c13b6251c09ded63af593cc5ada6dd01f19d0f5942a59117f0323fd58a9"}
Sep 14 02:55:06.874098 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.874081    2564 scope.go:117] "RemoveContainer" containerID="34d5b4c42bd7f2ed7ddd63b21861378c290a610fe6fd5186632daef4db9edd81"
Sep 14 02:55:06.880931 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.880914    2564 scope.go:117] "RemoveContainer" containerID="01104db36bf308971798d8562e27c24d014803f40a52576922181a5634f7ad1e"
Sep 14 02:55:06.887709 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.887690    2564 scope.go:117] "RemoveContainer" containerID="fdff46c1b857b5255645de42ef2c7c4e76db2a945f92d70903a18d7fdfb86187"
Sep 14 02:55:06.894203 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.894187    2564 scope.go:117] "RemoveContainer" containerID="53be009b3e5bbced63e16b80179bfef9c84204e00d481429971cc9be6ca7f11d"
Sep 14 02:55:06.899711 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.899690    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 14 02:55:06.901847 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.901827    2564 scope.go:117] "RemoveContainer" containerID="a29a0eb7517dcc7c19389cb1cd97fb7a0debd5e5672f0383296fb7255a5531f6"
Sep 14 02:55:06.907096 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.907069    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 14 02:55:06.909595 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.909581    2564 scope.go:117] "RemoveContainer" containerID="35879a0a0769cfc374379b4d93cdab1b2d9bc1d95d51317d090cce1fda427d38"
Sep 14 02:55:06.916071 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.916056    2564 scope.go:117] "RemoveContainer" containerID="64ee82c7d3547678c022e1726d07c12e9ff159cedfedd39f914cefd522fad67f"
Sep 14 02:55:06.916324 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:55:06.916303    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64ee82c7d3547678c022e1726d07c12e9ff159cedfedd39f914cefd522fad67f\": container with ID starting with 64ee82c7d3547678c022e1726d07c12e9ff159cedfedd39f914cefd522fad67f not found: ID does not exist" containerID="64ee82c7d3547678c022e1726d07c12e9ff159cedfedd39f914cefd522fad67f"
Sep 14 02:55:06.916388 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.916331    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64ee82c7d3547678c022e1726d07c12e9ff159cedfedd39f914cefd522fad67f"} err="failed to get container status \"64ee82c7d3547678c022e1726d07c12e9ff159cedfedd39f914cefd522fad67f\": rpc error: code = NotFound desc = could not find container \"64ee82c7d3547678c022e1726d07c12e9ff159cedfedd39f914cefd522fad67f\": container with ID starting with 64ee82c7d3547678c022e1726d07c12e9ff159cedfedd39f914cefd522fad67f not found: ID does not exist"
Sep 14 02:55:06.916388 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.916348    2564 scope.go:117] "RemoveContainer" containerID="34d5b4c42bd7f2ed7ddd63b21861378c290a610fe6fd5186632daef4db9edd81"
Sep 14 02:55:06.916639 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:55:06.916621    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34d5b4c42bd7f2ed7ddd63b21861378c290a610fe6fd5186632daef4db9edd81\": container with ID starting with 34d5b4c42bd7f2ed7ddd63b21861378c290a610fe6fd5186632daef4db9edd81 not found: ID does not exist" containerID="34d5b4c42bd7f2ed7ddd63b21861378c290a610fe6fd5186632daef4db9edd81"
Sep 14 02:55:06.916680 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.916647    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34d5b4c42bd7f2ed7ddd63b21861378c290a610fe6fd5186632daef4db9edd81"} err="failed to get container status \"34d5b4c42bd7f2ed7ddd63b21861378c290a610fe6fd5186632daef4db9edd81\": rpc error: code = NotFound desc = could not find container \"34d5b4c42bd7f2ed7ddd63b21861378c290a610fe6fd5186632daef4db9edd81\": container with ID starting with 34d5b4c42bd7f2ed7ddd63b21861378c290a610fe6fd5186632daef4db9edd81 not found: ID does not exist"
Sep 14 02:55:06.916680 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.916673    2564 scope.go:117] "RemoveContainer" containerID="01104db36bf308971798d8562e27c24d014803f40a52576922181a5634f7ad1e"
Sep 14 02:55:06.916945 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:55:06.916925    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01104db36bf308971798d8562e27c24d014803f40a52576922181a5634f7ad1e\": container with ID starting with 01104db36bf308971798d8562e27c24d014803f40a52576922181a5634f7ad1e not found: ID does not exist" containerID="01104db36bf308971798d8562e27c24d014803f40a52576922181a5634f7ad1e"
Sep 14 02:55:06.916985 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.916951    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01104db36bf308971798d8562e27c24d014803f40a52576922181a5634f7ad1e"} err="failed to get container status \"01104db36bf308971798d8562e27c24d014803f40a52576922181a5634f7ad1e\": rpc error: code = NotFound desc = could not find container \"01104db36bf308971798d8562e27c24d014803f40a52576922181a5634f7ad1e\": container with ID starting with 01104db36bf308971798d8562e27c24d014803f40a52576922181a5634f7ad1e not found: ID does not exist"
Sep 14 02:55:06.916985 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.916966    2564 scope.go:117] "RemoveContainer" containerID="fdff46c1b857b5255645de42ef2c7c4e76db2a945f92d70903a18d7fdfb86187"
Sep 14 02:55:06.917209 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:55:06.917192    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdff46c1b857b5255645de42ef2c7c4e76db2a945f92d70903a18d7fdfb86187\": container with ID starting with fdff46c1b857b5255645de42ef2c7c4e76db2a945f92d70903a18d7fdfb86187 not found: ID does not exist" containerID="fdff46c1b857b5255645de42ef2c7c4e76db2a945f92d70903a18d7fdfb86187"
Sep 14 02:55:06.917248 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.917216    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdff46c1b857b5255645de42ef2c7c4e76db2a945f92d70903a18d7fdfb86187"} err="failed to get container status \"fdff46c1b857b5255645de42ef2c7c4e76db2a945f92d70903a18d7fdfb86187\": rpc error: code = NotFound desc = could not find container \"fdff46c1b857b5255645de42ef2c7c4e76db2a945f92d70903a18d7fdfb86187\": container with ID starting with fdff46c1b857b5255645de42ef2c7c4e76db2a945f92d70903a18d7fdfb86187 not found: ID does not exist"
Sep 14 02:55:06.917248 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.917238    2564 scope.go:117] "RemoveContainer" containerID="53be009b3e5bbced63e16b80179bfef9c84204e00d481429971cc9be6ca7f11d"
Sep 14 02:55:06.917489 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:55:06.917472    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53be009b3e5bbced63e16b80179bfef9c84204e00d481429971cc9be6ca7f11d\": container with ID starting with 53be009b3e5bbced63e16b80179bfef9c84204e00d481429971cc9be6ca7f11d not found: ID does not exist" containerID="53be009b3e5bbced63e16b80179bfef9c84204e00d481429971cc9be6ca7f11d"
Sep 14 02:55:06.917527 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.917494    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53be009b3e5bbced63e16b80179bfef9c84204e00d481429971cc9be6ca7f11d"} err="failed to get container status \"53be009b3e5bbced63e16b80179bfef9c84204e00d481429971cc9be6ca7f11d\": rpc error: code = NotFound desc = could not find container \"53be009b3e5bbced63e16b80179bfef9c84204e00d481429971cc9be6ca7f11d\": container with ID starting with 53be009b3e5bbced63e16b80179bfef9c84204e00d481429971cc9be6ca7f11d not found: ID does not exist"
Sep 14 02:55:06.917527 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.917511    2564 scope.go:117] "RemoveContainer" containerID="a29a0eb7517dcc7c19389cb1cd97fb7a0debd5e5672f0383296fb7255a5531f6"
Sep 14 02:55:06.917741 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:55:06.917718    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a29a0eb7517dcc7c19389cb1cd97fb7a0debd5e5672f0383296fb7255a5531f6\": container with ID starting with a29a0eb7517dcc7c19389cb1cd97fb7a0debd5e5672f0383296fb7255a5531f6 not found: ID does not exist" containerID="a29a0eb7517dcc7c19389cb1cd97fb7a0debd5e5672f0383296fb7255a5531f6"
Sep 14 02:55:06.917786 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.917744    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a29a0eb7517dcc7c19389cb1cd97fb7a0debd5e5672f0383296fb7255a5531f6"} err="failed to get container status \"a29a0eb7517dcc7c19389cb1cd97fb7a0debd5e5672f0383296fb7255a5531f6\": rpc error: code = NotFound desc = could not find container \"a29a0eb7517dcc7c19389cb1cd97fb7a0debd5e5672f0383296fb7255a5531f6\": container with ID starting with a29a0eb7517dcc7c19389cb1cd97fb7a0debd5e5672f0383296fb7255a5531f6 not found: ID does not exist"
Sep 14 02:55:06.917786 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.917761    2564 scope.go:117] "RemoveContainer" containerID="35879a0a0769cfc374379b4d93cdab1b2d9bc1d95d51317d090cce1fda427d38"
Sep 14 02:55:06.918007 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:55:06.917991    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35879a0a0769cfc374379b4d93cdab1b2d9bc1d95d51317d090cce1fda427d38\": container with ID starting with 35879a0a0769cfc374379b4d93cdab1b2d9bc1d95d51317d090cce1fda427d38 not found: ID does not exist" containerID="35879a0a0769cfc374379b4d93cdab1b2d9bc1d95d51317d090cce1fda427d38"
Sep 14 02:55:06.918045 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.918021    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35879a0a0769cfc374379b4d93cdab1b2d9bc1d95d51317d090cce1fda427d38"} err="failed to get container status \"35879a0a0769cfc374379b4d93cdab1b2d9bc1d95d51317d090cce1fda427d38\": rpc error: code = NotFound desc = could not find container \"35879a0a0769cfc374379b4d93cdab1b2d9bc1d95d51317d090cce1fda427d38\": container with ID starting with 35879a0a0769cfc374379b4d93cdab1b2d9bc1d95d51317d090cce1fda427d38 not found: ID does not exist"
Sep 14 02:55:06.918045 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.918035    2564 scope.go:117] "RemoveContainer" containerID="64ee82c7d3547678c022e1726d07c12e9ff159cedfedd39f914cefd522fad67f"
Sep 14 02:55:06.918285 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.918266    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64ee82c7d3547678c022e1726d07c12e9ff159cedfedd39f914cefd522fad67f"} err="failed to get container status \"64ee82c7d3547678c022e1726d07c12e9ff159cedfedd39f914cefd522fad67f\": rpc error: code = NotFound desc = could not find container \"64ee82c7d3547678c022e1726d07c12e9ff159cedfedd39f914cefd522fad67f\": container with ID starting with 64ee82c7d3547678c022e1726d07c12e9ff159cedfedd39f914cefd522fad67f not found: ID does not exist"
Sep 14 02:55:06.918285 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.918284    2564 scope.go:117] "RemoveContainer" containerID="34d5b4c42bd7f2ed7ddd63b21861378c290a610fe6fd5186632daef4db9edd81"
Sep 14 02:55:06.918529 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.918512    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34d5b4c42bd7f2ed7ddd63b21861378c290a610fe6fd5186632daef4db9edd81"} err="failed to get container status \"34d5b4c42bd7f2ed7ddd63b21861378c290a610fe6fd5186632daef4db9edd81\": rpc error: code = NotFound desc = could not find container \"34d5b4c42bd7f2ed7ddd63b21861378c290a610fe6fd5186632daef4db9edd81\": container with ID starting with 34d5b4c42bd7f2ed7ddd63b21861378c290a610fe6fd5186632daef4db9edd81 not found: ID does not exist"
Sep 14 02:55:06.918569 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.918528    2564 scope.go:117] "RemoveContainer" containerID="01104db36bf308971798d8562e27c24d014803f40a52576922181a5634f7ad1e"
Sep 14 02:55:06.918754 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.918738    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01104db36bf308971798d8562e27c24d014803f40a52576922181a5634f7ad1e"} err="failed to get container status \"01104db36bf308971798d8562e27c24d014803f40a52576922181a5634f7ad1e\": rpc error: code = NotFound desc = could not find container \"01104db36bf308971798d8562e27c24d014803f40a52576922181a5634f7ad1e\": container with ID starting with 01104db36bf308971798d8562e27c24d014803f40a52576922181a5634f7ad1e not found: ID does not exist"
Sep 14 02:55:06.918796 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.918755    2564 scope.go:117] "RemoveContainer" containerID="fdff46c1b857b5255645de42ef2c7c4e76db2a945f92d70903a18d7fdfb86187"
Sep 14 02:55:06.918975 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.918955    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdff46c1b857b5255645de42ef2c7c4e76db2a945f92d70903a18d7fdfb86187"} err="failed to get container status \"fdff46c1b857b5255645de42ef2c7c4e76db2a945f92d70903a18d7fdfb86187\": rpc error: code = NotFound desc = could not find container \"fdff46c1b857b5255645de42ef2c7c4e76db2a945f92d70903a18d7fdfb86187\": container with ID starting with fdff46c1b857b5255645de42ef2c7c4e76db2a945f92d70903a18d7fdfb86187 not found: ID does not exist"
Sep 14 02:55:06.919020 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.918975    2564 scope.go:117] "RemoveContainer" containerID="53be009b3e5bbced63e16b80179bfef9c84204e00d481429971cc9be6ca7f11d"
Sep 14 02:55:06.919198 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.919182    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53be009b3e5bbced63e16b80179bfef9c84204e00d481429971cc9be6ca7f11d"} err="failed to get container status \"53be009b3e5bbced63e16b80179bfef9c84204e00d481429971cc9be6ca7f11d\": rpc error: code = NotFound desc = could not find container \"53be009b3e5bbced63e16b80179bfef9c84204e00d481429971cc9be6ca7f11d\": container with ID starting with 53be009b3e5bbced63e16b80179bfef9c84204e00d481429971cc9be6ca7f11d not found: ID does not exist"
Sep 14 02:55:06.919239 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.919198    2564 scope.go:117] "RemoveContainer" containerID="a29a0eb7517dcc7c19389cb1cd97fb7a0debd5e5672f0383296fb7255a5531f6"
Sep 14 02:55:06.919415 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.919389    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a29a0eb7517dcc7c19389cb1cd97fb7a0debd5e5672f0383296fb7255a5531f6"} err="failed to get container status \"a29a0eb7517dcc7c19389cb1cd97fb7a0debd5e5672f0383296fb7255a5531f6\": rpc error: code = NotFound desc = could not find container \"a29a0eb7517dcc7c19389cb1cd97fb7a0debd5e5672f0383296fb7255a5531f6\": container with ID starting with a29a0eb7517dcc7c19389cb1cd97fb7a0debd5e5672f0383296fb7255a5531f6 not found: ID does not exist"
Sep 14 02:55:06.919475 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.919427    2564 scope.go:117] "RemoveContainer" containerID="35879a0a0769cfc374379b4d93cdab1b2d9bc1d95d51317d090cce1fda427d38"
Sep 14 02:55:06.919640 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.919623    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35879a0a0769cfc374379b4d93cdab1b2d9bc1d95d51317d090cce1fda427d38"} err="failed to get container status \"35879a0a0769cfc374379b4d93cdab1b2d9bc1d95d51317d090cce1fda427d38\": rpc error: code = NotFound desc = could not find container \"35879a0a0769cfc374379b4d93cdab1b2d9bc1d95d51317d090cce1fda427d38\": container with ID starting with 35879a0a0769cfc374379b4d93cdab1b2d9bc1d95d51317d090cce1fda427d38 not found: ID does not exist"
Sep 14 02:55:06.919680 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.919641    2564 scope.go:117] "RemoveContainer" containerID="64ee82c7d3547678c022e1726d07c12e9ff159cedfedd39f914cefd522fad67f"
Sep 14 02:55:06.919853 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.919837    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64ee82c7d3547678c022e1726d07c12e9ff159cedfedd39f914cefd522fad67f"} err="failed to get container status \"64ee82c7d3547678c022e1726d07c12e9ff159cedfedd39f914cefd522fad67f\": rpc error: code = NotFound desc = could not find container \"64ee82c7d3547678c022e1726d07c12e9ff159cedfedd39f914cefd522fad67f\": container with ID starting with 64ee82c7d3547678c022e1726d07c12e9ff159cedfedd39f914cefd522fad67f not found: ID does not exist"
Sep 14 02:55:06.919899 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.919852    2564 scope.go:117] "RemoveContainer" containerID="34d5b4c42bd7f2ed7ddd63b21861378c290a610fe6fd5186632daef4db9edd81"
Sep 14 02:55:06.920042 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.920027    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34d5b4c42bd7f2ed7ddd63b21861378c290a610fe6fd5186632daef4db9edd81"} err="failed to get container status \"34d5b4c42bd7f2ed7ddd63b21861378c290a610fe6fd5186632daef4db9edd81\": rpc error: code = NotFound desc = could not find container \"34d5b4c42bd7f2ed7ddd63b21861378c290a610fe6fd5186632daef4db9edd81\": container with ID starting with 34d5b4c42bd7f2ed7ddd63b21861378c290a610fe6fd5186632daef4db9edd81 not found: ID does not exist"
Sep 14 02:55:06.920092 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.920043    2564 scope.go:117] "RemoveContainer" containerID="01104db36bf308971798d8562e27c24d014803f40a52576922181a5634f7ad1e"
Sep 14 02:55:06.920251 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.920234    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01104db36bf308971798d8562e27c24d014803f40a52576922181a5634f7ad1e"} err="failed to get container status \"01104db36bf308971798d8562e27c24d014803f40a52576922181a5634f7ad1e\": rpc error: code = NotFound desc = could not find container \"01104db36bf308971798d8562e27c24d014803f40a52576922181a5634f7ad1e\": container with ID starting with 01104db36bf308971798d8562e27c24d014803f40a52576922181a5634f7ad1e not found: ID does not exist"
Sep 14 02:55:06.920251 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.920250    2564 scope.go:117] "RemoveContainer" containerID="fdff46c1b857b5255645de42ef2c7c4e76db2a945f92d70903a18d7fdfb86187"
Sep 14 02:55:06.920440 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.920423    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdff46c1b857b5255645de42ef2c7c4e76db2a945f92d70903a18d7fdfb86187"} err="failed to get container status \"fdff46c1b857b5255645de42ef2c7c4e76db2a945f92d70903a18d7fdfb86187\": rpc error: code = NotFound desc = could not find container \"fdff46c1b857b5255645de42ef2c7c4e76db2a945f92d70903a18d7fdfb86187\": container with ID starting with fdff46c1b857b5255645de42ef2c7c4e76db2a945f92d70903a18d7fdfb86187 not found: ID does not exist"
Sep 14 02:55:06.920485 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.920441    2564 scope.go:117] "RemoveContainer" containerID="53be009b3e5bbced63e16b80179bfef9c84204e00d481429971cc9be6ca7f11d"
Sep 14 02:55:06.920658 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.920639    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53be009b3e5bbced63e16b80179bfef9c84204e00d481429971cc9be6ca7f11d"} err="failed to get container status \"53be009b3e5bbced63e16b80179bfef9c84204e00d481429971cc9be6ca7f11d\": rpc error: code = NotFound desc = could not find container \"53be009b3e5bbced63e16b80179bfef9c84204e00d481429971cc9be6ca7f11d\": container with ID starting with 53be009b3e5bbced63e16b80179bfef9c84204e00d481429971cc9be6ca7f11d not found: ID does not exist"
Sep 14 02:55:06.920701 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.920659    2564 scope.go:117] "RemoveContainer" containerID="a29a0eb7517dcc7c19389cb1cd97fb7a0debd5e5672f0383296fb7255a5531f6"
Sep 14 02:55:06.920871 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.920854    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a29a0eb7517dcc7c19389cb1cd97fb7a0debd5e5672f0383296fb7255a5531f6"} err="failed to get container status \"a29a0eb7517dcc7c19389cb1cd97fb7a0debd5e5672f0383296fb7255a5531f6\": rpc error: code = NotFound desc = could not find container \"a29a0eb7517dcc7c19389cb1cd97fb7a0debd5e5672f0383296fb7255a5531f6\": container with ID starting with a29a0eb7517dcc7c19389cb1cd97fb7a0debd5e5672f0383296fb7255a5531f6 not found: ID does not exist"
Sep 14 02:55:06.920871 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.920871    2564 scope.go:117] "RemoveContainer" containerID="35879a0a0769cfc374379b4d93cdab1b2d9bc1d95d51317d090cce1fda427d38"
Sep 14 02:55:06.921074 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.921059    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35879a0a0769cfc374379b4d93cdab1b2d9bc1d95d51317d090cce1fda427d38"} err="failed to get container status \"35879a0a0769cfc374379b4d93cdab1b2d9bc1d95d51317d090cce1fda427d38\": rpc error: code = NotFound desc = could not find container \"35879a0a0769cfc374379b4d93cdab1b2d9bc1d95d51317d090cce1fda427d38\": container with ID starting with 35879a0a0769cfc374379b4d93cdab1b2d9bc1d95d51317d090cce1fda427d38 not found: ID does not exist"
Sep 14 02:55:06.921119 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.921074    2564 scope.go:117] "RemoveContainer" containerID="64ee82c7d3547678c022e1726d07c12e9ff159cedfedd39f914cefd522fad67f"
Sep 14 02:55:06.921282 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.921264    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64ee82c7d3547678c022e1726d07c12e9ff159cedfedd39f914cefd522fad67f"} err="failed to get container status \"64ee82c7d3547678c022e1726d07c12e9ff159cedfedd39f914cefd522fad67f\": rpc error: code = NotFound desc = could not find container \"64ee82c7d3547678c022e1726d07c12e9ff159cedfedd39f914cefd522fad67f\": container with ID starting with 64ee82c7d3547678c022e1726d07c12e9ff159cedfedd39f914cefd522fad67f not found: ID does not exist"
Sep 14 02:55:06.921325 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.921282    2564 scope.go:117] "RemoveContainer" containerID="34d5b4c42bd7f2ed7ddd63b21861378c290a610fe6fd5186632daef4db9edd81"
Sep 14 02:55:06.921538 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.921519    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34d5b4c42bd7f2ed7ddd63b21861378c290a610fe6fd5186632daef4db9edd81"} err="failed to get container status \"34d5b4c42bd7f2ed7ddd63b21861378c290a610fe6fd5186632daef4db9edd81\": rpc error: code = NotFound desc = could not find container \"34d5b4c42bd7f2ed7ddd63b21861378c290a610fe6fd5186632daef4db9edd81\": container with ID starting with 34d5b4c42bd7f2ed7ddd63b21861378c290a610fe6fd5186632daef4db9edd81 not found: ID does not exist"
Sep 14 02:55:06.921588 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.921538    2564 scope.go:117] "RemoveContainer" containerID="01104db36bf308971798d8562e27c24d014803f40a52576922181a5634f7ad1e"
Sep 14 02:55:06.921770 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.921752    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01104db36bf308971798d8562e27c24d014803f40a52576922181a5634f7ad1e"} err="failed to get container status \"01104db36bf308971798d8562e27c24d014803f40a52576922181a5634f7ad1e\": rpc error: code = NotFound desc = could not find container \"01104db36bf308971798d8562e27c24d014803f40a52576922181a5634f7ad1e\": container with ID starting with 01104db36bf308971798d8562e27c24d014803f40a52576922181a5634f7ad1e not found: ID does not exist"
Sep 14 02:55:06.921809 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.921771    2564 scope.go:117] "RemoveContainer" containerID="fdff46c1b857b5255645de42ef2c7c4e76db2a945f92d70903a18d7fdfb86187"
Sep 14 02:55:06.921997 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.921980    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdff46c1b857b5255645de42ef2c7c4e76db2a945f92d70903a18d7fdfb86187"} err="failed to get container status \"fdff46c1b857b5255645de42ef2c7c4e76db2a945f92d70903a18d7fdfb86187\": rpc error: code = NotFound desc = could not find container \"fdff46c1b857b5255645de42ef2c7c4e76db2a945f92d70903a18d7fdfb86187\": container with ID starting with fdff46c1b857b5255645de42ef2c7c4e76db2a945f92d70903a18d7fdfb86187 not found: ID does not exist"
Sep 14 02:55:06.922034 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.921997    2564 scope.go:117] "RemoveContainer" containerID="53be009b3e5bbced63e16b80179bfef9c84204e00d481429971cc9be6ca7f11d"
Sep 14 02:55:06.922202 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.922183    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53be009b3e5bbced63e16b80179bfef9c84204e00d481429971cc9be6ca7f11d"} err="failed to get container status \"53be009b3e5bbced63e16b80179bfef9c84204e00d481429971cc9be6ca7f11d\": rpc error: code = NotFound desc = could not find container \"53be009b3e5bbced63e16b80179bfef9c84204e00d481429971cc9be6ca7f11d\": container with ID starting with 53be009b3e5bbced63e16b80179bfef9c84204e00d481429971cc9be6ca7f11d not found: ID does not exist"
Sep 14 02:55:06.922239 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.922202    2564 scope.go:117] "RemoveContainer" containerID="a29a0eb7517dcc7c19389cb1cd97fb7a0debd5e5672f0383296fb7255a5531f6"
Sep 14 02:55:06.922396 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.922381    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a29a0eb7517dcc7c19389cb1cd97fb7a0debd5e5672f0383296fb7255a5531f6"} err="failed to get container status \"a29a0eb7517dcc7c19389cb1cd97fb7a0debd5e5672f0383296fb7255a5531f6\": rpc error: code = NotFound desc = could not find container \"a29a0eb7517dcc7c19389cb1cd97fb7a0debd5e5672f0383296fb7255a5531f6\": container with ID starting with a29a0eb7517dcc7c19389cb1cd97fb7a0debd5e5672f0383296fb7255a5531f6 not found: ID does not exist"
Sep 14 02:55:06.922532 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.922395    2564 scope.go:117] "RemoveContainer" containerID="35879a0a0769cfc374379b4d93cdab1b2d9bc1d95d51317d090cce1fda427d38"
Sep 14 02:55:06.922648 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.922633    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35879a0a0769cfc374379b4d93cdab1b2d9bc1d95d51317d090cce1fda427d38"} err="failed to get container status \"35879a0a0769cfc374379b4d93cdab1b2d9bc1d95d51317d090cce1fda427d38\": rpc error: code = NotFound desc = could not find container \"35879a0a0769cfc374379b4d93cdab1b2d9bc1d95d51317d090cce1fda427d38\": container with ID starting with 35879a0a0769cfc374379b4d93cdab1b2d9bc1d95d51317d090cce1fda427d38 not found: ID does not exist"
Sep 14 02:55:06.922648 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.922647    2564 scope.go:117] "RemoveContainer" containerID="64ee82c7d3547678c022e1726d07c12e9ff159cedfedd39f914cefd522fad67f"
Sep 14 02:55:06.922862 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.922844    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64ee82c7d3547678c022e1726d07c12e9ff159cedfedd39f914cefd522fad67f"} err="failed to get container status \"64ee82c7d3547678c022e1726d07c12e9ff159cedfedd39f914cefd522fad67f\": rpc error: code = NotFound desc = could not find container \"64ee82c7d3547678c022e1726d07c12e9ff159cedfedd39f914cefd522fad67f\": container with ID starting with 64ee82c7d3547678c022e1726d07c12e9ff159cedfedd39f914cefd522fad67f not found: ID does not exist"
Sep 14 02:55:06.922904 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.922862    2564 scope.go:117] "RemoveContainer" containerID="34d5b4c42bd7f2ed7ddd63b21861378c290a610fe6fd5186632daef4db9edd81"
Sep 14 02:55:06.923063 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.923044    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34d5b4c42bd7f2ed7ddd63b21861378c290a610fe6fd5186632daef4db9edd81"} err="failed to get container status \"34d5b4c42bd7f2ed7ddd63b21861378c290a610fe6fd5186632daef4db9edd81\": rpc error: code = NotFound desc = could not find container \"34d5b4c42bd7f2ed7ddd63b21861378c290a610fe6fd5186632daef4db9edd81\": container with ID starting with 34d5b4c42bd7f2ed7ddd63b21861378c290a610fe6fd5186632daef4db9edd81 not found: ID does not exist"
Sep 14 02:55:06.923063 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.923063    2564 scope.go:117] "RemoveContainer" containerID="01104db36bf308971798d8562e27c24d014803f40a52576922181a5634f7ad1e"
Sep 14 02:55:06.923289 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.923273    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01104db36bf308971798d8562e27c24d014803f40a52576922181a5634f7ad1e"} err="failed to get container status \"01104db36bf308971798d8562e27c24d014803f40a52576922181a5634f7ad1e\": rpc error: code = NotFound desc = could not find container \"01104db36bf308971798d8562e27c24d014803f40a52576922181a5634f7ad1e\": container with ID starting with 01104db36bf308971798d8562e27c24d014803f40a52576922181a5634f7ad1e not found: ID does not exist"
Sep 14 02:55:06.923328 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.923290    2564 scope.go:117] "RemoveContainer" containerID="fdff46c1b857b5255645de42ef2c7c4e76db2a945f92d70903a18d7fdfb86187"
Sep 14 02:55:06.923507 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.923487    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdff46c1b857b5255645de42ef2c7c4e76db2a945f92d70903a18d7fdfb86187"} err="failed to get container status \"fdff46c1b857b5255645de42ef2c7c4e76db2a945f92d70903a18d7fdfb86187\": rpc error: code = NotFound desc = could not find container \"fdff46c1b857b5255645de42ef2c7c4e76db2a945f92d70903a18d7fdfb86187\": container with ID starting with fdff46c1b857b5255645de42ef2c7c4e76db2a945f92d70903a18d7fdfb86187 not found: ID does not exist"
Sep 14 02:55:06.923554 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.923508    2564 scope.go:117] "RemoveContainer" containerID="53be009b3e5bbced63e16b80179bfef9c84204e00d481429971cc9be6ca7f11d"
Sep 14 02:55:06.923722 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.923705    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53be009b3e5bbced63e16b80179bfef9c84204e00d481429971cc9be6ca7f11d"} err="failed to get container status \"53be009b3e5bbced63e16b80179bfef9c84204e00d481429971cc9be6ca7f11d\": rpc error: code = NotFound desc = could not find container \"53be009b3e5bbced63e16b80179bfef9c84204e00d481429971cc9be6ca7f11d\": container with ID starting with 53be009b3e5bbced63e16b80179bfef9c84204e00d481429971cc9be6ca7f11d not found: ID does not exist"
Sep 14 02:55:06.923763 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.923723    2564 scope.go:117] "RemoveContainer" containerID="a29a0eb7517dcc7c19389cb1cd97fb7a0debd5e5672f0383296fb7255a5531f6"
Sep 14 02:55:06.923916 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.923898    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a29a0eb7517dcc7c19389cb1cd97fb7a0debd5e5672f0383296fb7255a5531f6"} err="failed to get container status \"a29a0eb7517dcc7c19389cb1cd97fb7a0debd5e5672f0383296fb7255a5531f6\": rpc error: code = NotFound desc = could not find container \"a29a0eb7517dcc7c19389cb1cd97fb7a0debd5e5672f0383296fb7255a5531f6\": container with ID starting with a29a0eb7517dcc7c19389cb1cd97fb7a0debd5e5672f0383296fb7255a5531f6 not found: ID does not exist"
Sep 14 02:55:06.923953 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.923916    2564 scope.go:117] "RemoveContainer" containerID="35879a0a0769cfc374379b4d93cdab1b2d9bc1d95d51317d090cce1fda427d38"
Sep 14 02:55:06.924136 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.924119    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35879a0a0769cfc374379b4d93cdab1b2d9bc1d95d51317d090cce1fda427d38"} err="failed to get container status \"35879a0a0769cfc374379b4d93cdab1b2d9bc1d95d51317d090cce1fda427d38\": rpc error: code = NotFound desc = could not find container \"35879a0a0769cfc374379b4d93cdab1b2d9bc1d95d51317d090cce1fda427d38\": container with ID starting with 35879a0a0769cfc374379b4d93cdab1b2d9bc1d95d51317d090cce1fda427d38 not found: ID does not exist"
Sep 14 02:55:06.924183 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.924136    2564 scope.go:117] "RemoveContainer" containerID="64ee82c7d3547678c022e1726d07c12e9ff159cedfedd39f914cefd522fad67f"
Sep 14 02:55:06.924336 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.924316    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64ee82c7d3547678c022e1726d07c12e9ff159cedfedd39f914cefd522fad67f"} err="failed to get container status \"64ee82c7d3547678c022e1726d07c12e9ff159cedfedd39f914cefd522fad67f\": rpc error: code = NotFound desc = could not find container \"64ee82c7d3547678c022e1726d07c12e9ff159cedfedd39f914cefd522fad67f\": container with ID starting with 64ee82c7d3547678c022e1726d07c12e9ff159cedfedd39f914cefd522fad67f not found: ID does not exist"
Sep 14 02:55:06.924382 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.924336    2564 scope.go:117] "RemoveContainer" containerID="34d5b4c42bd7f2ed7ddd63b21861378c290a610fe6fd5186632daef4db9edd81"
Sep 14 02:55:06.924563 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.924542    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34d5b4c42bd7f2ed7ddd63b21861378c290a610fe6fd5186632daef4db9edd81"} err="failed to get container status \"34d5b4c42bd7f2ed7ddd63b21861378c290a610fe6fd5186632daef4db9edd81\": rpc error: code = NotFound desc = could not find container \"34d5b4c42bd7f2ed7ddd63b21861378c290a610fe6fd5186632daef4db9edd81\": container with ID starting with 34d5b4c42bd7f2ed7ddd63b21861378c290a610fe6fd5186632daef4db9edd81 not found: ID does not exist"
Sep 14 02:55:06.924608 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.924566    2564 scope.go:117] "RemoveContainer" containerID="01104db36bf308971798d8562e27c24d014803f40a52576922181a5634f7ad1e"
Sep 14 02:55:06.924778 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.924762    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01104db36bf308971798d8562e27c24d014803f40a52576922181a5634f7ad1e"} err="failed to get container status \"01104db36bf308971798d8562e27c24d014803f40a52576922181a5634f7ad1e\": rpc error: code = NotFound desc = could not find container \"01104db36bf308971798d8562e27c24d014803f40a52576922181a5634f7ad1e\": container with ID starting with 01104db36bf308971798d8562e27c24d014803f40a52576922181a5634f7ad1e not found: ID does not exist"
Sep 14 02:55:06.924818 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.924778    2564 scope.go:117] "RemoveContainer" containerID="fdff46c1b857b5255645de42ef2c7c4e76db2a945f92d70903a18d7fdfb86187"
Sep 14 02:55:06.925000 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.924983    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdff46c1b857b5255645de42ef2c7c4e76db2a945f92d70903a18d7fdfb86187"} err="failed to get container status \"fdff46c1b857b5255645de42ef2c7c4e76db2a945f92d70903a18d7fdfb86187\": rpc error: code = NotFound desc = could not find container \"fdff46c1b857b5255645de42ef2c7c4e76db2a945f92d70903a18d7fdfb86187\": container with ID starting with fdff46c1b857b5255645de42ef2c7c4e76db2a945f92d70903a18d7fdfb86187 not found: ID does not exist"
Sep 14 02:55:06.925000 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.925000    2564 scope.go:117] "RemoveContainer" containerID="53be009b3e5bbced63e16b80179bfef9c84204e00d481429971cc9be6ca7f11d"
Sep 14 02:55:06.925211 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.925195    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53be009b3e5bbced63e16b80179bfef9c84204e00d481429971cc9be6ca7f11d"} err="failed to get container status \"53be009b3e5bbced63e16b80179bfef9c84204e00d481429971cc9be6ca7f11d\": rpc error: code = NotFound desc = could not find container \"53be009b3e5bbced63e16b80179bfef9c84204e00d481429971cc9be6ca7f11d\": container with ID starting with 53be009b3e5bbced63e16b80179bfef9c84204e00d481429971cc9be6ca7f11d not found: ID does not exist"
Sep 14 02:55:06.925250 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.925211    2564 scope.go:117] "RemoveContainer" containerID="a29a0eb7517dcc7c19389cb1cd97fb7a0debd5e5672f0383296fb7255a5531f6"
Sep 14 02:55:06.925433 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.925415    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a29a0eb7517dcc7c19389cb1cd97fb7a0debd5e5672f0383296fb7255a5531f6"} err="failed to get container status \"a29a0eb7517dcc7c19389cb1cd97fb7a0debd5e5672f0383296fb7255a5531f6\": rpc error: code = NotFound desc = could not find container \"a29a0eb7517dcc7c19389cb1cd97fb7a0debd5e5672f0383296fb7255a5531f6\": container with ID starting with a29a0eb7517dcc7c19389cb1cd97fb7a0debd5e5672f0383296fb7255a5531f6 not found: ID does not exist"
Sep 14 02:55:06.925477 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.925434    2564 scope.go:117] "RemoveContainer" containerID="35879a0a0769cfc374379b4d93cdab1b2d9bc1d95d51317d090cce1fda427d38"
Sep 14 02:55:06.925626 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.925611    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35879a0a0769cfc374379b4d93cdab1b2d9bc1d95d51317d090cce1fda427d38"} err="failed to get container status \"35879a0a0769cfc374379b4d93cdab1b2d9bc1d95d51317d090cce1fda427d38\": rpc error: code = NotFound desc = could not find container \"35879a0a0769cfc374379b4d93cdab1b2d9bc1d95d51317d090cce1fda427d38\": container with ID starting with 35879a0a0769cfc374379b4d93cdab1b2d9bc1d95d51317d090cce1fda427d38 not found: ID does not exist"
Sep 14 02:55:06.958983 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.958957    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 14 02:55:06.959420 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.959390    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="603095db-c9ac-44a7-b91a-c8f7ebde9397" containerName="config-reloader"
Sep 14 02:55:06.959420 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.959421    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="603095db-c9ac-44a7-b91a-c8f7ebde9397" containerName="config-reloader"
Sep 14 02:55:06.959554 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.959433    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="603095db-c9ac-44a7-b91a-c8f7ebde9397" containerName="kube-rbac-proxy-thanos"
Sep 14 02:55:06.959554 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.959441    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="603095db-c9ac-44a7-b91a-c8f7ebde9397" containerName="kube-rbac-proxy-thanos"
Sep 14 02:55:06.959554 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.959467    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="603095db-c9ac-44a7-b91a-c8f7ebde9397" containerName="init-config-reloader"
Sep 14 02:55:06.959554 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.959476    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="603095db-c9ac-44a7-b91a-c8f7ebde9397" containerName="init-config-reloader"
Sep 14 02:55:06.959554 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.959486    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="603095db-c9ac-44a7-b91a-c8f7ebde9397" containerName="kube-rbac-proxy-web"
Sep 14 02:55:06.959554 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.959494    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="603095db-c9ac-44a7-b91a-c8f7ebde9397" containerName="kube-rbac-proxy-web"
Sep 14 02:55:06.959554 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.959507    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="603095db-c9ac-44a7-b91a-c8f7ebde9397" containerName="prometheus"
Sep 14 02:55:06.959554 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.959522    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="603095db-c9ac-44a7-b91a-c8f7ebde9397" containerName="prometheus"
Sep 14 02:55:06.959554 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.959538    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="603095db-c9ac-44a7-b91a-c8f7ebde9397" containerName="thanos-sidecar"
Sep 14 02:55:06.959554 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.959546    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="603095db-c9ac-44a7-b91a-c8f7ebde9397" containerName="thanos-sidecar"
Sep 14 02:55:06.959554 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.959557    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="603095db-c9ac-44a7-b91a-c8f7ebde9397" containerName="kube-rbac-proxy"
Sep 14 02:55:06.960027 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.959566    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="603095db-c9ac-44a7-b91a-c8f7ebde9397" containerName="kube-rbac-proxy"
Sep 14 02:55:06.960027 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.959674    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="603095db-c9ac-44a7-b91a-c8f7ebde9397" containerName="prometheus"
Sep 14 02:55:06.960027 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.959686    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="603095db-c9ac-44a7-b91a-c8f7ebde9397" containerName="kube-rbac-proxy-thanos"
Sep 14 02:55:06.960027 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.959699    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="603095db-c9ac-44a7-b91a-c8f7ebde9397" containerName="thanos-sidecar"
Sep 14 02:55:06.960027 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.959707    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="603095db-c9ac-44a7-b91a-c8f7ebde9397" containerName="config-reloader"
Sep 14 02:55:06.960027 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.959716    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="603095db-c9ac-44a7-b91a-c8f7ebde9397" containerName="kube-rbac-proxy-web"
Sep 14 02:55:06.960027 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.959726    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="603095db-c9ac-44a7-b91a-c8f7ebde9397" containerName="kube-rbac-proxy"
Sep 14 02:55:06.965239 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.965223    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:55:06.969618 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.969599    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 14 02:55:06.969747 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.969612    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9l9d1jhjomc4c\""
Sep 14 02:55:06.969747 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.969602    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 14 02:55:06.969747 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.969610    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 14 02:55:06.969747 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.969694    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 14 02:55:06.974504 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.974485    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 14 02:55:06.975652 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.975637    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 14 02:55:06.975809 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.975797    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 14 02:55:06.976520 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.976482    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 14 02:55:06.978709 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.978693    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 14 02:55:06.978825 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.978806    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 14 02:55:06.978967 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.978950    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-rdxjb\""
Sep 14 02:55:06.979023 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:06.979010    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 14 02:55:07.011913 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:07.011880    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 14 02:55:07.021726 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:07.021700    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 14 02:55:07.029753 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:07.029721    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 14 02:55:07.033765 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:07.033737    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 14 02:55:07.097884 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:07.097844    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="603095db-c9ac-44a7-b91a-c8f7ebde9397" path="/var/lib/kubelet/pods/603095db-c9ac-44a7-b91a-c8f7ebde9397/volumes"
Sep 14 02:55:07.157518 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:07.157432    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9dc99581-91d5-43c1-95d6-3c42e3f21de8-config\") pod \"prometheus-k8s-0\" (UID: \"9dc99581-91d5-43c1-95d6-3c42e3f21de8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:55:07.157518 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:07.157485    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9dc99581-91d5-43c1-95d6-3c42e3f21de8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9dc99581-91d5-43c1-95d6-3c42e3f21de8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:55:07.157518 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:07.157510    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dc99581-91d5-43c1-95d6-3c42e3f21de8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9dc99581-91d5-43c1-95d6-3c42e3f21de8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:55:07.157710 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:07.157534    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dc99581-91d5-43c1-95d6-3c42e3f21de8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9dc99581-91d5-43c1-95d6-3c42e3f21de8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:55:07.157710 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:07.157593    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9dc99581-91d5-43c1-95d6-3c42e3f21de8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9dc99581-91d5-43c1-95d6-3c42e3f21de8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:55:07.157710 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:07.157614    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9dc99581-91d5-43c1-95d6-3c42e3f21de8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9dc99581-91d5-43c1-95d6-3c42e3f21de8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:55:07.157710 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:07.157632    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9dc99581-91d5-43c1-95d6-3c42e3f21de8-config-out\") pod \"prometheus-k8s-0\" (UID: \"9dc99581-91d5-43c1-95d6-3c42e3f21de8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:55:07.157710 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:07.157661    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9dc99581-91d5-43c1-95d6-3c42e3f21de8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9dc99581-91d5-43c1-95d6-3c42e3f21de8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:55:07.157710 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:07.157695    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9dc99581-91d5-43c1-95d6-3c42e3f21de8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9dc99581-91d5-43c1-95d6-3c42e3f21de8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:55:07.157710 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:07.157710    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dc99581-91d5-43c1-95d6-3c42e3f21de8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9dc99581-91d5-43c1-95d6-3c42e3f21de8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:55:07.157927 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:07.157738    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9dc99581-91d5-43c1-95d6-3c42e3f21de8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9dc99581-91d5-43c1-95d6-3c42e3f21de8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:55:07.157927 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:07.157755    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9dc99581-91d5-43c1-95d6-3c42e3f21de8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9dc99581-91d5-43c1-95d6-3c42e3f21de8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:55:07.157927 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:07.157770    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9dc99581-91d5-43c1-95d6-3c42e3f21de8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9dc99581-91d5-43c1-95d6-3c42e3f21de8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:55:07.157927 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:07.157800    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9dc99581-91d5-43c1-95d6-3c42e3f21de8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9dc99581-91d5-43c1-95d6-3c42e3f21de8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:55:07.157927 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:07.157819    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9dc99581-91d5-43c1-95d6-3c42e3f21de8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9dc99581-91d5-43c1-95d6-3c42e3f21de8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:55:07.157927 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:07.157842    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8smkp\" (UniqueName: \"kubernetes.io/projected/9dc99581-91d5-43c1-95d6-3c42e3f21de8-kube-api-access-8smkp\") pod \"prometheus-k8s-0\" (UID: \"9dc99581-91d5-43c1-95d6-3c42e3f21de8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:55:07.157927 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:07.157860    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9dc99581-91d5-43c1-95d6-3c42e3f21de8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9dc99581-91d5-43c1-95d6-3c42e3f21de8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:55:07.157927 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:07.157907    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9dc99581-91d5-43c1-95d6-3c42e3f21de8-web-config\") pod \"prometheus-k8s-0\" (UID: \"9dc99581-91d5-43c1-95d6-3c42e3f21de8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:55:07.158149 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:07.157939    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9dc99581-91d5-43c1-95d6-3c42e3f21de8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9dc99581-91d5-43c1-95d6-3c42e3f21de8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:55:07.158149 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:07.157966    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9dc99581-91d5-43c1-95d6-3c42e3f21de8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9dc99581-91d5-43c1-95d6-3c42e3f21de8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:55:07.259221 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:07.259185    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9dc99581-91d5-43c1-95d6-3c42e3f21de8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9dc99581-91d5-43c1-95d6-3c42e3f21de8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:55:07.259397 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:07.259234    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dc99581-91d5-43c1-95d6-3c42e3f21de8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9dc99581-91d5-43c1-95d6-3c42e3f21de8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:55:07.259513 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:07.259426    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9dc99581-91d5-43c1-95d6-3c42e3f21de8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9dc99581-91d5-43c1-95d6-3c42e3f21de8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:55:07.259513 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:07.259474    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9dc99581-91d5-43c1-95d6-3c42e3f21de8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9dc99581-91d5-43c1-95d6-3c42e3f21de8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:55:07.260252 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:07.259654    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9dc99581-91d5-43c1-95d6-3c42e3f21de8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9dc99581-91d5-43c1-95d6-3c42e3f21de8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:55:07.260252 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:07.259691    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9dc99581-91d5-43c1-95d6-3c42e3f21de8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9dc99581-91d5-43c1-95d6-3c42e3f21de8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:55:07.260252 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:07.259721    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9dc99581-91d5-43c1-95d6-3c42e3f21de8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9dc99581-91d5-43c1-95d6-3c42e3f21de8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:55:07.260252 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:07.259768    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8smkp\" (UniqueName: \"kubernetes.io/projected/9dc99581-91d5-43c1-95d6-3c42e3f21de8-kube-api-access-8smkp\") pod \"prometheus-k8s-0\" (UID: \"9dc99581-91d5-43c1-95d6-3c42e3f21de8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:55:07.260252 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:07.259808    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9dc99581-91d5-43c1-95d6-3c42e3f21de8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9dc99581-91d5-43c1-95d6-3c42e3f21de8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:55:07.260252 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:07.259837    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9dc99581-91d5-43c1-95d6-3c42e3f21de8-web-config\") pod \"prometheus-k8s-0\" (UID: \"9dc99581-91d5-43c1-95d6-3c42e3f21de8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:55:07.260252 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:07.259868    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9dc99581-91d5-43c1-95d6-3c42e3f21de8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9dc99581-91d5-43c1-95d6-3c42e3f21de8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:55:07.260252 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:07.259900    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9dc99581-91d5-43c1-95d6-3c42e3f21de8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9dc99581-91d5-43c1-95d6-3c42e3f21de8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:55:07.260252 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:07.259953    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9dc99581-91d5-43c1-95d6-3c42e3f21de8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9dc99581-91d5-43c1-95d6-3c42e3f21de8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:55:07.260252 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:07.259998    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9dc99581-91d5-43c1-95d6-3c42e3f21de8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9dc99581-91d5-43c1-95d6-3c42e3f21de8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:55:07.260252 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:07.260002    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dc99581-91d5-43c1-95d6-3c42e3f21de8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9dc99581-91d5-43c1-95d6-3c42e3f21de8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:55:07.260252 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:07.259961    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9dc99581-91d5-43c1-95d6-3c42e3f21de8-config\") pod \"prometheus-k8s-0\" (UID: \"9dc99581-91d5-43c1-95d6-3c42e3f21de8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:55:07.260939 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:07.260261    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9dc99581-91d5-43c1-95d6-3c42e3f21de8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9dc99581-91d5-43c1-95d6-3c42e3f21de8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:55:07.260939 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:07.260295    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dc99581-91d5-43c1-95d6-3c42e3f21de8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9dc99581-91d5-43c1-95d6-3c42e3f21de8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:55:07.260939 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:07.260342    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dc99581-91d5-43c1-95d6-3c42e3f21de8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9dc99581-91d5-43c1-95d6-3c42e3f21de8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:55:07.260939 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:07.260395    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9dc99581-91d5-43c1-95d6-3c42e3f21de8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9dc99581-91d5-43c1-95d6-3c42e3f21de8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:55:07.260939 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:07.260451    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9dc99581-91d5-43c1-95d6-3c42e3f21de8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9dc99581-91d5-43c1-95d6-3c42e3f21de8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:55:07.260939 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:07.260479    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9dc99581-91d5-43c1-95d6-3c42e3f21de8-config-out\") pod \"prometheus-k8s-0\" (UID: \"9dc99581-91d5-43c1-95d6-3c42e3f21de8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:55:07.260939 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:07.260490    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9dc99581-91d5-43c1-95d6-3c42e3f21de8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9dc99581-91d5-43c1-95d6-3c42e3f21de8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:55:07.260939 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:07.260505    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9dc99581-91d5-43c1-95d6-3c42e3f21de8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9dc99581-91d5-43c1-95d6-3c42e3f21de8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:55:07.261379 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:07.261122    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9dc99581-91d5-43c1-95d6-3c42e3f21de8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9dc99581-91d5-43c1-95d6-3c42e3f21de8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:55:07.262094 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:07.262068    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dc99581-91d5-43c1-95d6-3c42e3f21de8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9dc99581-91d5-43c1-95d6-3c42e3f21de8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:55:07.262380 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:07.262352    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dc99581-91d5-43c1-95d6-3c42e3f21de8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9dc99581-91d5-43c1-95d6-3c42e3f21de8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:55:07.262799 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:07.262773    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9dc99581-91d5-43c1-95d6-3c42e3f21de8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9dc99581-91d5-43c1-95d6-3c42e3f21de8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:55:07.263441 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:07.263418    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9dc99581-91d5-43c1-95d6-3c42e3f21de8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9dc99581-91d5-43c1-95d6-3c42e3f21de8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:55:07.264171 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:07.264144    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9dc99581-91d5-43c1-95d6-3c42e3f21de8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9dc99581-91d5-43c1-95d6-3c42e3f21de8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:55:07.264262 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:07.264169    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9dc99581-91d5-43c1-95d6-3c42e3f21de8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9dc99581-91d5-43c1-95d6-3c42e3f21de8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:55:07.264560 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:07.264532    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9dc99581-91d5-43c1-95d6-3c42e3f21de8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9dc99581-91d5-43c1-95d6-3c42e3f21de8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:55:07.265080 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:07.265037    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9dc99581-91d5-43c1-95d6-3c42e3f21de8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9dc99581-91d5-43c1-95d6-3c42e3f21de8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:55:07.265177 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:07.265094    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9dc99581-91d5-43c1-95d6-3c42e3f21de8-web-config\") pod \"prometheus-k8s-0\" (UID: \"9dc99581-91d5-43c1-95d6-3c42e3f21de8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:55:07.265236 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:07.265199    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9dc99581-91d5-43c1-95d6-3c42e3f21de8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9dc99581-91d5-43c1-95d6-3c42e3f21de8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:55:07.265834 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:07.265536    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9dc99581-91d5-43c1-95d6-3c42e3f21de8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9dc99581-91d5-43c1-95d6-3c42e3f21de8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:55:07.265834 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:07.265799    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9dc99581-91d5-43c1-95d6-3c42e3f21de8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9dc99581-91d5-43c1-95d6-3c42e3f21de8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:55:07.266266 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:07.266242    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/9dc99581-91d5-43c1-95d6-3c42e3f21de8-config\") pod \"prometheus-k8s-0\" (UID: \"9dc99581-91d5-43c1-95d6-3c42e3f21de8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:55:07.266592 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:07.266573    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9dc99581-91d5-43c1-95d6-3c42e3f21de8-config-out\") pod \"prometheus-k8s-0\" (UID: \"9dc99581-91d5-43c1-95d6-3c42e3f21de8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:55:07.270930 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:07.270911    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8smkp\" (UniqueName: \"kubernetes.io/projected/9dc99581-91d5-43c1-95d6-3c42e3f21de8-kube-api-access-8smkp\") pod \"prometheus-k8s-0\" (UID: \"9dc99581-91d5-43c1-95d6-3c42e3f21de8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:55:07.274801 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:07.274783    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:55:07.574315 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:07.574285    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 14 02:55:07.576337 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:55:07.576306    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dc99581_91d5_43c1_95d6_3c42e3f21de8.slice/crio-bfa58eff95ffec16f4e7dcfd2647ce6f3c47b6526a4c26d536b565a214d4c1eb WatchSource:0}: Error finding container bfa58eff95ffec16f4e7dcfd2647ce6f3c47b6526a4c26d536b565a214d4c1eb: Status 404 returned error can't find the container with id bfa58eff95ffec16f4e7dcfd2647ce6f3c47b6526a4c26d536b565a214d4c1eb
Sep 14 02:55:07.873916 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:07.873833    2564 generic.go:356] "Generic (PLEG): container finished" podID="9dc99581-91d5-43c1-95d6-3c42e3f21de8" containerID="3c618621f46084f65285bebfcae39782a8afb33fcf0c4c4e0b5096ef0c5c87d4" exitCode=0
Sep 14 02:55:07.873916 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:07.873891    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9dc99581-91d5-43c1-95d6-3c42e3f21de8","Type":"ContainerDied","Data":"3c618621f46084f65285bebfcae39782a8afb33fcf0c4c4e0b5096ef0c5c87d4"}
Sep 14 02:55:07.874377 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:07.873985    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9dc99581-91d5-43c1-95d6-3c42e3f21de8","Type":"ContainerStarted","Data":"bfa58eff95ffec16f4e7dcfd2647ce6f3c47b6526a4c26d536b565a214d4c1eb"}
Sep 14 02:55:08.883833 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:08.883800    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-df686ffff-z4729" event={"ID":"1dffd99b-f463-4251-96de-0b8d7f6d5b71","Type":"ContainerStarted","Data":"cf368e36797969f7d304638cdbc9607b64b12284ef93be78a949d9c75f6136e4"}
Sep 14 02:55:08.884289 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:08.883890    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-df686ffff-z4729" event={"ID":"1dffd99b-f463-4251-96de-0b8d7f6d5b71","Type":"ContainerStarted","Data":"8b10ae2d0ab3994ba101e5aa2c24af8b786ba66f5e4731b679f2caeadba93221"}
Sep 14 02:55:08.884289 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:08.883922    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-df686ffff-z4729" event={"ID":"1dffd99b-f463-4251-96de-0b8d7f6d5b71","Type":"ContainerStarted","Data":"00365c58533ac914c7252e839d4885d20bb62a0d9dc85c538b0133220351a6de"}
Sep 14 02:55:08.887059 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:08.887033    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9dc99581-91d5-43c1-95d6-3c42e3f21de8","Type":"ContainerStarted","Data":"fb6231f445d02c2a67f7fd91a6b6670102eee4cf67ff7f6d858066d8bef406a0"}
Sep 14 02:55:08.887150 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:08.887089    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9dc99581-91d5-43c1-95d6-3c42e3f21de8","Type":"ContainerStarted","Data":"6a53563732dea1a57ef820e997d1b43b332df49feaac2708646a15afc27e19c7"}
Sep 14 02:55:08.887150 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:08.887115    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9dc99581-91d5-43c1-95d6-3c42e3f21de8","Type":"ContainerStarted","Data":"fef0ccff76612b0beb41926ed7a7927306d7266b04406c1547f1946355e0f97b"}
Sep 14 02:55:08.887150 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:08.887142    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9dc99581-91d5-43c1-95d6-3c42e3f21de8","Type":"ContainerStarted","Data":"cfe9cd85e4524770d931bea9729649191e16f2b1421a5c9401d57e1c57494b45"}
Sep 14 02:55:08.887246 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:08.887165    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9dc99581-91d5-43c1-95d6-3c42e3f21de8","Type":"ContainerStarted","Data":"0e5cd14e02c3e120851094bd04c53bd1a13982817750cabfc95415aed547f479"}
Sep 14 02:55:08.887246 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:08.887189    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9dc99581-91d5-43c1-95d6-3c42e3f21de8","Type":"ContainerStarted","Data":"d94d539b6f3b15028fcc0ecef8df7231fc1a22b749389749d82a8671c617d929"}
Sep 14 02:55:08.912672 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:08.912634    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-df686ffff-z4729" podStartSLOduration=1.352351163 podStartE2EDuration="2.912621048s" totalImagesPullingTime="1.560269885s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:55:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:55:08.910333825 +0000 UTC m=+154.291418037" watchObservedRunningTime="2026-09-14 02:55:08.912621048 +0000 UTC m=+154.293705247"
Sep 14 02:55:08.962936 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:08.962886    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.9382044130000002 podStartE2EDuration="2.962868768s" totalImagesPullingTime="0s" totalInitContainerRuntime="24.664355ms" isStatefulPod=false podCreationTimestamp="2026-09-14 02:55:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:55:08.961271336 +0000 UTC m=+154.342355568" watchObservedRunningTime="2026-09-14 02:55:08.962868768 +0000 UTC m=+154.343952996"
Sep 14 02:55:09.167667 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:09.167590    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:55:09.287778 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:09.287720    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:55:12.275213 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:12.275181    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:55:13.940076 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:13.940038    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:55:19.741492 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:19.741458    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cd9dc9dbb-256pz"]
Sep 14 02:55:19.895017 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:19.894977    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:55:19.995786 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:19.995696    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:55:20.073830 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:20.073801    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:55:44.769903 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:44.769844    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-cd9dc9dbb-256pz" podUID="3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b" containerName="console" containerID="cri-o://191b491e45f3e873b27b6f81346908c8b8908a90364b8f6def907ad8975bdad7" gracePeriod=15
Sep 14 02:55:45.027630 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:45.027605    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cd9dc9dbb-256pz"
Sep 14 02:55:45.033683 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:45.033653    2564 generic.go:356] "Generic (PLEG): container finished" podID="3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b" containerID="191b491e45f3e873b27b6f81346908c8b8908a90364b8f6def907ad8975bdad7" exitCode=0
Sep 14 02:55:45.033854 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:45.033691    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cd9dc9dbb-256pz"
Sep 14 02:55:45.033854 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:45.033746    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cd9dc9dbb-256pz" event={"ID":"3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b","Type":"ContainerDied","Data":"191b491e45f3e873b27b6f81346908c8b8908a90364b8f6def907ad8975bdad7"}
Sep 14 02:55:45.033854 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:45.033803    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cd9dc9dbb-256pz" event={"ID":"3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b","Type":"ContainerDied","Data":"1ee8edb2b2051afe71650b0b3394b07427f793e9fd28547bc949e3e7541aeff1"}
Sep 14 02:55:45.033854 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:45.033831    2564 scope.go:117] "RemoveContainer" containerID="191b491e45f3e873b27b6f81346908c8b8908a90364b8f6def907ad8975bdad7"
Sep 14 02:55:45.042433 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:45.042398    2564 scope.go:117] "RemoveContainer" containerID="191b491e45f3e873b27b6f81346908c8b8908a90364b8f6def907ad8975bdad7"
Sep 14 02:55:45.042758 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:55:45.042740    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"191b491e45f3e873b27b6f81346908c8b8908a90364b8f6def907ad8975bdad7\": container with ID starting with 191b491e45f3e873b27b6f81346908c8b8908a90364b8f6def907ad8975bdad7 not found: ID does not exist" containerID="191b491e45f3e873b27b6f81346908c8b8908a90364b8f6def907ad8975bdad7"
Sep 14 02:55:45.042820 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:45.042766    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"191b491e45f3e873b27b6f81346908c8b8908a90364b8f6def907ad8975bdad7"} err="failed to get container status \"191b491e45f3e873b27b6f81346908c8b8908a90364b8f6def907ad8975bdad7\": rpc error: code = NotFound desc = could not find container \"191b491e45f3e873b27b6f81346908c8b8908a90364b8f6def907ad8975bdad7\": container with ID starting with 191b491e45f3e873b27b6f81346908c8b8908a90364b8f6def907ad8975bdad7 not found: ID does not exist"
Sep 14 02:55:45.196691 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:45.196645    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b-service-ca\") pod \"3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b\" (UID: \"3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b\") "
Sep 14 02:55:45.196691 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:45.196710    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b-console-serving-cert\") pod \"3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b\" (UID: \"3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b\") "
Sep 14 02:55:45.196939 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:45.196758    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4th7l\" (UniqueName: \"kubernetes.io/projected/3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b-kube-api-access-4th7l\") pod \"3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b\" (UID: \"3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b\") "
Sep 14 02:55:45.196939 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:45.196791    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b-trusted-ca-bundle\") pod \"3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b\" (UID: \"3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b\") "
Sep 14 02:55:45.196939 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:45.196819    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b-console-config\") pod \"3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b\" (UID: \"3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b\") "
Sep 14 02:55:45.196939 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:45.196877    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b-console-oauth-config\") pod \"3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b\" (UID: \"3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b\") "
Sep 14 02:55:45.196939 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:45.196921    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b-oauth-serving-cert\") pod \"3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b\" (UID: \"3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b\") "
Sep 14 02:55:45.197172 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:45.197130    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b-service-ca" (OuterVolumeSpecName: "service-ca") pod "3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b" (UID: "3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 02:55:45.197271 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:45.197237    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b" (UID: "3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 02:55:45.197422 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:45.197341    2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b-trusted-ca-bundle\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 02:55:45.197422 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:45.197364    2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b-service-ca\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 02:55:45.197542 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:45.197383    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b" (UID: "3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 02:55:45.197542 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:45.197460    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b-console-config" (OuterVolumeSpecName: "console-config") pod "3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b" (UID: "3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 02:55:45.199001 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:45.198971    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b" (UID: "3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 02:55:45.199440 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:45.199390    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b" (UID: "3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 02:55:45.199440 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:45.199428    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b-kube-api-access-4th7l" (OuterVolumeSpecName: "kube-api-access-4th7l") pod "3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b" (UID: "3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b"). InnerVolumeSpecName "kube-api-access-4th7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 02:55:45.298819 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:45.298736    2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b-console-oauth-config\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 02:55:45.298819 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:45.298769    2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b-oauth-serving-cert\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 02:55:45.298819 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:45.298782    2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b-console-serving-cert\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 02:55:45.298819 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:45.298795    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4th7l\" (UniqueName: \"kubernetes.io/projected/3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b-kube-api-access-4th7l\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 02:55:45.298819 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:45.298808    2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b-console-config\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 02:55:45.362246 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:45.362209    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cd9dc9dbb-256pz"]
Sep 14 02:55:45.363508 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:45.363486    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-cd9dc9dbb-256pz"]
Sep 14 02:55:45.676671 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:45.676580    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:55:45.778984 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:45.778946    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:55:47.097237 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:55:47.097208    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b" path="/var/lib/kubelet/pods/3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b/volumes"
Sep 14 02:56:07.275007 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:56:07.274955    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:56:07.290363 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:56:07.290331    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:56:08.138186 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:56:08.138159    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 02:56:16.163493 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:56:16.163463    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:56:16.230840 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:56:16.230807    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:56:20.104796 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:56:20.104756    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:56:20.206600 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:56:20.206566    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:56:20.282669 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:56:20.282633    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:56:20.776859 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:56:20.776804    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:56:20.920252 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:56:20.920214    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:56:28.136275 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:56:28.136240    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:56:28.204161 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:56:28.204124    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:56:31.135797 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:56:31.135755    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:56:31.235821 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:56:31.235789    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:56:31.305002 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:56:31.304954    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:56:32.838724 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:56:32.838683    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:56:32.949007 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:56:32.948968    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:56:33.067663 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:56:33.067630    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:56:50.312858 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:56:50.312823    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:56:50.494246 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:56:50.494179    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:56:52.413065 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:56:52.413030    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:56:53.695248 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:56:53.695204    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:56:53.790228 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:56:53.790192    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:57:15.677055 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:15.677007    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:57:16.499317 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:16.499288    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:57:19.296698 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:19.296658    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:57:19.398398 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:19.398358    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:57:20.904640 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:20.904608    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:57:21.004417 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:21.004350    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:57:21.097139 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:21.097109    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-jr59m"]
Sep 14 02:57:21.097434 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:21.097422    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b" containerName="console"
Sep 14 02:57:21.097489 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:21.097435    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b" containerName="console"
Sep 14 02:57:21.097525 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:21.097493    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="3acdfdd4-0913-4d4d-bd8e-fdc448e5ff7b" containerName="console"
Sep 14 02:57:21.100382 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:21.100369    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-jr59m"
Sep 14 02:57:21.103220 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:21.103197    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-g7jgw\""
Sep 14 02:57:21.104338 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:21.104320    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 14 02:57:21.104479 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:21.104342    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 14 02:57:21.107549 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:21.107530    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-jr59m"]
Sep 14 02:57:21.211802 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:21.211712    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fgzx\" (UniqueName: \"kubernetes.io/projected/227ccfab-334d-4d8a-81b6-471d213d7890-kube-api-access-5fgzx\") pod \"cert-manager-webhook-6486ffd997-jr59m\" (UID: \"227ccfab-334d-4d8a-81b6-471d213d7890\") " pod="cert-manager/cert-manager-webhook-6486ffd997-jr59m"
Sep 14 02:57:21.211978 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:21.211849    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/227ccfab-334d-4d8a-81b6-471d213d7890-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-jr59m\" (UID: \"227ccfab-334d-4d8a-81b6-471d213d7890\") " pod="cert-manager/cert-manager-webhook-6486ffd997-jr59m"
Sep 14 02:57:21.312498 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:21.312458    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fgzx\" (UniqueName: \"kubernetes.io/projected/227ccfab-334d-4d8a-81b6-471d213d7890-kube-api-access-5fgzx\") pod \"cert-manager-webhook-6486ffd997-jr59m\" (UID: \"227ccfab-334d-4d8a-81b6-471d213d7890\") " pod="cert-manager/cert-manager-webhook-6486ffd997-jr59m"
Sep 14 02:57:21.312688 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:21.312549    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/227ccfab-334d-4d8a-81b6-471d213d7890-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-jr59m\" (UID: \"227ccfab-334d-4d8a-81b6-471d213d7890\") " pod="cert-manager/cert-manager-webhook-6486ffd997-jr59m"
Sep 14 02:57:21.322584 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:21.322543    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/227ccfab-334d-4d8a-81b6-471d213d7890-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-jr59m\" (UID: \"227ccfab-334d-4d8a-81b6-471d213d7890\") " pod="cert-manager/cert-manager-webhook-6486ffd997-jr59m"
Sep 14 02:57:21.323263 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:21.323246    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fgzx\" (UniqueName: \"kubernetes.io/projected/227ccfab-334d-4d8a-81b6-471d213d7890-kube-api-access-5fgzx\") pod \"cert-manager-webhook-6486ffd997-jr59m\" (UID: \"227ccfab-334d-4d8a-81b6-471d213d7890\") " pod="cert-manager/cert-manager-webhook-6486ffd997-jr59m"
Sep 14 02:57:21.419492 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:21.419461    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-jr59m"
Sep 14 02:57:21.709434 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:21.709351    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-jr59m"]
Sep 14 02:57:21.711632 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:57:21.711605    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod227ccfab_334d_4d8a_81b6_471d213d7890.slice/crio-860afffdd2b94483c7ee529a2969fc5933be4017af679e75668e7d1b1880696f WatchSource:0}: Error finding container 860afffdd2b94483c7ee529a2969fc5933be4017af679e75668e7d1b1880696f: Status 404 returned error can't find the container with id 860afffdd2b94483c7ee529a2969fc5933be4017af679e75668e7d1b1880696f
Sep 14 02:57:21.714072 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:21.714048    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:57:21.821303 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:21.821272    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:57:22.419374 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:22.419336    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-jr59m" event={"ID":"227ccfab-334d-4d8a-81b6-471d213d7890","Type":"ContainerStarted","Data":"860afffdd2b94483c7ee529a2969fc5933be4017af679e75668e7d1b1880696f"}
Sep 14 02:57:23.330816 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:23.330774    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:57:23.714189 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:23.714074    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:57:24.736018 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:24.735987    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-nnnb5"]
Sep 14 02:57:24.740620 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:24.740595    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-nnnb5"
Sep 14 02:57:24.743220 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:24.743200    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-sq7h4\""
Sep 14 02:57:24.749819 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:24.749794    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-nnnb5"]
Sep 14 02:57:24.843594 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:24.843568    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/91301cc0-c72f-460b-8391-5edcc5aba36c-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-nnnb5\" (UID: \"91301cc0-c72f-460b-8391-5edcc5aba36c\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-nnnb5"
Sep 14 02:57:24.843697 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:24.843598    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wjr6\" (UniqueName: \"kubernetes.io/projected/91301cc0-c72f-460b-8391-5edcc5aba36c-kube-api-access-7wjr6\") pod \"cert-manager-cainjector-559cdd8c87-nnnb5\" (UID: \"91301cc0-c72f-460b-8391-5edcc5aba36c\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-nnnb5"
Sep 14 02:57:24.944180 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:24.944142    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/91301cc0-c72f-460b-8391-5edcc5aba36c-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-nnnb5\" (UID: \"91301cc0-c72f-460b-8391-5edcc5aba36c\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-nnnb5"
Sep 14 02:57:24.944180 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:24.944177    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wjr6\" (UniqueName: \"kubernetes.io/projected/91301cc0-c72f-460b-8391-5edcc5aba36c-kube-api-access-7wjr6\") pod \"cert-manager-cainjector-559cdd8c87-nnnb5\" (UID: \"91301cc0-c72f-460b-8391-5edcc5aba36c\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-nnnb5"
Sep 14 02:57:24.953054 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:24.953023    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/91301cc0-c72f-460b-8391-5edcc5aba36c-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-nnnb5\" (UID: \"91301cc0-c72f-460b-8391-5edcc5aba36c\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-nnnb5"
Sep 14 02:57:24.953338 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:24.953313    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wjr6\" (UniqueName: \"kubernetes.io/projected/91301cc0-c72f-460b-8391-5edcc5aba36c-kube-api-access-7wjr6\") pod \"cert-manager-cainjector-559cdd8c87-nnnb5\" (UID: \"91301cc0-c72f-460b-8391-5edcc5aba36c\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-nnnb5"
Sep 14 02:57:25.050342 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:25.050296    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-nnnb5"
Sep 14 02:57:25.326195 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:25.326168    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-nnnb5"]
Sep 14 02:57:25.328305 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:57:25.328277    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod91301cc0_c72f_460b_8391_5edcc5aba36c.slice/crio-b3ecb2ba3e133a5f3efa6da8fc17a5e5eb586bc1e02e782d3f1dc6fb99809696 WatchSource:0}: Error finding container b3ecb2ba3e133a5f3efa6da8fc17a5e5eb586bc1e02e782d3f1dc6fb99809696: Status 404 returned error can't find the container with id b3ecb2ba3e133a5f3efa6da8fc17a5e5eb586bc1e02e782d3f1dc6fb99809696
Sep 14 02:57:25.434786 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:25.434745    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-jr59m" event={"ID":"227ccfab-334d-4d8a-81b6-471d213d7890","Type":"ContainerStarted","Data":"7fb2b41607283608e8c29078324144a17bb43d62ddc390c5dad4601cc2d3f89e"}
Sep 14 02:57:25.434972 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:25.434800    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-jr59m"
Sep 14 02:57:25.436245 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:25.436215    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-nnnb5" event={"ID":"91301cc0-c72f-460b-8391-5edcc5aba36c","Type":"ContainerStarted","Data":"af3bea7bb164247fac4db49d5d5e09c199adedcc777013c08857e8b2885bff28"}
Sep 14 02:57:25.436328 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:25.436247    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-nnnb5" event={"ID":"91301cc0-c72f-460b-8391-5edcc5aba36c","Type":"ContainerStarted","Data":"b3ecb2ba3e133a5f3efa6da8fc17a5e5eb586bc1e02e782d3f1dc6fb99809696"}
Sep 14 02:57:25.454088 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:25.454045    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-jr59m" podStartSLOduration=1.359253651 podStartE2EDuration="4.454030741s" totalImagesPullingTime="3.09477709s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:57:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:57:25.452902351 +0000 UTC m=+290.833986551" watchObservedRunningTime="2026-09-14 02:57:25.454030741 +0000 UTC m=+290.835114942"
Sep 14 02:57:25.470501 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:25.470455    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-nnnb5" podStartSLOduration=1.470440289 podStartE2EDuration="1.470440289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:57:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:57:25.468796679 +0000 UTC m=+290.849880877" watchObservedRunningTime="2026-09-14 02:57:25.470440289 +0000 UTC m=+290.851524488"
Sep 14 02:57:26.291456 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:26.291389    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:57:26.355263 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:26.355223    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:57:29.693674 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:29.693639    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:57:29.792678 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:29.792647    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:57:30.718760 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:30.718722    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:57:31.442211 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:31.442183    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-jr59m"
Sep 14 02:57:35.033948 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:35.033916    2564 kubelet.go:1648] "Image garbage collection succeeded"
Sep 14 02:57:35.981214 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:35.981175    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:57:36.088629 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:36.088583    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:57:38.215770 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:38.215739    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:57:49.486231 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:49.486195    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:57:49.576421 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:49.576369    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:57:50.059958 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:50.059925    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:57:50.330233 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:50.330148    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:57:50.428861 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:50.428824    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:57:50.494698 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:50.494665    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67ffb7699c-45bzd"]
Sep 14 02:57:50.498702 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:50.498686    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-67ffb7699c-45bzd"
Sep 14 02:57:50.503937 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:50.503916    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 14 02:57:50.504095 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:50.504071    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 14 02:57:50.504223 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:50.504201    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 14 02:57:50.504284 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:50.504223    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 14 02:57:50.504536 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:50.504520    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-49p7j\""
Sep 14 02:57:50.537196 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:50.537165    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67ffb7699c-45bzd"]
Sep 14 02:57:50.556550 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:50.556515    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vd65\" (UniqueName: \"kubernetes.io/projected/8ffdca28-8d3a-4588-a121-972b21c116ac-kube-api-access-8vd65\") pod \"opendatahub-operator-controller-manager-67ffb7699c-45bzd\" (UID: \"8ffdca28-8d3a-4588-a121-972b21c116ac\") " pod="opendatahub/opendatahub-operator-controller-manager-67ffb7699c-45bzd"
Sep 14 02:57:50.556738 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:50.556594    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8ffdca28-8d3a-4588-a121-972b21c116ac-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67ffb7699c-45bzd\" (UID: \"8ffdca28-8d3a-4588-a121-972b21c116ac\") " pod="opendatahub/opendatahub-operator-controller-manager-67ffb7699c-45bzd"
Sep 14 02:57:50.556738 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:50.556627    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8ffdca28-8d3a-4588-a121-972b21c116ac-manifests\") pod \"opendatahub-operator-controller-manager-67ffb7699c-45bzd\" (UID: \"8ffdca28-8d3a-4588-a121-972b21c116ac\") " pod="opendatahub/opendatahub-operator-controller-manager-67ffb7699c-45bzd"
Sep 14 02:57:50.556738 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:50.556669    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8ffdca28-8d3a-4588-a121-972b21c116ac-webhook-cert\") pod \"opendatahub-operator-controller-manager-67ffb7699c-45bzd\" (UID: \"8ffdca28-8d3a-4588-a121-972b21c116ac\") " pod="opendatahub/opendatahub-operator-controller-manager-67ffb7699c-45bzd"
Sep 14 02:57:50.556738 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:50.556734    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ffdca28-8d3a-4588-a121-972b21c116ac-tmp\") pod \"opendatahub-operator-controller-manager-67ffb7699c-45bzd\" (UID: \"8ffdca28-8d3a-4588-a121-972b21c116ac\") " pod="opendatahub/opendatahub-operator-controller-manager-67ffb7699c-45bzd"
Sep 14 02:57:50.657523 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:50.657436    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8ffdca28-8d3a-4588-a121-972b21c116ac-webhook-cert\") pod \"opendatahub-operator-controller-manager-67ffb7699c-45bzd\" (UID: \"8ffdca28-8d3a-4588-a121-972b21c116ac\") " pod="opendatahub/opendatahub-operator-controller-manager-67ffb7699c-45bzd"
Sep 14 02:57:50.657661 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:50.657526    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ffdca28-8d3a-4588-a121-972b21c116ac-tmp\") pod \"opendatahub-operator-controller-manager-67ffb7699c-45bzd\" (UID: \"8ffdca28-8d3a-4588-a121-972b21c116ac\") " pod="opendatahub/opendatahub-operator-controller-manager-67ffb7699c-45bzd"
Sep 14 02:57:50.657661 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:50.657569    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vd65\" (UniqueName: \"kubernetes.io/projected/8ffdca28-8d3a-4588-a121-972b21c116ac-kube-api-access-8vd65\") pod \"opendatahub-operator-controller-manager-67ffb7699c-45bzd\" (UID: \"8ffdca28-8d3a-4588-a121-972b21c116ac\") " pod="opendatahub/opendatahub-operator-controller-manager-67ffb7699c-45bzd"
Sep 14 02:57:50.657661 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:50.657641    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8ffdca28-8d3a-4588-a121-972b21c116ac-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67ffb7699c-45bzd\" (UID: \"8ffdca28-8d3a-4588-a121-972b21c116ac\") " pod="opendatahub/opendatahub-operator-controller-manager-67ffb7699c-45bzd"
Sep 14 02:57:50.657814 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:50.657667    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8ffdca28-8d3a-4588-a121-972b21c116ac-manifests\") pod \"opendatahub-operator-controller-manager-67ffb7699c-45bzd\" (UID: \"8ffdca28-8d3a-4588-a121-972b21c116ac\") " pod="opendatahub/opendatahub-operator-controller-manager-67ffb7699c-45bzd"
Sep 14 02:57:50.658080 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:50.658053    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8ffdca28-8d3a-4588-a121-972b21c116ac-manifests\") pod \"opendatahub-operator-controller-manager-67ffb7699c-45bzd\" (UID: \"8ffdca28-8d3a-4588-a121-972b21c116ac\") " pod="opendatahub/opendatahub-operator-controller-manager-67ffb7699c-45bzd"
Sep 14 02:57:50.660189 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:50.660144    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ffdca28-8d3a-4588-a121-972b21c116ac-tmp\") pod \"opendatahub-operator-controller-manager-67ffb7699c-45bzd\" (UID: \"8ffdca28-8d3a-4588-a121-972b21c116ac\") " pod="opendatahub/opendatahub-operator-controller-manager-67ffb7699c-45bzd"
Sep 14 02:57:50.660771 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:50.660745    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8ffdca28-8d3a-4588-a121-972b21c116ac-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67ffb7699c-45bzd\" (UID: \"8ffdca28-8d3a-4588-a121-972b21c116ac\") " pod="opendatahub/opendatahub-operator-controller-manager-67ffb7699c-45bzd"
Sep 14 02:57:50.660771 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:50.660768    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8ffdca28-8d3a-4588-a121-972b21c116ac-webhook-cert\") pod \"opendatahub-operator-controller-manager-67ffb7699c-45bzd\" (UID: \"8ffdca28-8d3a-4588-a121-972b21c116ac\") " pod="opendatahub/opendatahub-operator-controller-manager-67ffb7699c-45bzd"
Sep 14 02:57:50.684427 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:50.684380    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vd65\" (UniqueName: \"kubernetes.io/projected/8ffdca28-8d3a-4588-a121-972b21c116ac-kube-api-access-8vd65\") pod \"opendatahub-operator-controller-manager-67ffb7699c-45bzd\" (UID: \"8ffdca28-8d3a-4588-a121-972b21c116ac\") " pod="opendatahub/opendatahub-operator-controller-manager-67ffb7699c-45bzd"
Sep 14 02:57:50.724750 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:50.724714    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:57:50.809864 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:50.809827    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-67ffb7699c-45bzd"
Sep 14 02:57:50.823673 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:50.823642    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:57:50.922856 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:50.922781    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:57:51.026620 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:51.026587    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:57:51.089443 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:51.089420    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67ffb7699c-45bzd"]
Sep 14 02:57:51.091281 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:57:51.091259    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ffdca28_8d3a_4588_a121_972b21c116ac.slice/crio-8c728b359d93bdd4aeefc3496847d71620526dbf5f7db82674c757a31ed5a52e WatchSource:0}: Error finding container 8c728b359d93bdd4aeefc3496847d71620526dbf5f7db82674c757a31ed5a52e: Status 404 returned error can't find the container with id 8c728b359d93bdd4aeefc3496847d71620526dbf5f7db82674c757a31ed5a52e
Sep 14 02:57:51.093283 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:51.093262    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 14 02:57:51.106014 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:51.105981    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:57:51.325045 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:51.325014    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:57:51.457734 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:51.457705    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:57:51.544068 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:51.544032    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67ffb7699c-45bzd" event={"ID":"8ffdca28-8d3a-4588-a121-972b21c116ac","Type":"ContainerStarted","Data":"8c728b359d93bdd4aeefc3496847d71620526dbf5f7db82674c757a31ed5a52e"}
Sep 14 02:57:54.559595 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:54.559551    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67ffb7699c-45bzd" event={"ID":"8ffdca28-8d3a-4588-a121-972b21c116ac","Type":"ContainerStarted","Data":"d4b2f5f2b81720691f121a6ee791b75c1a312cd763402fffb1d9b8d4b6168d9b"}
Sep 14 02:57:55.565318 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:55.565290    2564 generic.go:356] "Generic (PLEG): container finished" podID="8ffdca28-8d3a-4588-a121-972b21c116ac" containerID="d4b2f5f2b81720691f121a6ee791b75c1a312cd763402fffb1d9b8d4b6168d9b" exitCode=0
Sep 14 02:57:55.565684 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:55.565355    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67ffb7699c-45bzd" event={"ID":"8ffdca28-8d3a-4588-a121-972b21c116ac","Type":"ContainerDied","Data":"d4b2f5f2b81720691f121a6ee791b75c1a312cd763402fffb1d9b8d4b6168d9b"}
Sep 14 02:57:56.571496 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:56.571461    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67ffb7699c-45bzd" event={"ID":"8ffdca28-8d3a-4588-a121-972b21c116ac","Type":"ContainerStarted","Data":"0550c8fe017fa462c9683a1338bf272e853fbf67b9b652015ceda6b456655b19"}
Sep 14 02:57:56.571932 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:56.571603    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-67ffb7699c-45bzd"
Sep 14 02:57:56.601399 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:56.601342    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-67ffb7699c-45bzd" podStartSLOduration=2.29183759 podStartE2EDuration="6.601322475s" totalImagesPullingTime="3.246423687s" totalInitContainerRuntime="1.063061198s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:57:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:57:56.598679308 +0000 UTC m=+321.979763531" watchObservedRunningTime="2026-09-14 02:57:56.601322475 +0000 UTC m=+321.982406675"
Sep 14 02:57:58.814028 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:58.813993    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:57:58.881614 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:57:58.881579    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:58:06.520227 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:06.520187    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:58:06.604941 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:06.604899    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:58:07.583337 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:07.583311    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-67ffb7699c-45bzd"
Sep 14 02:58:08.529850 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:08.529817    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-59b9ccfbf8-2xdrk"]
Sep 14 02:58:08.537275 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:08.537247    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-59b9ccfbf8-2xdrk"
Sep 14 02:58:08.543130 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:08.543081    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 14 02:58:08.543130 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:08.543107    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 14 02:58:08.543430 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:08.543227    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-mqbb7\""
Sep 14 02:58:08.543430 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:08.543094    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 14 02:58:08.543430 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:08.543096    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 14 02:58:08.545080 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:08.544579    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 14 02:58:08.562918 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:08.562882    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-59b9ccfbf8-2xdrk"]
Sep 14 02:58:08.616174 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:08.616142    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/71f93d7d-55d4-43d5-bdb4-53e942788755-metrics-cert\") pod \"lws-controller-manager-59b9ccfbf8-2xdrk\" (UID: \"71f93d7d-55d4-43d5-bdb4-53e942788755\") " pod="openshift-lws-operator/lws-controller-manager-59b9ccfbf8-2xdrk"
Sep 14 02:58:08.616583 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:08.616185    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/71f93d7d-55d4-43d5-bdb4-53e942788755-manager-config\") pod \"lws-controller-manager-59b9ccfbf8-2xdrk\" (UID: \"71f93d7d-55d4-43d5-bdb4-53e942788755\") " pod="openshift-lws-operator/lws-controller-manager-59b9ccfbf8-2xdrk"
Sep 14 02:58:08.616583 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:08.616212    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/71f93d7d-55d4-43d5-bdb4-53e942788755-cert\") pod \"lws-controller-manager-59b9ccfbf8-2xdrk\" (UID: \"71f93d7d-55d4-43d5-bdb4-53e942788755\") " pod="openshift-lws-operator/lws-controller-manager-59b9ccfbf8-2xdrk"
Sep 14 02:58:08.616583 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:08.616300    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd27k\" (UniqueName: \"kubernetes.io/projected/71f93d7d-55d4-43d5-bdb4-53e942788755-kube-api-access-fd27k\") pod \"lws-controller-manager-59b9ccfbf8-2xdrk\" (UID: \"71f93d7d-55d4-43d5-bdb4-53e942788755\") " pod="openshift-lws-operator/lws-controller-manager-59b9ccfbf8-2xdrk"
Sep 14 02:58:08.717028 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:08.716989    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/71f93d7d-55d4-43d5-bdb4-53e942788755-manager-config\") pod \"lws-controller-manager-59b9ccfbf8-2xdrk\" (UID: \"71f93d7d-55d4-43d5-bdb4-53e942788755\") " pod="openshift-lws-operator/lws-controller-manager-59b9ccfbf8-2xdrk"
Sep 14 02:58:08.717193 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:08.717042    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/71f93d7d-55d4-43d5-bdb4-53e942788755-cert\") pod \"lws-controller-manager-59b9ccfbf8-2xdrk\" (UID: \"71f93d7d-55d4-43d5-bdb4-53e942788755\") " pod="openshift-lws-operator/lws-controller-manager-59b9ccfbf8-2xdrk"
Sep 14 02:58:08.717193 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:08.717103    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd27k\" (UniqueName: \"kubernetes.io/projected/71f93d7d-55d4-43d5-bdb4-53e942788755-kube-api-access-fd27k\") pod \"lws-controller-manager-59b9ccfbf8-2xdrk\" (UID: \"71f93d7d-55d4-43d5-bdb4-53e942788755\") " pod="openshift-lws-operator/lws-controller-manager-59b9ccfbf8-2xdrk"
Sep 14 02:58:08.717193 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:08.717158    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/71f93d7d-55d4-43d5-bdb4-53e942788755-metrics-cert\") pod \"lws-controller-manager-59b9ccfbf8-2xdrk\" (UID: \"71f93d7d-55d4-43d5-bdb4-53e942788755\") " pod="openshift-lws-operator/lws-controller-manager-59b9ccfbf8-2xdrk"
Sep 14 02:58:08.717805 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:08.717784    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/71f93d7d-55d4-43d5-bdb4-53e942788755-manager-config\") pod \"lws-controller-manager-59b9ccfbf8-2xdrk\" (UID: \"71f93d7d-55d4-43d5-bdb4-53e942788755\") " pod="openshift-lws-operator/lws-controller-manager-59b9ccfbf8-2xdrk"
Sep 14 02:58:08.719649 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:08.719631    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/71f93d7d-55d4-43d5-bdb4-53e942788755-cert\") pod \"lws-controller-manager-59b9ccfbf8-2xdrk\" (UID: \"71f93d7d-55d4-43d5-bdb4-53e942788755\") " pod="openshift-lws-operator/lws-controller-manager-59b9ccfbf8-2xdrk"
Sep 14 02:58:08.719731 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:08.719654    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/71f93d7d-55d4-43d5-bdb4-53e942788755-metrics-cert\") pod \"lws-controller-manager-59b9ccfbf8-2xdrk\" (UID: \"71f93d7d-55d4-43d5-bdb4-53e942788755\") " pod="openshift-lws-operator/lws-controller-manager-59b9ccfbf8-2xdrk"
Sep 14 02:58:08.726693 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:08.726675    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd27k\" (UniqueName: \"kubernetes.io/projected/71f93d7d-55d4-43d5-bdb4-53e942788755-kube-api-access-fd27k\") pod \"lws-controller-manager-59b9ccfbf8-2xdrk\" (UID: \"71f93d7d-55d4-43d5-bdb4-53e942788755\") " pod="openshift-lws-operator/lws-controller-manager-59b9ccfbf8-2xdrk"
Sep 14 02:58:08.848929 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:08.848841    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-59b9ccfbf8-2xdrk"
Sep 14 02:58:09.162013 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:09.161934    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-59b9ccfbf8-2xdrk"]
Sep 14 02:58:09.164884 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:58:09.164841    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71f93d7d_55d4_43d5_bdb4_53e942788755.slice/crio-2e6fecb86944413e4ce4c539b99d677ce9d1799c215732230f9de256f54fdad1 WatchSource:0}: Error finding container 2e6fecb86944413e4ce4c539b99d677ce9d1799c215732230f9de256f54fdad1: Status 404 returned error can't find the container with id 2e6fecb86944413e4ce4c539b99d677ce9d1799c215732230f9de256f54fdad1
Sep 14 02:58:09.490760 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:09.490664    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:58:09.635047 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:09.635015    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-59b9ccfbf8-2xdrk" event={"ID":"71f93d7d-55d4-43d5-bdb4-53e942788755","Type":"ContainerStarted","Data":"2e6fecb86944413e4ce4c539b99d677ce9d1799c215732230f9de256f54fdad1"}
Sep 14 02:58:09.728460 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:09.728427    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:58:10.148380 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:10.148339    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:58:10.233505 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:10.233472    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-798674575b-cjnkz"]
Sep 14 02:58:10.237870 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:10.237845    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-798674575b-cjnkz"
Sep 14 02:58:10.248292 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:10.248112    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 14 02:58:10.249309 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:10.248994    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 14 02:58:10.249309 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:10.249084    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 14 02:58:10.250044 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:10.249819    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 14 02:58:10.250725 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:10.250573    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-r9g8v\""
Sep 14 02:58:10.265103 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:10.265040    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-798674575b-cjnkz"]
Sep 14 02:58:10.331800 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:10.331765    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7qc2\" (UniqueName: \"kubernetes.io/projected/f6249c90-2e4c-4814-9df6-08221ea26ae3-kube-api-access-j7qc2\") pod \"kube-auth-proxy-798674575b-cjnkz\" (UID: \"f6249c90-2e4c-4814-9df6-08221ea26ae3\") " pod="openshift-ingress/kube-auth-proxy-798674575b-cjnkz"
Sep 14 02:58:10.331985 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:10.331828    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6249c90-2e4c-4814-9df6-08221ea26ae3-tls-certs\") pod \"kube-auth-proxy-798674575b-cjnkz\" (UID: \"f6249c90-2e4c-4814-9df6-08221ea26ae3\") " pod="openshift-ingress/kube-auth-proxy-798674575b-cjnkz"
Sep 14 02:58:10.331985 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:10.331914    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f6249c90-2e4c-4814-9df6-08221ea26ae3-tmp\") pod \"kube-auth-proxy-798674575b-cjnkz\" (UID: \"f6249c90-2e4c-4814-9df6-08221ea26ae3\") " pod="openshift-ingress/kube-auth-proxy-798674575b-cjnkz"
Sep 14 02:58:10.433416 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:10.433302    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7qc2\" (UniqueName: \"kubernetes.io/projected/f6249c90-2e4c-4814-9df6-08221ea26ae3-kube-api-access-j7qc2\") pod \"kube-auth-proxy-798674575b-cjnkz\" (UID: \"f6249c90-2e4c-4814-9df6-08221ea26ae3\") " pod="openshift-ingress/kube-auth-proxy-798674575b-cjnkz"
Sep 14 02:58:10.433590 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:10.433452    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6249c90-2e4c-4814-9df6-08221ea26ae3-tls-certs\") pod \"kube-auth-proxy-798674575b-cjnkz\" (UID: \"f6249c90-2e4c-4814-9df6-08221ea26ae3\") " pod="openshift-ingress/kube-auth-proxy-798674575b-cjnkz"
Sep 14 02:58:10.433590 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:10.433543    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f6249c90-2e4c-4814-9df6-08221ea26ae3-tmp\") pod \"kube-auth-proxy-798674575b-cjnkz\" (UID: \"f6249c90-2e4c-4814-9df6-08221ea26ae3\") " pod="openshift-ingress/kube-auth-proxy-798674575b-cjnkz"
Sep 14 02:58:10.433805 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:58:10.433778    2564 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 14 02:58:10.433934 ip-10-0-141-4 kubenswrapper[2564]: E0914 02:58:10.433843    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6249c90-2e4c-4814-9df6-08221ea26ae3-tls-certs podName:f6249c90-2e4c-4814-9df6-08221ea26ae3 nodeName:}" failed. No retries permitted until 2026-09-14 02:58:10.933823062 +0000 UTC m=+336.314907253 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/f6249c90-2e4c-4814-9df6-08221ea26ae3-tls-certs") pod "kube-auth-proxy-798674575b-cjnkz" (UID: "f6249c90-2e4c-4814-9df6-08221ea26ae3") : secret "kube-auth-proxy-tls" not found
Sep 14 02:58:10.436857 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:10.436832    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f6249c90-2e4c-4814-9df6-08221ea26ae3-tmp\") pod \"kube-auth-proxy-798674575b-cjnkz\" (UID: \"f6249c90-2e4c-4814-9df6-08221ea26ae3\") " pod="openshift-ingress/kube-auth-proxy-798674575b-cjnkz"
Sep 14 02:58:10.463541 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:10.463509    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7qc2\" (UniqueName: \"kubernetes.io/projected/f6249c90-2e4c-4814-9df6-08221ea26ae3-kube-api-access-j7qc2\") pod \"kube-auth-proxy-798674575b-cjnkz\" (UID: \"f6249c90-2e4c-4814-9df6-08221ea26ae3\") " pod="openshift-ingress/kube-auth-proxy-798674575b-cjnkz"
Sep 14 02:58:10.659528 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:10.659489    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:58:10.938733 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:10.938695    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6249c90-2e4c-4814-9df6-08221ea26ae3-tls-certs\") pod \"kube-auth-proxy-798674575b-cjnkz\" (UID: \"f6249c90-2e4c-4814-9df6-08221ea26ae3\") " pod="openshift-ingress/kube-auth-proxy-798674575b-cjnkz"
Sep 14 02:58:10.941380 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:10.941355    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6249c90-2e4c-4814-9df6-08221ea26ae3-tls-certs\") pod \"kube-auth-proxy-798674575b-cjnkz\" (UID: \"f6249c90-2e4c-4814-9df6-08221ea26ae3\") " pod="openshift-ingress/kube-auth-proxy-798674575b-cjnkz"
Sep 14 02:58:11.150225 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:11.150193    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-798674575b-cjnkz"
Sep 14 02:58:12.025442 ip-10-0-141-4 kubenswrapper[2564]: W0914 02:58:12.025395    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6249c90_2e4c_4814_9df6_08221ea26ae3.slice/crio-114136ad09dd84a32460d8944cdc85b5add1fb66c06fe3bff0d02fec8df21b9a WatchSource:0}: Error finding container 114136ad09dd84a32460d8944cdc85b5add1fb66c06fe3bff0d02fec8df21b9a: Status 404 returned error can't find the container with id 114136ad09dd84a32460d8944cdc85b5add1fb66c06fe3bff0d02fec8df21b9a
Sep 14 02:58:12.025993 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:12.025958    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-798674575b-cjnkz"]
Sep 14 02:58:12.654096 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:12.654057    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-798674575b-cjnkz" event={"ID":"f6249c90-2e4c-4814-9df6-08221ea26ae3","Type":"ContainerStarted","Data":"114136ad09dd84a32460d8944cdc85b5add1fb66c06fe3bff0d02fec8df21b9a"}
Sep 14 02:58:12.656014 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:12.655964    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-59b9ccfbf8-2xdrk" event={"ID":"71f93d7d-55d4-43d5-bdb4-53e942788755","Type":"ContainerStarted","Data":"9bf77804957132b0982155c1c6e52025560384b07d9da18e9fcfabe91f932939"}
Sep 14 02:58:12.656149 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:12.656089    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-59b9ccfbf8-2xdrk"
Sep 14 02:58:12.717046 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:12.716986    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-59b9ccfbf8-2xdrk" podStartSLOduration=2.084594076 podStartE2EDuration="4.716963083s" totalImagesPullingTime="2.632369007s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:58:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:58:12.713591706 +0000 UTC m=+338.094675906" watchObservedRunningTime="2026-09-14 02:58:12.716963083 +0000 UTC m=+338.098047285"
Sep 14 02:58:13.268908 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:13.268842    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:58:13.415605 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:13.415522    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:58:14.668334 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:14.668285    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-798674575b-cjnkz" event={"ID":"f6249c90-2e4c-4814-9df6-08221ea26ae3","Type":"ContainerStarted","Data":"9c317af3925035bd7b62a24a79530b5680664d8c299e120603f9ba3ccbe3c0f1"}
Sep 14 02:58:14.693897 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:14.693848    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-798674575b-cjnkz" podStartSLOduration=2.199649439 podStartE2EDuration="4.69383187s" totalImagesPullingTime="2.494182431s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:58:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:58:14.691518052 +0000 UTC m=+340.072602251" watchObservedRunningTime="2026-09-14 02:58:14.69383187 +0000 UTC m=+340.074916068"
Sep 14 02:58:15.088820 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:15.088783    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:58:17.902035 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:17.901994    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:58:18.041491 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:18.041458    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:58:18.920779 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:18.920746    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:58:19.130499 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:19.130468    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:58:19.259704 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:19.259624    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:58:20.807782 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:20.807739    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:58:20.906251 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:20.906212    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:58:21.008178 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:21.008147    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:58:21.108133 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:21.108041    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:58:21.418584 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:21.418504    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:58:21.801443 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:21.801388    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:58:23.663145 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:23.663116    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-59b9ccfbf8-2xdrk"
Sep 14 02:58:29.329369 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:29.329287    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:58:29.626533 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:29.626453    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:58:32.543012 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:32.542980    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:58:33.008057 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:33.007969    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:58:33.322841 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:33.322807    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:58:33.673738 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:33.673641    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:58:33.975756 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:33.975654    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:58:40.767946 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:40.767902    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:58:40.870012 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:40.869978    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:58:40.969613 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:40.969573    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:58:58.902093 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:58:58.902055    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:59:21.749889 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:59:21.749853    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:59:21.853222 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:59:21.853176    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:59:46.452159 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:59:46.452119    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:59:56.177340 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:59:56.177308    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:59:56.980123 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:59:56.980081    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:59:59.688569 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:59:59.688473    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 02:59:59.804900 ip-10-0-141-4 kubenswrapper[2564]: I0914 02:59:59.804862    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:00:00.395355 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:00.395315    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:00:02.051103 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:02.051064    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:00:02.656937 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:02.656907    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:00:04.086742 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:04.086710    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-4dfs6"]
Sep 14 03:00:04.089641 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:04.089624    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-4dfs6"
Sep 14 03:00:04.094622 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:04.094600    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 14 03:00:04.094750 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:04.094634    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 14 03:00:04.094750 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:04.094605    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-49zlr\""
Sep 14 03:00:04.095005 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:04.094991    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 14 03:00:04.124319 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:04.124279    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-4dfs6"]
Sep 14 03:00:04.144712 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:04.144679    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfsnh\" (UniqueName: \"kubernetes.io/projected/bc928c23-7ce5-4a1a-a199-1a999e819964-kube-api-access-xfsnh\") pod \"dns-operator-controller-manager-7b84cd7c9c-4dfs6\" (UID: \"bc928c23-7ce5-4a1a-a199-1a999e819964\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-4dfs6"
Sep 14 03:00:04.246048 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:04.246008    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfsnh\" (UniqueName: \"kubernetes.io/projected/bc928c23-7ce5-4a1a-a199-1a999e819964-kube-api-access-xfsnh\") pod \"dns-operator-controller-manager-7b84cd7c9c-4dfs6\" (UID: \"bc928c23-7ce5-4a1a-a199-1a999e819964\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-4dfs6"
Sep 14 03:00:04.265674 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:04.265649    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfsnh\" (UniqueName: \"kubernetes.io/projected/bc928c23-7ce5-4a1a-a199-1a999e819964-kube-api-access-xfsnh\") pod \"dns-operator-controller-manager-7b84cd7c9c-4dfs6\" (UID: \"bc928c23-7ce5-4a1a-a199-1a999e819964\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-4dfs6"
Sep 14 03:00:04.347269 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:04.347184    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:00:04.399787 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:04.399759    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-4dfs6"
Sep 14 03:00:04.689459 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:04.689433    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-4dfs6"]
Sep 14 03:00:04.691748 ip-10-0-141-4 kubenswrapper[2564]: W0914 03:00:04.691719    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc928c23_7ce5_4a1a_a199_1a999e819964.slice/crio-8f60552b1fb17c3e374f16e91c245591477a2db4e1ad7cb95e11965d4e5110e5 WatchSource:0}: Error finding container 8f60552b1fb17c3e374f16e91c245591477a2db4e1ad7cb95e11965d4e5110e5: Status 404 returned error can't find the container with id 8f60552b1fb17c3e374f16e91c245591477a2db4e1ad7cb95e11965d4e5110e5
Sep 14 03:00:05.005718 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:05.005624    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:00:05.162212 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:05.162179    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-4dfs6" event={"ID":"bc928c23-7ce5-4a1a-a199-1a999e819964","Type":"ContainerStarted","Data":"8f60552b1fb17c3e374f16e91c245591477a2db4e1ad7cb95e11965d4e5110e5"}
Sep 14 03:00:05.290607 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:05.290573    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:00:05.449979 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:05.449943    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:00:05.566338 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:05.566238    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:00:05.662610 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:05.662575    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:00:08.480473 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:08.480433    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dvhkg"]
Sep 14 03:00:08.485002 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:08.484968    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dvhkg"
Sep 14 03:00:08.488130 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:08.488105    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-8sprk\""
Sep 14 03:00:08.500761 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:08.500739    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dvhkg"]
Sep 14 03:00:08.583616 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:08.583567    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/64c33235-b924-4047-82a3-cd87d17807a9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-dvhkg\" (UID: \"64c33235-b924-4047-82a3-cd87d17807a9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dvhkg"
Sep 14 03:00:08.583827 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:08.583638    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87rv8\" (UniqueName: \"kubernetes.io/projected/64c33235-b924-4047-82a3-cd87d17807a9-kube-api-access-87rv8\") pod \"kuadrant-operator-controller-manager-76f8f686bf-dvhkg\" (UID: \"64c33235-b924-4047-82a3-cd87d17807a9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dvhkg"
Sep 14 03:00:08.684893 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:08.684853    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/64c33235-b924-4047-82a3-cd87d17807a9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-dvhkg\" (UID: \"64c33235-b924-4047-82a3-cd87d17807a9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dvhkg"
Sep 14 03:00:08.685095 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:08.684935    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87rv8\" (UniqueName: \"kubernetes.io/projected/64c33235-b924-4047-82a3-cd87d17807a9-kube-api-access-87rv8\") pod \"kuadrant-operator-controller-manager-76f8f686bf-dvhkg\" (UID: \"64c33235-b924-4047-82a3-cd87d17807a9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dvhkg"
Sep 14 03:00:08.685353 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:08.685323    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/64c33235-b924-4047-82a3-cd87d17807a9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-dvhkg\" (UID: \"64c33235-b924-4047-82a3-cd87d17807a9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dvhkg"
Sep 14 03:00:08.697737 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:08.697640    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87rv8\" (UniqueName: \"kubernetes.io/projected/64c33235-b924-4047-82a3-cd87d17807a9-kube-api-access-87rv8\") pod \"kuadrant-operator-controller-manager-76f8f686bf-dvhkg\" (UID: \"64c33235-b924-4047-82a3-cd87d17807a9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dvhkg"
Sep 14 03:00:08.795016 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:08.794985    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dvhkg"
Sep 14 03:00:09.087281 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:09.087244    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dvhkg"]
Sep 14 03:00:09.183070 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:09.183032    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dvhkg" event={"ID":"64c33235-b924-4047-82a3-cd87d17807a9","Type":"ContainerStarted","Data":"83bc603f2a81ead6c31d88e6b9c555020b0c4c6e3e87e5cc2a9ab065f2b8a494"}
Sep 14 03:00:10.406940 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:10.406913    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:00:10.595229 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:10.595185    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:00:10.850005 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:10.849973    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:00:11.195772 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:11.195688    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-4dfs6" event={"ID":"bc928c23-7ce5-4a1a-a199-1a999e819964","Type":"ContainerStarted","Data":"3388a62ce718483a7118dcd5ab4988bea9da74b3ddea8b8e26db284b18181682"}
Sep 14 03:00:11.195942 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:11.195775    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-4dfs6"
Sep 14 03:00:11.231112 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:11.231050    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-4dfs6" podStartSLOduration=1.542081552 podStartE2EDuration="7.231028195s" totalImagesPullingTime="5.688946643s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:00:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:00:11.228472813 +0000 UTC m=+456.609557012" watchObservedRunningTime="2026-09-14 03:00:11.231028195 +0000 UTC m=+456.612112394"
Sep 14 03:00:14.217793 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:14.217760    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dvhkg" event={"ID":"64c33235-b924-4047-82a3-cd87d17807a9","Type":"ContainerStarted","Data":"373faa89b745a71932e527bea8a7ea039e6876bda58702e4ec0da61b4cb457ea"}
Sep 14 03:00:14.218215 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:14.217895    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dvhkg"
Sep 14 03:00:14.242530 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:14.242440    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dvhkg" podStartSLOduration=1.345799755 podStartE2EDuration="6.242421608s" totalImagesPullingTime="4.896621853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:00:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:00:14.239877944 +0000 UTC m=+459.620962145" watchObservedRunningTime="2026-09-14 03:00:14.242421608 +0000 UTC m=+459.623505808"
Sep 14 03:00:14.427543 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:14.427507    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:00:14.531417 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:14.531369    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:00:16.591791 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:16.591755    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:00:16.809416 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:16.809359    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:00:18.046055 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:18.046023    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:00:18.133646 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:18.133610    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:00:22.207999 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:22.207967    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-4dfs6"
Sep 14 03:00:23.812625 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:23.812592    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:00:23.914242 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:23.914203    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:00:24.397019 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:24.396988    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:00:24.498732 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:24.498704    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:00:24.985250 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:24.985213    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:00:25.071716 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:25.071688    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:00:25.224850 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:25.224808    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dvhkg"
Sep 14 03:00:27.192575 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:27.192534    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:00:27.625529 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:27.625494    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:00:27.859501 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:27.859469    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dvhkg"]
Sep 14 03:00:27.859687 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:27.859664    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dvhkg" podUID="64c33235-b924-4047-82a3-cd87d17807a9" containerName="manager" containerID="cri-o://373faa89b745a71932e527bea8a7ea039e6876bda58702e4ec0da61b4cb457ea" gracePeriod=2
Sep 14 03:00:27.872627 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:27.872601    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dvhkg"]
Sep 14 03:00:27.936628 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:27.936598    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gv7kr"]
Sep 14 03:00:27.937121 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:27.937095    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64c33235-b924-4047-82a3-cd87d17807a9" containerName="manager"
Sep 14 03:00:27.937121 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:27.937123    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="64c33235-b924-4047-82a3-cd87d17807a9" containerName="manager"
Sep 14 03:00:27.937276 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:27.937211    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="64c33235-b924-4047-82a3-cd87d17807a9" containerName="manager"
Sep 14 03:00:27.941158 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:27.941140    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gv7kr"
Sep 14 03:00:27.991217 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:27.991181    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gv7kr"]
Sep 14 03:00:28.053785 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:28.053755    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e8d238bc-b0bf-40f2-88df-484c884f6c0a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-gv7kr\" (UID: \"e8d238bc-b0bf-40f2-88df-484c884f6c0a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gv7kr"
Sep 14 03:00:28.053934 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:28.053798    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnt7g\" (UniqueName: \"kubernetes.io/projected/e8d238bc-b0bf-40f2-88df-484c884f6c0a-kube-api-access-nnt7g\") pod \"kuadrant-operator-controller-manager-76f8f686bf-gv7kr\" (UID: \"e8d238bc-b0bf-40f2-88df-484c884f6c0a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gv7kr"
Sep 14 03:00:28.094504 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:00:28.094466    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-dvhkg\" is forbidden: User \"system:node:ip-10-0-141-4.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-4.ec2.internal' and this object" podUID="64c33235-b924-4047-82a3-cd87d17807a9" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dvhkg"
Sep 14 03:00:28.109494 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:28.109473    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dvhkg"
Sep 14 03:00:28.112511 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:00:28.112481    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-dvhkg\" is forbidden: User \"system:node:ip-10-0-141-4.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-4.ec2.internal' and this object" podUID="64c33235-b924-4047-82a3-cd87d17807a9" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dvhkg"
Sep 14 03:00:28.155108 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:28.155031    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e8d238bc-b0bf-40f2-88df-484c884f6c0a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-gv7kr\" (UID: \"e8d238bc-b0bf-40f2-88df-484c884f6c0a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gv7kr"
Sep 14 03:00:28.155108 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:28.155068    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnt7g\" (UniqueName: \"kubernetes.io/projected/e8d238bc-b0bf-40f2-88df-484c884f6c0a-kube-api-access-nnt7g\") pod \"kuadrant-operator-controller-manager-76f8f686bf-gv7kr\" (UID: \"e8d238bc-b0bf-40f2-88df-484c884f6c0a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gv7kr"
Sep 14 03:00:28.155421 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:28.155385    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e8d238bc-b0bf-40f2-88df-484c884f6c0a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-gv7kr\" (UID: \"e8d238bc-b0bf-40f2-88df-484c884f6c0a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gv7kr"
Sep 14 03:00:28.168444 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:28.168420    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnt7g\" (UniqueName: \"kubernetes.io/projected/e8d238bc-b0bf-40f2-88df-484c884f6c0a-kube-api-access-nnt7g\") pod \"kuadrant-operator-controller-manager-76f8f686bf-gv7kr\" (UID: \"e8d238bc-b0bf-40f2-88df-484c884f6c0a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gv7kr"
Sep 14 03:00:28.252484 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:28.252455    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gv7kr"
Sep 14 03:00:28.255480 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:28.255457    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-87rv8\" (UniqueName: \"kubernetes.io/projected/64c33235-b924-4047-82a3-cd87d17807a9-kube-api-access-87rv8\") pod \"64c33235-b924-4047-82a3-cd87d17807a9\" (UID: \"64c33235-b924-4047-82a3-cd87d17807a9\") "
Sep 14 03:00:28.255558 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:28.255543    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/64c33235-b924-4047-82a3-cd87d17807a9-extensions-socket-volume\") pod \"64c33235-b924-4047-82a3-cd87d17807a9\" (UID: \"64c33235-b924-4047-82a3-cd87d17807a9\") "
Sep 14 03:00:28.256083 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:28.256050    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64c33235-b924-4047-82a3-cd87d17807a9-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "64c33235-b924-4047-82a3-cd87d17807a9" (UID: "64c33235-b924-4047-82a3-cd87d17807a9"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:00:28.257632 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:28.257611    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64c33235-b924-4047-82a3-cd87d17807a9-kube-api-access-87rv8" (OuterVolumeSpecName: "kube-api-access-87rv8") pod "64c33235-b924-4047-82a3-cd87d17807a9" (UID: "64c33235-b924-4047-82a3-cd87d17807a9"). InnerVolumeSpecName "kube-api-access-87rv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:00:28.283133 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:28.283103    2564 generic.go:356] "Generic (PLEG): container finished" podID="64c33235-b924-4047-82a3-cd87d17807a9" containerID="373faa89b745a71932e527bea8a7ea039e6876bda58702e4ec0da61b4cb457ea" exitCode=0
Sep 14 03:00:28.283238 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:28.283149    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dvhkg"
Sep 14 03:00:28.283238 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:28.283184    2564 scope.go:117] "RemoveContainer" containerID="373faa89b745a71932e527bea8a7ea039e6876bda58702e4ec0da61b4cb457ea"
Sep 14 03:00:28.286659 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:00:28.286622    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-dvhkg\" is forbidden: User \"system:node:ip-10-0-141-4.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-4.ec2.internal' and this object" podUID="64c33235-b924-4047-82a3-cd87d17807a9" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dvhkg"
Sep 14 03:00:28.292382 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:28.292334    2564 scope.go:117] "RemoveContainer" containerID="373faa89b745a71932e527bea8a7ea039e6876bda58702e4ec0da61b4cb457ea"
Sep 14 03:00:28.292964 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:00:28.292940    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"373faa89b745a71932e527bea8a7ea039e6876bda58702e4ec0da61b4cb457ea\": container with ID starting with 373faa89b745a71932e527bea8a7ea039e6876bda58702e4ec0da61b4cb457ea not found: ID does not exist" containerID="373faa89b745a71932e527bea8a7ea039e6876bda58702e4ec0da61b4cb457ea"
Sep 14 03:00:28.293061 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:28.292974    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"373faa89b745a71932e527bea8a7ea039e6876bda58702e4ec0da61b4cb457ea"} err="failed to get container status \"373faa89b745a71932e527bea8a7ea039e6876bda58702e4ec0da61b4cb457ea\": rpc error: code = NotFound desc = could not find container \"373faa89b745a71932e527bea8a7ea039e6876bda58702e4ec0da61b4cb457ea\": container with ID starting with 373faa89b745a71932e527bea8a7ea039e6876bda58702e4ec0da61b4cb457ea not found: ID does not exist"
Sep 14 03:00:28.296313 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:00:28.296283    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-dvhkg\" is forbidden: User \"system:node:ip-10-0-141-4.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-4.ec2.internal' and this object" podUID="64c33235-b924-4047-82a3-cd87d17807a9" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dvhkg"
Sep 14 03:00:28.357141 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:28.357109    2564 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/64c33235-b924-4047-82a3-cd87d17807a9-extensions-socket-volume\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:00:28.357141 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:28.357135    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87rv8\" (UniqueName: \"kubernetes.io/projected/64c33235-b924-4047-82a3-cd87d17807a9-kube-api-access-87rv8\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:00:28.547305 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:28.547278    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gv7kr"]
Sep 14 03:00:28.547701 ip-10-0-141-4 kubenswrapper[2564]: W0914 03:00:28.547670    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8d238bc_b0bf_40f2_88df_484c884f6c0a.slice/crio-25e99aa264fc369aec265fd5c8e39fe85f0817dd1a2d1c4518acd18783046e88 WatchSource:0}: Error finding container 25e99aa264fc369aec265fd5c8e39fe85f0817dd1a2d1c4518acd18783046e88: Status 404 returned error can't find the container with id 25e99aa264fc369aec265fd5c8e39fe85f0817dd1a2d1c4518acd18783046e88
Sep 14 03:00:29.097487 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:29.097455    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64c33235-b924-4047-82a3-cd87d17807a9" path="/var/lib/kubelet/pods/64c33235-b924-4047-82a3-cd87d17807a9/volumes"
Sep 14 03:00:29.290022 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:29.289985    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gv7kr" event={"ID":"e8d238bc-b0bf-40f2-88df-484c884f6c0a","Type":"ContainerStarted","Data":"a6e06f3c62aeff3ebf04f8749608654a2bddf901e9cf095045c6981549fa966f"}
Sep 14 03:00:29.290508 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:29.290034    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gv7kr" event={"ID":"e8d238bc-b0bf-40f2-88df-484c884f6c0a","Type":"ContainerStarted","Data":"25e99aa264fc369aec265fd5c8e39fe85f0817dd1a2d1c4518acd18783046e88"}
Sep 14 03:00:29.290508 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:29.290109    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gv7kr"
Sep 14 03:00:29.323325 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:29.323280    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gv7kr" podStartSLOduration=2.323267126 podStartE2EDuration="2.323267126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:00:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:00:29.322538721 +0000 UTC m=+474.703622919" watchObservedRunningTime="2026-09-14 03:00:29.323267126 +0000 UTC m=+474.704351325"
Sep 14 03:00:29.411488 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:29.411390    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:00:29.519787 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:29.519753    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:00:33.083192 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:33.083152    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:00:33.194328 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:33.194293    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:00:33.636904 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:33.636869    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:00:33.748663 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:33.748635    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:00:36.964254 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:36.964215    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:00:37.059200 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:37.059161    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:00:39.206148 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:39.206113    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:00:39.302472 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:39.302425    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:00:40.297014 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:40.296989    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gv7kr"
Sep 14 03:00:45.197448 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:45.197396    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:00:45.295670 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:45.295634    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:00:45.396214 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:45.396174    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:00:46.901935 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:46.901900    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gv7kr"]
Sep 14 03:00:46.902377 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:46.902121    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gv7kr" podUID="e8d238bc-b0bf-40f2-88df-484c884f6c0a" containerName="manager" containerID="cri-o://a6e06f3c62aeff3ebf04f8749608654a2bddf901e9cf095045c6981549fa966f" gracePeriod=10
Sep 14 03:00:47.150260 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:47.150237    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gv7kr"
Sep 14 03:00:47.220577 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:47.220501    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nnt7g\" (UniqueName: \"kubernetes.io/projected/e8d238bc-b0bf-40f2-88df-484c884f6c0a-kube-api-access-nnt7g\") pod \"e8d238bc-b0bf-40f2-88df-484c884f6c0a\" (UID: \"e8d238bc-b0bf-40f2-88df-484c884f6c0a\") "
Sep 14 03:00:47.220716 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:47.220589    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e8d238bc-b0bf-40f2-88df-484c884f6c0a-extensions-socket-volume\") pod \"e8d238bc-b0bf-40f2-88df-484c884f6c0a\" (UID: \"e8d238bc-b0bf-40f2-88df-484c884f6c0a\") "
Sep 14 03:00:47.220988 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:47.220969    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8d238bc-b0bf-40f2-88df-484c884f6c0a-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "e8d238bc-b0bf-40f2-88df-484c884f6c0a" (UID: "e8d238bc-b0bf-40f2-88df-484c884f6c0a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:00:47.222486 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:47.222457    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8d238bc-b0bf-40f2-88df-484c884f6c0a-kube-api-access-nnt7g" (OuterVolumeSpecName: "kube-api-access-nnt7g") pod "e8d238bc-b0bf-40f2-88df-484c884f6c0a" (UID: "e8d238bc-b0bf-40f2-88df-484c884f6c0a"). InnerVolumeSpecName "kube-api-access-nnt7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:00:47.321587 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:47.321558    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnt7g\" (UniqueName: \"kubernetes.io/projected/e8d238bc-b0bf-40f2-88df-484c884f6c0a-kube-api-access-nnt7g\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:00:47.321587 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:47.321584    2564 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e8d238bc-b0bf-40f2-88df-484c884f6c0a-extensions-socket-volume\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:00:47.373755 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:47.373724    2564 generic.go:356] "Generic (PLEG): container finished" podID="e8d238bc-b0bf-40f2-88df-484c884f6c0a" containerID="a6e06f3c62aeff3ebf04f8749608654a2bddf901e9cf095045c6981549fa966f" exitCode=0
Sep 14 03:00:47.373918 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:47.373763    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gv7kr"
Sep 14 03:00:47.373918 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:47.373785    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gv7kr" event={"ID":"e8d238bc-b0bf-40f2-88df-484c884f6c0a","Type":"ContainerDied","Data":"a6e06f3c62aeff3ebf04f8749608654a2bddf901e9cf095045c6981549fa966f"}
Sep 14 03:00:47.373918 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:47.373837    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gv7kr" event={"ID":"e8d238bc-b0bf-40f2-88df-484c884f6c0a","Type":"ContainerDied","Data":"25e99aa264fc369aec265fd5c8e39fe85f0817dd1a2d1c4518acd18783046e88"}
Sep 14 03:00:47.373918 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:47.373866    2564 scope.go:117] "RemoveContainer" containerID="a6e06f3c62aeff3ebf04f8749608654a2bddf901e9cf095045c6981549fa966f"
Sep 14 03:00:47.383300 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:47.383282    2564 scope.go:117] "RemoveContainer" containerID="a6e06f3c62aeff3ebf04f8749608654a2bddf901e9cf095045c6981549fa966f"
Sep 14 03:00:47.383578 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:00:47.383557    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6e06f3c62aeff3ebf04f8749608654a2bddf901e9cf095045c6981549fa966f\": container with ID starting with a6e06f3c62aeff3ebf04f8749608654a2bddf901e9cf095045c6981549fa966f not found: ID does not exist" containerID="a6e06f3c62aeff3ebf04f8749608654a2bddf901e9cf095045c6981549fa966f"
Sep 14 03:00:47.383638 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:47.383586    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6e06f3c62aeff3ebf04f8749608654a2bddf901e9cf095045c6981549fa966f"} err="failed to get container status \"a6e06f3c62aeff3ebf04f8749608654a2bddf901e9cf095045c6981549fa966f\": rpc error: code = NotFound desc = could not find container \"a6e06f3c62aeff3ebf04f8749608654a2bddf901e9cf095045c6981549fa966f\": container with ID starting with a6e06f3c62aeff3ebf04f8749608654a2bddf901e9cf095045c6981549fa966f not found: ID does not exist"
Sep 14 03:00:47.400635 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:47.400609    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gv7kr"]
Sep 14 03:00:47.409146 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:47.409123    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gv7kr"]
Sep 14 03:00:49.097176 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:49.097143    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8d238bc-b0bf-40f2-88df-484c884f6c0a" path="/var/lib/kubelet/pods/e8d238bc-b0bf-40f2-88df-484c884f6c0a/volumes"
Sep 14 03:00:56.266167 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:00:56.266126    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:01:03.352896 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:03.352859    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:01:03.483260 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:03.483221    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:01:05.297103 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:05.297062    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-t6qql"]
Sep 14 03:01:05.297634 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:05.297615    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8d238bc-b0bf-40f2-88df-484c884f6c0a" containerName="manager"
Sep 14 03:01:05.297710 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:05.297636    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8d238bc-b0bf-40f2-88df-484c884f6c0a" containerName="manager"
Sep 14 03:01:05.297772 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:05.297732    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8d238bc-b0bf-40f2-88df-484c884f6c0a" containerName="manager"
Sep 14 03:01:05.299530 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:05.299500    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:01:05.300303 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:05.300287    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-t6qql"
Sep 14 03:01:05.302755 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:05.302735    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-ztb9t\""
Sep 14 03:01:05.302831 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:05.302735    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 14 03:01:05.310912 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:05.310892    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-t6qql"]
Sep 14 03:01:05.382776 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:05.382748    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ed553fef-d072-4b71-8753-0086055f9192-config-file\") pod \"limitador-limitador-8588749977-t6qql\" (UID: \"ed553fef-d072-4b71-8753-0086055f9192\") " pod="kuadrant-system/limitador-limitador-8588749977-t6qql"
Sep 14 03:01:05.382776 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:05.382778    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmzqv\" (UniqueName: \"kubernetes.io/projected/ed553fef-d072-4b71-8753-0086055f9192-kube-api-access-bmzqv\") pod \"limitador-limitador-8588749977-t6qql\" (UID: \"ed553fef-d072-4b71-8753-0086055f9192\") " pod="kuadrant-system/limitador-limitador-8588749977-t6qql"
Sep 14 03:01:05.416344 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:05.416316    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-t6qql"]
Sep 14 03:01:05.482481 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:05.482451    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:01:05.483536 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:05.483498    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ed553fef-d072-4b71-8753-0086055f9192-config-file\") pod \"limitador-limitador-8588749977-t6qql\" (UID: \"ed553fef-d072-4b71-8753-0086055f9192\") " pod="kuadrant-system/limitador-limitador-8588749977-t6qql"
Sep 14 03:01:05.483536 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:05.483530    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmzqv\" (UniqueName: \"kubernetes.io/projected/ed553fef-d072-4b71-8753-0086055f9192-kube-api-access-bmzqv\") pod \"limitador-limitador-8588749977-t6qql\" (UID: \"ed553fef-d072-4b71-8753-0086055f9192\") " pod="kuadrant-system/limitador-limitador-8588749977-t6qql"
Sep 14 03:01:05.484104 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:05.484088    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ed553fef-d072-4b71-8753-0086055f9192-config-file\") pod \"limitador-limitador-8588749977-t6qql\" (UID: \"ed553fef-d072-4b71-8753-0086055f9192\") " pod="kuadrant-system/limitador-limitador-8588749977-t6qql"
Sep 14 03:01:05.497215 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:05.497187    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmzqv\" (UniqueName: \"kubernetes.io/projected/ed553fef-d072-4b71-8753-0086055f9192-kube-api-access-bmzqv\") pod \"limitador-limitador-8588749977-t6qql\" (UID: \"ed553fef-d072-4b71-8753-0086055f9192\") " pod="kuadrant-system/limitador-limitador-8588749977-t6qql"
Sep 14 03:01:05.591243 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:05.591151    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:01:05.612042 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:05.612012    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-t6qql"
Sep 14 03:01:05.885512 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:05.885489    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-t6qql"]
Sep 14 03:01:05.887886 ip-10-0-141-4 kubenswrapper[2564]: W0914 03:01:05.887854    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded553fef_d072_4b71_8753_0086055f9192.slice/crio-dc340317b84ee8653050581004ea381d4bb06cbd120bdb527658084e85cce8e1 WatchSource:0}: Error finding container dc340317b84ee8653050581004ea381d4bb06cbd120bdb527658084e85cce8e1: Status 404 returned error can't find the container with id dc340317b84ee8653050581004ea381d4bb06cbd120bdb527658084e85cce8e1
Sep 14 03:01:06.152194 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:06.152116    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-tjjms"]
Sep 14 03:01:06.156781 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:06.156765    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-tjjms"
Sep 14 03:01:06.159991 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:06.159969    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-9z7kg\""
Sep 14 03:01:06.163838 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:06.163677    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-tjjms"]
Sep 14 03:01:06.199240 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:06.199206    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:01:06.263503 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:06.263471    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:01:06.292153 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:06.292122    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hrwc\" (UniqueName: \"kubernetes.io/projected/772e4583-67db-4726-9b8e-4ee148d81720-kube-api-access-8hrwc\") pod \"authorino-7bcfd8f9cd-tjjms\" (UID: \"772e4583-67db-4726-9b8e-4ee148d81720\") " pod="kuadrant-system/authorino-7bcfd8f9cd-tjjms"
Sep 14 03:01:06.344224 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:06.344195    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-84tvq"]
Sep 14 03:01:06.347700 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:06.347683    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-84tvq"
Sep 14 03:01:06.357525 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:06.357481    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-84tvq"]
Sep 14 03:01:06.393237 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:06.393208    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hrwc\" (UniqueName: \"kubernetes.io/projected/772e4583-67db-4726-9b8e-4ee148d81720-kube-api-access-8hrwc\") pod \"authorino-7bcfd8f9cd-tjjms\" (UID: \"772e4583-67db-4726-9b8e-4ee148d81720\") " pod="kuadrant-system/authorino-7bcfd8f9cd-tjjms"
Sep 14 03:01:06.402968 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:06.402904    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hrwc\" (UniqueName: \"kubernetes.io/projected/772e4583-67db-4726-9b8e-4ee148d81720-kube-api-access-8hrwc\") pod \"authorino-7bcfd8f9cd-tjjms\" (UID: \"772e4583-67db-4726-9b8e-4ee148d81720\") " pod="kuadrant-system/authorino-7bcfd8f9cd-tjjms"
Sep 14 03:01:06.466883 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:06.466838    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-tjjms"
Sep 14 03:01:06.470780 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:06.470754    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-t6qql" event={"ID":"ed553fef-d072-4b71-8753-0086055f9192","Type":"ContainerStarted","Data":"dc340317b84ee8653050581004ea381d4bb06cbd120bdb527658084e85cce8e1"}
Sep 14 03:01:06.494315 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:06.494286    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfmh6\" (UniqueName: \"kubernetes.io/projected/b7802890-157b-49f0-818b-56e636ba8eb8-kube-api-access-vfmh6\") pod \"authorino-59c974bd48-84tvq\" (UID: \"b7802890-157b-49f0-818b-56e636ba8eb8\") " pod="kuadrant-system/authorino-59c974bd48-84tvq"
Sep 14 03:01:06.595224 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:06.595187    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfmh6\" (UniqueName: \"kubernetes.io/projected/b7802890-157b-49f0-818b-56e636ba8eb8-kube-api-access-vfmh6\") pod \"authorino-59c974bd48-84tvq\" (UID: \"b7802890-157b-49f0-818b-56e636ba8eb8\") " pod="kuadrant-system/authorino-59c974bd48-84tvq"
Sep 14 03:01:06.605670 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:06.605641    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfmh6\" (UniqueName: \"kubernetes.io/projected/b7802890-157b-49f0-818b-56e636ba8eb8-kube-api-access-vfmh6\") pod \"authorino-59c974bd48-84tvq\" (UID: \"b7802890-157b-49f0-818b-56e636ba8eb8\") " pod="kuadrant-system/authorino-59c974bd48-84tvq"
Sep 14 03:01:06.656693 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:06.656607    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-84tvq"
Sep 14 03:01:06.740086 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:06.740056    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-tjjms"]
Sep 14 03:01:06.742050 ip-10-0-141-4 kubenswrapper[2564]: W0914 03:01:06.742022    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod772e4583_67db_4726_9b8e_4ee148d81720.slice/crio-1826150d60b003bf4741ddeb00a1c4b7ee428d5513d597832ad00250eba6994f WatchSource:0}: Error finding container 1826150d60b003bf4741ddeb00a1c4b7ee428d5513d597832ad00250eba6994f: Status 404 returned error can't find the container with id 1826150d60b003bf4741ddeb00a1c4b7ee428d5513d597832ad00250eba6994f
Sep 14 03:01:06.940804 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:06.940682    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-84tvq"]
Sep 14 03:01:06.967474 ip-10-0-141-4 kubenswrapper[2564]: W0914 03:01:06.967437    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb7802890_157b_49f0_818b_56e636ba8eb8.slice/crio-d4b94254ddf9cd6e4256258b6384b808ab9ac11ed006ce19738cbb9683171117 WatchSource:0}: Error finding container d4b94254ddf9cd6e4256258b6384b808ab9ac11ed006ce19738cbb9683171117: Status 404 returned error can't find the container with id d4b94254ddf9cd6e4256258b6384b808ab9ac11ed006ce19738cbb9683171117
Sep 14 03:01:07.479674 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:07.479626    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-84tvq" event={"ID":"b7802890-157b-49f0-818b-56e636ba8eb8","Type":"ContainerStarted","Data":"d4b94254ddf9cd6e4256258b6384b808ab9ac11ed006ce19738cbb9683171117"}
Sep 14 03:01:07.481979 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:07.481944    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-t6qql" event={"ID":"ed553fef-d072-4b71-8753-0086055f9192","Type":"ContainerStarted","Data":"96166eb70169821f39b7ebe8f0551b29598312efea835f68474b796426b5103d"}
Sep 14 03:01:07.482771 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:07.482751    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-t6qql"
Sep 14 03:01:07.484147 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:07.484107    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-tjjms" event={"ID":"772e4583-67db-4726-9b8e-4ee148d81720","Type":"ContainerStarted","Data":"1826150d60b003bf4741ddeb00a1c4b7ee428d5513d597832ad00250eba6994f"}
Sep 14 03:01:07.502328 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:07.501554    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-t6qql" podStartSLOduration=1.3663691519999999 podStartE2EDuration="2.501534731s" totalImagesPullingTime="1.135165579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:01:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:01:07.50057018 +0000 UTC m=+512.881654401" watchObservedRunningTime="2026-09-14 03:01:07.501534731 +0000 UTC m=+512.882618931"
Sep 14 03:01:09.495116 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:09.495075    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-84tvq" event={"ID":"b7802890-157b-49f0-818b-56e636ba8eb8","Type":"ContainerStarted","Data":"83413888673a0152dd0aab47830bef1824bed78dd4f27c74e512c00d524ad5e0"}
Sep 14 03:01:09.496463 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:09.496440    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-tjjms" event={"ID":"772e4583-67db-4726-9b8e-4ee148d81720","Type":"ContainerStarted","Data":"b3c937cfad20d0550c333009464a7259c289a17e6b1c1ec6130f8df9c23b0f85"}
Sep 14 03:01:09.511700 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:09.511659    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-84tvq" podStartSLOduration=2.011123091 podStartE2EDuration="3.511645276s" totalImagesPullingTime="1.500522185s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:01:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:01:09.51072223 +0000 UTC m=+514.891806430" watchObservedRunningTime="2026-09-14 03:01:09.511645276 +0000 UTC m=+514.892729472"
Sep 14 03:01:09.529308 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:09.529252    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-tjjms" podStartSLOduration=1.808110052 podStartE2EDuration="3.529231405s" totalImagesPullingTime="1.721121353s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:01:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:01:09.526916118 +0000 UTC m=+514.908000328" watchObservedRunningTime="2026-09-14 03:01:09.529231405 +0000 UTC m=+514.910315605"
Sep 14 03:01:09.553166 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:09.553130    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-tjjms"]
Sep 14 03:01:11.506547 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:11.506510    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-tjjms" podUID="772e4583-67db-4726-9b8e-4ee148d81720" containerName="authorino" containerID="cri-o://b3c937cfad20d0550c333009464a7259c289a17e6b1c1ec6130f8df9c23b0f85" gracePeriod=30
Sep 14 03:01:11.755667 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:11.755636    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-tjjms"
Sep 14 03:01:11.846099 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:11.846057    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8hrwc\" (UniqueName: \"kubernetes.io/projected/772e4583-67db-4726-9b8e-4ee148d81720-kube-api-access-8hrwc\") pod \"772e4583-67db-4726-9b8e-4ee148d81720\" (UID: \"772e4583-67db-4726-9b8e-4ee148d81720\") "
Sep 14 03:01:11.848127 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:11.848100    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/772e4583-67db-4726-9b8e-4ee148d81720-kube-api-access-8hrwc" (OuterVolumeSpecName: "kube-api-access-8hrwc") pod "772e4583-67db-4726-9b8e-4ee148d81720" (UID: "772e4583-67db-4726-9b8e-4ee148d81720"). InnerVolumeSpecName "kube-api-access-8hrwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:01:11.947014 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:11.946974    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hrwc\" (UniqueName: \"kubernetes.io/projected/772e4583-67db-4726-9b8e-4ee148d81720-kube-api-access-8hrwc\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:01:12.512335 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:12.512303    2564 generic.go:356] "Generic (PLEG): container finished" podID="772e4583-67db-4726-9b8e-4ee148d81720" containerID="b3c937cfad20d0550c333009464a7259c289a17e6b1c1ec6130f8df9c23b0f85" exitCode=0
Sep 14 03:01:12.512775 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:12.512336    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-tjjms" event={"ID":"772e4583-67db-4726-9b8e-4ee148d81720","Type":"ContainerDied","Data":"b3c937cfad20d0550c333009464a7259c289a17e6b1c1ec6130f8df9c23b0f85"}
Sep 14 03:01:12.512775 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:12.512385    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-tjjms" event={"ID":"772e4583-67db-4726-9b8e-4ee148d81720","Type":"ContainerDied","Data":"1826150d60b003bf4741ddeb00a1c4b7ee428d5513d597832ad00250eba6994f"}
Sep 14 03:01:12.512775 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:12.512423    2564 scope.go:117] "RemoveContainer" containerID="b3c937cfad20d0550c333009464a7259c289a17e6b1c1ec6130f8df9c23b0f85"
Sep 14 03:01:12.512775 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:12.512346    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-tjjms"
Sep 14 03:01:12.521571 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:12.521556    2564 scope.go:117] "RemoveContainer" containerID="b3c937cfad20d0550c333009464a7259c289a17e6b1c1ec6130f8df9c23b0f85"
Sep 14 03:01:12.521806 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:01:12.521788    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3c937cfad20d0550c333009464a7259c289a17e6b1c1ec6130f8df9c23b0f85\": container with ID starting with b3c937cfad20d0550c333009464a7259c289a17e6b1c1ec6130f8df9c23b0f85 not found: ID does not exist" containerID="b3c937cfad20d0550c333009464a7259c289a17e6b1c1ec6130f8df9c23b0f85"
Sep 14 03:01:12.521858 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:12.521814    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3c937cfad20d0550c333009464a7259c289a17e6b1c1ec6130f8df9c23b0f85"} err="failed to get container status \"b3c937cfad20d0550c333009464a7259c289a17e6b1c1ec6130f8df9c23b0f85\": rpc error: code = NotFound desc = could not find container \"b3c937cfad20d0550c333009464a7259c289a17e6b1c1ec6130f8df9c23b0f85\": container with ID starting with b3c937cfad20d0550c333009464a7259c289a17e6b1c1ec6130f8df9c23b0f85 not found: ID does not exist"
Sep 14 03:01:12.536274 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:12.536248    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-tjjms"]
Sep 14 03:01:12.538809 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:12.538792    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-tjjms"]
Sep 14 03:01:13.098532 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:13.098503    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="772e4583-67db-4726-9b8e-4ee148d81720" path="/var/lib/kubelet/pods/772e4583-67db-4726-9b8e-4ee148d81720/volumes"
Sep 14 03:01:19.373103 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:19.373068    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-t6qql"]
Sep 14 03:01:19.373671 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:19.373278    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-t6qql" podUID="ed553fef-d072-4b71-8753-0086055f9192" containerName="limitador" containerID="cri-o://96166eb70169821f39b7ebe8f0551b29598312efea835f68474b796426b5103d" gracePeriod=30
Sep 14 03:01:19.374082 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:19.374023    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-t6qql"
Sep 14 03:01:19.922469 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:19.922446    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-t6qql"
Sep 14 03:01:20.016600 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:20.016524    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ed553fef-d072-4b71-8753-0086055f9192-config-file\") pod \"ed553fef-d072-4b71-8753-0086055f9192\" (UID: \"ed553fef-d072-4b71-8753-0086055f9192\") "
Sep 14 03:01:20.016600 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:20.016572    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bmzqv\" (UniqueName: \"kubernetes.io/projected/ed553fef-d072-4b71-8753-0086055f9192-kube-api-access-bmzqv\") pod \"ed553fef-d072-4b71-8753-0086055f9192\" (UID: \"ed553fef-d072-4b71-8753-0086055f9192\") "
Sep 14 03:01:20.016853 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:20.016833    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed553fef-d072-4b71-8753-0086055f9192-config-file" (OuterVolumeSpecName: "config-file") pod "ed553fef-d072-4b71-8753-0086055f9192" (UID: "ed553fef-d072-4b71-8753-0086055f9192"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 03:01:20.018558 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:20.018537    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed553fef-d072-4b71-8753-0086055f9192-kube-api-access-bmzqv" (OuterVolumeSpecName: "kube-api-access-bmzqv") pod "ed553fef-d072-4b71-8753-0086055f9192" (UID: "ed553fef-d072-4b71-8753-0086055f9192"). InnerVolumeSpecName "kube-api-access-bmzqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:01:20.117937 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:20.117909    2564 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ed553fef-d072-4b71-8753-0086055f9192-config-file\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:01:20.117937 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:20.117933    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmzqv\" (UniqueName: \"kubernetes.io/projected/ed553fef-d072-4b71-8753-0086055f9192-kube-api-access-bmzqv\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:01:20.551294 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:20.551263    2564 generic.go:356] "Generic (PLEG): container finished" podID="ed553fef-d072-4b71-8753-0086055f9192" containerID="96166eb70169821f39b7ebe8f0551b29598312efea835f68474b796426b5103d" exitCode=0
Sep 14 03:01:20.551733 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:20.551308    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-t6qql"
Sep 14 03:01:20.551733 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:20.551334    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-t6qql" event={"ID":"ed553fef-d072-4b71-8753-0086055f9192","Type":"ContainerDied","Data":"96166eb70169821f39b7ebe8f0551b29598312efea835f68474b796426b5103d"}
Sep 14 03:01:20.551733 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:20.551373    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-t6qql" event={"ID":"ed553fef-d072-4b71-8753-0086055f9192","Type":"ContainerDied","Data":"dc340317b84ee8653050581004ea381d4bb06cbd120bdb527658084e85cce8e1"}
Sep 14 03:01:20.551733 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:20.551393    2564 scope.go:117] "RemoveContainer" containerID="96166eb70169821f39b7ebe8f0551b29598312efea835f68474b796426b5103d"
Sep 14 03:01:20.560211 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:20.560193    2564 scope.go:117] "RemoveContainer" containerID="96166eb70169821f39b7ebe8f0551b29598312efea835f68474b796426b5103d"
Sep 14 03:01:20.560491 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:01:20.560468    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96166eb70169821f39b7ebe8f0551b29598312efea835f68474b796426b5103d\": container with ID starting with 96166eb70169821f39b7ebe8f0551b29598312efea835f68474b796426b5103d not found: ID does not exist" containerID="96166eb70169821f39b7ebe8f0551b29598312efea835f68474b796426b5103d"
Sep 14 03:01:20.560593 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:20.560495    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96166eb70169821f39b7ebe8f0551b29598312efea835f68474b796426b5103d"} err="failed to get container status \"96166eb70169821f39b7ebe8f0551b29598312efea835f68474b796426b5103d\": rpc error: code = NotFound desc = could not find container \"96166eb70169821f39b7ebe8f0551b29598312efea835f68474b796426b5103d\": container with ID starting with 96166eb70169821f39b7ebe8f0551b29598312efea835f68474b796426b5103d not found: ID does not exist"
Sep 14 03:01:20.574362 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:20.574338    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-t6qql"]
Sep 14 03:01:20.578696 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:20.578677    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-t6qql"]
Sep 14 03:01:21.097782 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:21.097749    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed553fef-d072-4b71-8753-0086055f9192" path="/var/lib/kubelet/pods/ed553fef-d072-4b71-8753-0086055f9192/volumes"
Sep 14 03:01:21.444899 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:21.444809    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:01:21.541372 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:21.541337    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:01:21.879303 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:21.879274    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-vz9hc"]
Sep 14 03:01:21.879678 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:21.879661    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed553fef-d072-4b71-8753-0086055f9192" containerName="limitador"
Sep 14 03:01:21.879678 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:21.879674    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed553fef-d072-4b71-8753-0086055f9192" containerName="limitador"
Sep 14 03:01:21.879753 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:21.879689    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="772e4583-67db-4726-9b8e-4ee148d81720" containerName="authorino"
Sep 14 03:01:21.879753 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:21.879694    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="772e4583-67db-4726-9b8e-4ee148d81720" containerName="authorino"
Sep 14 03:01:21.879817 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:21.879765    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="772e4583-67db-4726-9b8e-4ee148d81720" containerName="authorino"
Sep 14 03:01:21.879817 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:21.879775    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed553fef-d072-4b71-8753-0086055f9192" containerName="limitador"
Sep 14 03:01:21.884135 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:21.884118    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vz9hc"
Sep 14 03:01:21.887542 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:21.887520    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 14 03:01:21.887920 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:21.887905    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 14 03:01:21.887979 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:21.887962    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 14 03:01:21.888942 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:21.888887    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-frczh\""
Sep 14 03:01:21.892023 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:21.891994    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-vz9hc"]
Sep 14 03:01:21.928227 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:21.928192    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:01:22.033986 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:22.033953    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6656\" (UniqueName: \"kubernetes.io/projected/8c3a6db4-6599-4e2f-8af6-d0c937edc94b-kube-api-access-k6656\") pod \"postgres-69fb4594cf-vz9hc\" (UID: \"8c3a6db4-6599-4e2f-8af6-d0c937edc94b\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-vz9hc"
Sep 14 03:01:22.034143 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:22.033991    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8c3a6db4-6599-4e2f-8af6-d0c937edc94b-data\") pod \"postgres-69fb4594cf-vz9hc\" (UID: \"8c3a6db4-6599-4e2f-8af6-d0c937edc94b\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-vz9hc"
Sep 14 03:01:22.135056 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:22.134974    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8c3a6db4-6599-4e2f-8af6-d0c937edc94b-data\") pod \"postgres-69fb4594cf-vz9hc\" (UID: \"8c3a6db4-6599-4e2f-8af6-d0c937edc94b\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-vz9hc"
Sep 14 03:01:22.135187 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:22.135072    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6656\" (UniqueName: \"kubernetes.io/projected/8c3a6db4-6599-4e2f-8af6-d0c937edc94b-kube-api-access-k6656\") pod \"postgres-69fb4594cf-vz9hc\" (UID: \"8c3a6db4-6599-4e2f-8af6-d0c937edc94b\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-vz9hc"
Sep 14 03:01:22.135354 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:22.135335    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8c3a6db4-6599-4e2f-8af6-d0c937edc94b-data\") pod \"postgres-69fb4594cf-vz9hc\" (UID: \"8c3a6db4-6599-4e2f-8af6-d0c937edc94b\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-vz9hc"
Sep 14 03:01:22.144840 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:22.144806    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6656\" (UniqueName: \"kubernetes.io/projected/8c3a6db4-6599-4e2f-8af6-d0c937edc94b-kube-api-access-k6656\") pod \"postgres-69fb4594cf-vz9hc\" (UID: \"8c3a6db4-6599-4e2f-8af6-d0c937edc94b\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-vz9hc"
Sep 14 03:01:22.196538 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:22.196501    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vz9hc"
Sep 14 03:01:22.468005 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:22.467982    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-vz9hc"]
Sep 14 03:01:22.469311 ip-10-0-141-4 kubenswrapper[2564]: W0914 03:01:22.469283    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c3a6db4_6599_4e2f_8af6_d0c937edc94b.slice/crio-20c1c1f81c88820015ac09e76217d7c3736b38640a8169ae6835654ac6293199 WatchSource:0}: Error finding container 20c1c1f81c88820015ac09e76217d7c3736b38640a8169ae6835654ac6293199: Status 404 returned error can't find the container with id 20c1c1f81c88820015ac09e76217d7c3736b38640a8169ae6835654ac6293199
Sep 14 03:01:22.562354 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:22.562318    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vz9hc" event={"ID":"8c3a6db4-6599-4e2f-8af6-d0c937edc94b","Type":"ContainerStarted","Data":"20c1c1f81c88820015ac09e76217d7c3736b38640a8169ae6835654ac6293199"}
Sep 14 03:01:27.596268 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:27.596188    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vz9hc" event={"ID":"8c3a6db4-6599-4e2f-8af6-d0c937edc94b","Type":"ContainerStarted","Data":"2aab13fb24bfc29dec4c90efc5e379b7898b5ae278e8382724728419d802ec75"}
Sep 14 03:01:27.596629 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:27.596292    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vz9hc"
Sep 14 03:01:27.615777 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:27.615725    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vz9hc" podStartSLOduration=1.783834754 podStartE2EDuration="6.615711088s" totalImagesPullingTime="4.831876334s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:01:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:01:27.615206022 +0000 UTC m=+532.996290220" watchObservedRunningTime="2026-09-14 03:01:27.615711088 +0000 UTC m=+532.996795310"
Sep 14 03:01:33.628420 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:33.628384    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vz9hc"
Sep 14 03:01:33.964469 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:33.964374    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:01:34.194889 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:34.194855    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-ftxdn"]
Sep 14 03:01:34.198478 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:34.198458    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-ftxdn"
Sep 14 03:01:34.206514 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:34.206490    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-ftxdn"]
Sep 14 03:01:34.246438 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:34.246344    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gstb2\" (UniqueName: \"kubernetes.io/projected/1ea6a757-4657-4439-8346-818ea0a91c52-kube-api-access-gstb2\") pod \"authorino-557f876f69-ftxdn\" (UID: \"1ea6a757-4657-4439-8346-818ea0a91c52\") " pod="kuadrant-system/authorino-557f876f69-ftxdn"
Sep 14 03:01:34.319739 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:34.319704    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:01:34.347022 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:34.346997    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gstb2\" (UniqueName: \"kubernetes.io/projected/1ea6a757-4657-4439-8346-818ea0a91c52-kube-api-access-gstb2\") pod \"authorino-557f876f69-ftxdn\" (UID: \"1ea6a757-4657-4439-8346-818ea0a91c52\") " pod="kuadrant-system/authorino-557f876f69-ftxdn"
Sep 14 03:01:34.356107 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:34.356087    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gstb2\" (UniqueName: \"kubernetes.io/projected/1ea6a757-4657-4439-8346-818ea0a91c52-kube-api-access-gstb2\") pod \"authorino-557f876f69-ftxdn\" (UID: \"1ea6a757-4657-4439-8346-818ea0a91c52\") " pod="kuadrant-system/authorino-557f876f69-ftxdn"
Sep 14 03:01:34.426729 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:34.426697    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-ftxdn"]
Sep 14 03:01:34.427008 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:34.426993    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-ftxdn"
Sep 14 03:01:34.462531 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:34.462496    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-676db85fb5-98j6g"]
Sep 14 03:01:34.471376 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:34.471346    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-676db85fb5-98j6g"
Sep 14 03:01:34.473915 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:34.473877    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-676db85fb5-98j6g"]
Sep 14 03:01:34.512834 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:34.512770    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-676db85fb5-98j6g"]
Sep 14 03:01:34.513101 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:01:34.513081    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-j84t5], unattached volumes=[], failed to process volumes=[kube-api-access-j84t5]: context canceled" pod="kuadrant-system/authorino-676db85fb5-98j6g" podUID="c3a1bedd-dddf-4441-b545-c979e478d301"
Sep 14 03:01:34.549261 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:34.549230    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j84t5\" (UniqueName: \"kubernetes.io/projected/c3a1bedd-dddf-4441-b545-c979e478d301-kube-api-access-j84t5\") pod \"authorino-676db85fb5-98j6g\" (UID: \"c3a1bedd-dddf-4441-b545-c979e478d301\") " pod="kuadrant-system/authorino-676db85fb5-98j6g"
Sep 14 03:01:34.627071 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:34.627037    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-676db85fb5-98j6g"
Sep 14 03:01:34.636271 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:34.636248    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-676db85fb5-98j6g"
Sep 14 03:01:34.650340 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:34.650311    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j84t5\" (UniqueName: \"kubernetes.io/projected/c3a1bedd-dddf-4441-b545-c979e478d301-kube-api-access-j84t5\") pod \"authorino-676db85fb5-98j6g\" (UID: \"c3a1bedd-dddf-4441-b545-c979e478d301\") " pod="kuadrant-system/authorino-676db85fb5-98j6g"
Sep 14 03:01:34.660765 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:34.660738    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j84t5\" (UniqueName: \"kubernetes.io/projected/c3a1bedd-dddf-4441-b545-c979e478d301-kube-api-access-j84t5\") pod \"authorino-676db85fb5-98j6g\" (UID: \"c3a1bedd-dddf-4441-b545-c979e478d301\") " pod="kuadrant-system/authorino-676db85fb5-98j6g"
Sep 14 03:01:34.713023 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:34.713002    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-ftxdn"]
Sep 14 03:01:34.715081 ip-10-0-141-4 kubenswrapper[2564]: W0914 03:01:34.715055    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1ea6a757_4657_4439_8346_818ea0a91c52.slice/crio-1b28f55cb7747b494b662a04736b7c0d043feac68ef10e0a9debf759ffed13bf WatchSource:0}: Error finding container 1b28f55cb7747b494b662a04736b7c0d043feac68ef10e0a9debf759ffed13bf: Status 404 returned error can't find the container with id 1b28f55cb7747b494b662a04736b7c0d043feac68ef10e0a9debf759ffed13bf
Sep 14 03:01:34.751379 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:34.751360    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j84t5\" (UniqueName: \"kubernetes.io/projected/c3a1bedd-dddf-4441-b545-c979e478d301-kube-api-access-j84t5\") pod \"c3a1bedd-dddf-4441-b545-c979e478d301\" (UID: \"c3a1bedd-dddf-4441-b545-c979e478d301\") "
Sep 14 03:01:34.753284 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:34.753263    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3a1bedd-dddf-4441-b545-c979e478d301-kube-api-access-j84t5" (OuterVolumeSpecName: "kube-api-access-j84t5") pod "c3a1bedd-dddf-4441-b545-c979e478d301" (UID: "c3a1bedd-dddf-4441-b545-c979e478d301"). InnerVolumeSpecName "kube-api-access-j84t5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:01:34.851947 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:34.851916    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j84t5\" (UniqueName: \"kubernetes.io/projected/c3a1bedd-dddf-4441-b545-c979e478d301-kube-api-access-j84t5\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:01:35.633775 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:35.633696    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-676db85fb5-98j6g"
Sep 14 03:01:35.633954 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:35.633766    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-557f876f69-ftxdn" podUID="1ea6a757-4657-4439-8346-818ea0a91c52" containerName="authorino" containerID="cri-o://baed7dee53568270a5caef8688c387cc30c2e0c95a78a725286934cc171828d1" gracePeriod=30
Sep 14 03:01:35.633954 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:35.633691    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-ftxdn" event={"ID":"1ea6a757-4657-4439-8346-818ea0a91c52","Type":"ContainerStarted","Data":"baed7dee53568270a5caef8688c387cc30c2e0c95a78a725286934cc171828d1"}
Sep 14 03:01:35.633954 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:35.633825    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-ftxdn" event={"ID":"1ea6a757-4657-4439-8346-818ea0a91c52","Type":"ContainerStarted","Data":"1b28f55cb7747b494b662a04736b7c0d043feac68ef10e0a9debf759ffed13bf"}
Sep 14 03:01:35.654822 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:35.654774    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-557f876f69-ftxdn" podStartSLOduration=0.986785496 podStartE2EDuration="1.654758246s" totalImagesPullingTime="667.97275ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:01:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:01:35.653872891 +0000 UTC m=+541.034957090" watchObservedRunningTime="2026-09-14 03:01:35.654758246 +0000 UTC m=+541.035842497"
Sep 14 03:01:35.680671 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:35.680646    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-676db85fb5-98j6g"]
Sep 14 03:01:35.683891 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:35.683866    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-676db85fb5-98j6g"]
Sep 14 03:01:35.903669 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:35.903602    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-ftxdn"
Sep 14 03:01:35.962086 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:35.962053    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gstb2\" (UniqueName: \"kubernetes.io/projected/1ea6a757-4657-4439-8346-818ea0a91c52-kube-api-access-gstb2\") pod \"1ea6a757-4657-4439-8346-818ea0a91c52\" (UID: \"1ea6a757-4657-4439-8346-818ea0a91c52\") "
Sep 14 03:01:35.964070 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:35.964050    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ea6a757-4657-4439-8346-818ea0a91c52-kube-api-access-gstb2" (OuterVolumeSpecName: "kube-api-access-gstb2") pod "1ea6a757-4657-4439-8346-818ea0a91c52" (UID: "1ea6a757-4657-4439-8346-818ea0a91c52"). InnerVolumeSpecName "kube-api-access-gstb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:01:36.062788 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:36.062760    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gstb2\" (UniqueName: \"kubernetes.io/projected/1ea6a757-4657-4439-8346-818ea0a91c52-kube-api-access-gstb2\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:01:36.639543 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:36.639455    2564 generic.go:356] "Generic (PLEG): container finished" podID="1ea6a757-4657-4439-8346-818ea0a91c52" containerID="baed7dee53568270a5caef8688c387cc30c2e0c95a78a725286934cc171828d1" exitCode=0
Sep 14 03:01:36.639543 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:36.639502    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-ftxdn"
Sep 14 03:01:36.639543 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:36.639529    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-ftxdn" event={"ID":"1ea6a757-4657-4439-8346-818ea0a91c52","Type":"ContainerDied","Data":"baed7dee53568270a5caef8688c387cc30c2e0c95a78a725286934cc171828d1"}
Sep 14 03:01:36.639768 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:36.639559    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-ftxdn" event={"ID":"1ea6a757-4657-4439-8346-818ea0a91c52","Type":"ContainerDied","Data":"1b28f55cb7747b494b662a04736b7c0d043feac68ef10e0a9debf759ffed13bf"}
Sep 14 03:01:36.639768 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:36.639574    2564 scope.go:117] "RemoveContainer" containerID="baed7dee53568270a5caef8688c387cc30c2e0c95a78a725286934cc171828d1"
Sep 14 03:01:36.648127 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:36.648108    2564 scope.go:117] "RemoveContainer" containerID="baed7dee53568270a5caef8688c387cc30c2e0c95a78a725286934cc171828d1"
Sep 14 03:01:36.648392 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:01:36.648375    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baed7dee53568270a5caef8688c387cc30c2e0c95a78a725286934cc171828d1\": container with ID starting with baed7dee53568270a5caef8688c387cc30c2e0c95a78a725286934cc171828d1 not found: ID does not exist" containerID="baed7dee53568270a5caef8688c387cc30c2e0c95a78a725286934cc171828d1"
Sep 14 03:01:36.648466 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:36.648419    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baed7dee53568270a5caef8688c387cc30c2e0c95a78a725286934cc171828d1"} err="failed to get container status \"baed7dee53568270a5caef8688c387cc30c2e0c95a78a725286934cc171828d1\": rpc error: code = NotFound desc = could not find container \"baed7dee53568270a5caef8688c387cc30c2e0c95a78a725286934cc171828d1\": container with ID starting with baed7dee53568270a5caef8688c387cc30c2e0c95a78a725286934cc171828d1 not found: ID does not exist"
Sep 14 03:01:36.669027 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:36.668995    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-ftxdn"]
Sep 14 03:01:36.687490 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:36.687463    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-ftxdn"]
Sep 14 03:01:37.097258 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:37.097227    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ea6a757-4657-4439-8346-818ea0a91c52" path="/var/lib/kubelet/pods/1ea6a757-4657-4439-8346-818ea0a91c52/volumes"
Sep 14 03:01:37.097583 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:37.097569    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3a1bedd-dddf-4441-b545-c979e478d301" path="/var/lib/kubelet/pods/c3a1bedd-dddf-4441-b545-c979e478d301/volumes"
Sep 14 03:01:37.148685 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:37.148647    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-84tvq"]
Sep 14 03:01:37.148879 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:37.148854    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-84tvq" podUID="b7802890-157b-49f0-818b-56e636ba8eb8" containerName="authorino" containerID="cri-o://83413888673a0152dd0aab47830bef1824bed78dd4f27c74e512c00d524ad5e0" gracePeriod=30
Sep 14 03:01:37.386173 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:37.386144    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-84tvq"
Sep 14 03:01:37.475791 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:37.475758    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vfmh6\" (UniqueName: \"kubernetes.io/projected/b7802890-157b-49f0-818b-56e636ba8eb8-kube-api-access-vfmh6\") pod \"b7802890-157b-49f0-818b-56e636ba8eb8\" (UID: \"b7802890-157b-49f0-818b-56e636ba8eb8\") "
Sep 14 03:01:37.477808 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:37.477780    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7802890-157b-49f0-818b-56e636ba8eb8-kube-api-access-vfmh6" (OuterVolumeSpecName: "kube-api-access-vfmh6") pod "b7802890-157b-49f0-818b-56e636ba8eb8" (UID: "b7802890-157b-49f0-818b-56e636ba8eb8"). InnerVolumeSpecName "kube-api-access-vfmh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:01:37.577395 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:37.577362    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfmh6\" (UniqueName: \"kubernetes.io/projected/b7802890-157b-49f0-818b-56e636ba8eb8-kube-api-access-vfmh6\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:01:37.646681 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:37.646604    2564 generic.go:356] "Generic (PLEG): container finished" podID="b7802890-157b-49f0-818b-56e636ba8eb8" containerID="83413888673a0152dd0aab47830bef1824bed78dd4f27c74e512c00d524ad5e0" exitCode=0
Sep 14 03:01:37.646681 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:37.646645    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-84tvq"
Sep 14 03:01:37.646681 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:37.646668    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-84tvq" event={"ID":"b7802890-157b-49f0-818b-56e636ba8eb8","Type":"ContainerDied","Data":"83413888673a0152dd0aab47830bef1824bed78dd4f27c74e512c00d524ad5e0"}
Sep 14 03:01:37.646941 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:37.646698    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-84tvq" event={"ID":"b7802890-157b-49f0-818b-56e636ba8eb8","Type":"ContainerDied","Data":"d4b94254ddf9cd6e4256258b6384b808ab9ac11ed006ce19738cbb9683171117"}
Sep 14 03:01:37.646941 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:37.646715    2564 scope.go:117] "RemoveContainer" containerID="83413888673a0152dd0aab47830bef1824bed78dd4f27c74e512c00d524ad5e0"
Sep 14 03:01:37.655289 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:37.655273    2564 scope.go:117] "RemoveContainer" containerID="83413888673a0152dd0aab47830bef1824bed78dd4f27c74e512c00d524ad5e0"
Sep 14 03:01:37.655559 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:01:37.655543    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83413888673a0152dd0aab47830bef1824bed78dd4f27c74e512c00d524ad5e0\": container with ID starting with 83413888673a0152dd0aab47830bef1824bed78dd4f27c74e512c00d524ad5e0 not found: ID does not exist" containerID="83413888673a0152dd0aab47830bef1824bed78dd4f27c74e512c00d524ad5e0"
Sep 14 03:01:37.655615 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:37.655567    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83413888673a0152dd0aab47830bef1824bed78dd4f27c74e512c00d524ad5e0"} err="failed to get container status \"83413888673a0152dd0aab47830bef1824bed78dd4f27c74e512c00d524ad5e0\": rpc error: code = NotFound desc = could not find container \"83413888673a0152dd0aab47830bef1824bed78dd4f27c74e512c00d524ad5e0\": container with ID starting with 83413888673a0152dd0aab47830bef1824bed78dd4f27c74e512c00d524ad5e0 not found: ID does not exist"
Sep 14 03:01:37.669675 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:37.669650    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-84tvq"]
Sep 14 03:01:37.673685 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:37.673659    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-84tvq"]
Sep 14 03:01:39.098024 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:39.097988    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7802890-157b-49f0-818b-56e636ba8eb8" path="/var/lib/kubelet/pods/b7802890-157b-49f0-818b-56e636ba8eb8/volumes"
Sep 14 03:01:42.034025 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:42.033993    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:01:42.062322 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:42.062291    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-d587dcdd5-bsf8t"]
Sep 14 03:01:42.062686 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:42.062672    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ea6a757-4657-4439-8346-818ea0a91c52" containerName="authorino"
Sep 14 03:01:42.062737 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:42.062691    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ea6a757-4657-4439-8346-818ea0a91c52" containerName="authorino"
Sep 14 03:01:42.062737 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:42.062706    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7802890-157b-49f0-818b-56e636ba8eb8" containerName="authorino"
Sep 14 03:01:42.062737 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:42.062713    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7802890-157b-49f0-818b-56e636ba8eb8" containerName="authorino"
Sep 14 03:01:42.062824 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:42.062787    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ea6a757-4657-4439-8346-818ea0a91c52" containerName="authorino"
Sep 14 03:01:42.062824 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:42.062797    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7802890-157b-49f0-818b-56e636ba8eb8" containerName="authorino"
Sep 14 03:01:42.065506 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:42.065488    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d587dcdd5-bsf8t"
Sep 14 03:01:42.068595 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:42.068576    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 14 03:01:42.068701 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:42.068605    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 14 03:01:42.068701 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:42.068665    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-m8df5\""
Sep 14 03:01:42.069563 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:42.069548    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 14 03:01:42.081848 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:42.081826    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d587dcdd5-bsf8t"]
Sep 14 03:01:42.114123 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:42.114091    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/57c03020-b4de-419f-9da8-ecc972abb9ae-webhook-cert\") pod \"maas-controller-d587dcdd5-bsf8t\" (UID: \"57c03020-b4de-419f-9da8-ecc972abb9ae\") " pod="opendatahub/maas-controller-d587dcdd5-bsf8t"
Sep 14 03:01:42.114274 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:42.114138    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57c03020-b4de-419f-9da8-ecc972abb9ae-metrics-certs\") pod \"maas-controller-d587dcdd5-bsf8t\" (UID: \"57c03020-b4de-419f-9da8-ecc972abb9ae\") " pod="opendatahub/maas-controller-d587dcdd5-bsf8t"
Sep 14 03:01:42.114274 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:42.114159    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q87qg\" (UniqueName: \"kubernetes.io/projected/57c03020-b4de-419f-9da8-ecc972abb9ae-kube-api-access-q87qg\") pod \"maas-controller-d587dcdd5-bsf8t\" (UID: \"57c03020-b4de-419f-9da8-ecc972abb9ae\") " pod="opendatahub/maas-controller-d587dcdd5-bsf8t"
Sep 14 03:01:42.136212 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:42.136179    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:01:42.215206 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:42.215175    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57c03020-b4de-419f-9da8-ecc972abb9ae-metrics-certs\") pod \"maas-controller-d587dcdd5-bsf8t\" (UID: \"57c03020-b4de-419f-9da8-ecc972abb9ae\") " pod="opendatahub/maas-controller-d587dcdd5-bsf8t"
Sep 14 03:01:42.215374 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:42.215214    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q87qg\" (UniqueName: \"kubernetes.io/projected/57c03020-b4de-419f-9da8-ecc972abb9ae-kube-api-access-q87qg\") pod \"maas-controller-d587dcdd5-bsf8t\" (UID: \"57c03020-b4de-419f-9da8-ecc972abb9ae\") " pod="opendatahub/maas-controller-d587dcdd5-bsf8t"
Sep 14 03:01:42.215374 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:42.215304    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/57c03020-b4de-419f-9da8-ecc972abb9ae-webhook-cert\") pod \"maas-controller-d587dcdd5-bsf8t\" (UID: \"57c03020-b4de-419f-9da8-ecc972abb9ae\") " pod="opendatahub/maas-controller-d587dcdd5-bsf8t"
Sep 14 03:01:42.215374 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:01:42.215314    2564 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found
Sep 14 03:01:42.215519 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:01:42.215427    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57c03020-b4de-419f-9da8-ecc972abb9ae-metrics-certs podName:57c03020-b4de-419f-9da8-ecc972abb9ae nodeName:}" failed. No retries permitted until 2026-09-14 03:01:42.715383531 +0000 UTC m=+548.096467721 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/57c03020-b4de-419f-9da8-ecc972abb9ae-metrics-certs") pod "maas-controller-d587dcdd5-bsf8t" (UID: "57c03020-b4de-419f-9da8-ecc972abb9ae") : secret "maas-controller-metrics-tls" not found
Sep 14 03:01:42.217465 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:42.217440    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:01:42.218004 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:42.217984    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/57c03020-b4de-419f-9da8-ecc972abb9ae-webhook-cert\") pod \"maas-controller-d587dcdd5-bsf8t\" (UID: \"57c03020-b4de-419f-9da8-ecc972abb9ae\") " pod="opendatahub/maas-controller-d587dcdd5-bsf8t"
Sep 14 03:01:42.222800 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:42.222764    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-57d64b7cfb-pbg2k"]
Sep 14 03:01:42.225603 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:42.225586    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57d64b7cfb-pbg2k"
Sep 14 03:01:42.228877 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:42.228856    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q87qg\" (UniqueName: \"kubernetes.io/projected/57c03020-b4de-419f-9da8-ecc972abb9ae-kube-api-access-q87qg\") pod \"maas-controller-d587dcdd5-bsf8t\" (UID: \"57c03020-b4de-419f-9da8-ecc972abb9ae\") " pod="opendatahub/maas-controller-d587dcdd5-bsf8t"
Sep 14 03:01:42.238532 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:42.238511    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-57d64b7cfb-pbg2k"]
Sep 14 03:01:42.316192 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:42.316110    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cec92154-773e-4dd0-b882-6b4e98bc51e4-metrics-certs\") pod \"maas-controller-57d64b7cfb-pbg2k\" (UID: \"cec92154-773e-4dd0-b882-6b4e98bc51e4\") " pod="opendatahub/maas-controller-57d64b7cfb-pbg2k"
Sep 14 03:01:42.316192 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:42.316152    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdkhf\" (UniqueName: \"kubernetes.io/projected/cec92154-773e-4dd0-b882-6b4e98bc51e4-kube-api-access-qdkhf\") pod \"maas-controller-57d64b7cfb-pbg2k\" (UID: \"cec92154-773e-4dd0-b882-6b4e98bc51e4\") " pod="opendatahub/maas-controller-57d64b7cfb-pbg2k"
Sep 14 03:01:42.316385 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:42.316194    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cec92154-773e-4dd0-b882-6b4e98bc51e4-webhook-cert\") pod \"maas-controller-57d64b7cfb-pbg2k\" (UID: \"cec92154-773e-4dd0-b882-6b4e98bc51e4\") " pod="opendatahub/maas-controller-57d64b7cfb-pbg2k"
Sep 14 03:01:42.317012 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:42.316984    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:01:42.416862 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:42.416833    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdkhf\" (UniqueName: \"kubernetes.io/projected/cec92154-773e-4dd0-b882-6b4e98bc51e4-kube-api-access-qdkhf\") pod \"maas-controller-57d64b7cfb-pbg2k\" (UID: \"cec92154-773e-4dd0-b882-6b4e98bc51e4\") " pod="opendatahub/maas-controller-57d64b7cfb-pbg2k"
Sep 14 03:01:42.417007 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:42.416877    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cec92154-773e-4dd0-b882-6b4e98bc51e4-webhook-cert\") pod \"maas-controller-57d64b7cfb-pbg2k\" (UID: \"cec92154-773e-4dd0-b882-6b4e98bc51e4\") " pod="opendatahub/maas-controller-57d64b7cfb-pbg2k"
Sep 14 03:01:42.417007 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:42.416949    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cec92154-773e-4dd0-b882-6b4e98bc51e4-metrics-certs\") pod \"maas-controller-57d64b7cfb-pbg2k\" (UID: \"cec92154-773e-4dd0-b882-6b4e98bc51e4\") " pod="opendatahub/maas-controller-57d64b7cfb-pbg2k"
Sep 14 03:01:42.419419 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:42.419381    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cec92154-773e-4dd0-b882-6b4e98bc51e4-webhook-cert\") pod \"maas-controller-57d64b7cfb-pbg2k\" (UID: \"cec92154-773e-4dd0-b882-6b4e98bc51e4\") " pod="opendatahub/maas-controller-57d64b7cfb-pbg2k"
Sep 14 03:01:42.419520 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:42.419460    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cec92154-773e-4dd0-b882-6b4e98bc51e4-metrics-certs\") pod \"maas-controller-57d64b7cfb-pbg2k\" (UID: \"cec92154-773e-4dd0-b882-6b4e98bc51e4\") " pod="opendatahub/maas-controller-57d64b7cfb-pbg2k"
Sep 14 03:01:42.426750 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:42.426727    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdkhf\" (UniqueName: \"kubernetes.io/projected/cec92154-773e-4dd0-b882-6b4e98bc51e4-kube-api-access-qdkhf\") pod \"maas-controller-57d64b7cfb-pbg2k\" (UID: \"cec92154-773e-4dd0-b882-6b4e98bc51e4\") " pod="opendatahub/maas-controller-57d64b7cfb-pbg2k"
Sep 14 03:01:42.545646 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:42.545610    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57d64b7cfb-pbg2k"
Sep 14 03:01:42.719866 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:42.719833    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57c03020-b4de-419f-9da8-ecc972abb9ae-metrics-certs\") pod \"maas-controller-d587dcdd5-bsf8t\" (UID: \"57c03020-b4de-419f-9da8-ecc972abb9ae\") " pod="opendatahub/maas-controller-d587dcdd5-bsf8t"
Sep 14 03:01:42.722166 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:42.722138    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57c03020-b4de-419f-9da8-ecc972abb9ae-metrics-certs\") pod \"maas-controller-d587dcdd5-bsf8t\" (UID: \"57c03020-b4de-419f-9da8-ecc972abb9ae\") " pod="opendatahub/maas-controller-d587dcdd5-bsf8t"
Sep 14 03:01:42.814310 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:42.814286    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-57d64b7cfb-pbg2k"]
Sep 14 03:01:42.815473 ip-10-0-141-4 kubenswrapper[2564]: W0914 03:01:42.815445    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcec92154_773e_4dd0_b882_6b4e98bc51e4.slice/crio-9aa0d8ff860bfe9857308733d14d3cf1146003231659dac6cac05d4bb343360b WatchSource:0}: Error finding container 9aa0d8ff860bfe9857308733d14d3cf1146003231659dac6cac05d4bb343360b: Status 404 returned error can't find the container with id 9aa0d8ff860bfe9857308733d14d3cf1146003231659dac6cac05d4bb343360b
Sep 14 03:01:42.976270 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:42.976185    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d587dcdd5-bsf8t"
Sep 14 03:01:43.244990 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:43.244964    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d587dcdd5-bsf8t"]
Sep 14 03:01:43.247108 ip-10-0-141-4 kubenswrapper[2564]: W0914 03:01:43.247077    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57c03020_b4de_419f_9da8_ecc972abb9ae.slice/crio-cd5f20b86b985a9da52fbea45cfef08518a5e6ab06f2ba1a9941189233de50c2 WatchSource:0}: Error finding container cd5f20b86b985a9da52fbea45cfef08518a5e6ab06f2ba1a9941189233de50c2: Status 404 returned error can't find the container with id cd5f20b86b985a9da52fbea45cfef08518a5e6ab06f2ba1a9941189233de50c2
Sep 14 03:01:43.688818 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:43.688780    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d587dcdd5-bsf8t" event={"ID":"57c03020-b4de-419f-9da8-ecc972abb9ae","Type":"ContainerStarted","Data":"cd5f20b86b985a9da52fbea45cfef08518a5e6ab06f2ba1a9941189233de50c2"}
Sep 14 03:01:43.691072 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:43.691025    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57d64b7cfb-pbg2k" event={"ID":"cec92154-773e-4dd0-b882-6b4e98bc51e4","Type":"ContainerStarted","Data":"9aa0d8ff860bfe9857308733d14d3cf1146003231659dac6cac05d4bb343360b"}
Sep 14 03:01:45.655211 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:45.655176    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:01:45.702363 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:45.702278    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d587dcdd5-bsf8t" event={"ID":"57c03020-b4de-419f-9da8-ecc972abb9ae","Type":"ContainerStarted","Data":"7bc41b5e7cc036b0a3a3d98c278e1d1c7a6a0d58d9fe835d9b47f945597f0ba1"}
Sep 14 03:01:45.702552 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:45.702393    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-d587dcdd5-bsf8t"
Sep 14 03:01:45.703933 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:45.703911    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57d64b7cfb-pbg2k" event={"ID":"cec92154-773e-4dd0-b882-6b4e98bc51e4","Type":"ContainerStarted","Data":"c7d25db228ef20e17d423426354210a33626e37d89744b6dc61bef4f0356bdf4"}
Sep 14 03:01:45.704042 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:45.704022    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-57d64b7cfb-pbg2k"
Sep 14 03:01:45.728156 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:45.728099    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-d587dcdd5-bsf8t" podStartSLOduration=1.510679076 podStartE2EDuration="3.728083271s" totalImagesPullingTime="2.217404195s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:01:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:01:45.726478446 +0000 UTC m=+551.107562645" watchObservedRunningTime="2026-09-14 03:01:45.728083271 +0000 UTC m=+551.109167469"
Sep 14 03:01:45.760231 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:45.760174    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-57d64b7cfb-pbg2k" podStartSLOduration=1.122108764 podStartE2EDuration="3.760157662s" totalImagesPullingTime="2.638048898s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:01:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:01:45.760044968 +0000 UTC m=+551.141129169" watchObservedRunningTime="2026-09-14 03:01:45.760157662 +0000 UTC m=+551.141241915"
Sep 14 03:01:45.762547 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:45.762520    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:01:56.710167 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:56.710131    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-57d64b7cfb-pbg2k"
Sep 14 03:01:56.710557 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:56.710204    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-d587dcdd5-bsf8t"
Sep 14 03:01:56.780614 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:56.780583    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-d587dcdd5-bsf8t"]
Sep 14 03:01:56.780837 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:56.780807    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-d587dcdd5-bsf8t" podUID="57c03020-b4de-419f-9da8-ecc972abb9ae" containerName="manager" containerID="cri-o://7bc41b5e7cc036b0a3a3d98c278e1d1c7a6a0d58d9fe835d9b47f945597f0ba1" gracePeriod=30
Sep 14 03:01:57.026892 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:57.026867    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d587dcdd5-bsf8t"
Sep 14 03:01:57.156355 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:57.156315    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q87qg\" (UniqueName: \"kubernetes.io/projected/57c03020-b4de-419f-9da8-ecc972abb9ae-kube-api-access-q87qg\") pod \"57c03020-b4de-419f-9da8-ecc972abb9ae\" (UID: \"57c03020-b4de-419f-9da8-ecc972abb9ae\") "
Sep 14 03:01:57.156554 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:57.156399    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/57c03020-b4de-419f-9da8-ecc972abb9ae-webhook-cert\") pod \"57c03020-b4de-419f-9da8-ecc972abb9ae\" (UID: \"57c03020-b4de-419f-9da8-ecc972abb9ae\") "
Sep 14 03:01:57.156554 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:57.156467    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57c03020-b4de-419f-9da8-ecc972abb9ae-metrics-certs\") pod \"57c03020-b4de-419f-9da8-ecc972abb9ae\" (UID: \"57c03020-b4de-419f-9da8-ecc972abb9ae\") "
Sep 14 03:01:57.158681 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:57.158645    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57c03020-b4de-419f-9da8-ecc972abb9ae-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "57c03020-b4de-419f-9da8-ecc972abb9ae" (UID: "57c03020-b4de-419f-9da8-ecc972abb9ae"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 03:01:57.158778 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:57.158655    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57c03020-b4de-419f-9da8-ecc972abb9ae-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "57c03020-b4de-419f-9da8-ecc972abb9ae" (UID: "57c03020-b4de-419f-9da8-ecc972abb9ae"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 03:01:57.158778 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:57.158688    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57c03020-b4de-419f-9da8-ecc972abb9ae-kube-api-access-q87qg" (OuterVolumeSpecName: "kube-api-access-q87qg") pod "57c03020-b4de-419f-9da8-ecc972abb9ae" (UID: "57c03020-b4de-419f-9da8-ecc972abb9ae"). InnerVolumeSpecName "kube-api-access-q87qg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:01:57.258297 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:57.258194    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57c03020-b4de-419f-9da8-ecc972abb9ae-metrics-certs\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:01:57.258297 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:57.258227    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q87qg\" (UniqueName: \"kubernetes.io/projected/57c03020-b4de-419f-9da8-ecc972abb9ae-kube-api-access-q87qg\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:01:57.258297 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:57.258242    2564 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/57c03020-b4de-419f-9da8-ecc972abb9ae-webhook-cert\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:01:57.767613 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:57.767578    2564 generic.go:356] "Generic (PLEG): container finished" podID="57c03020-b4de-419f-9da8-ecc972abb9ae" containerID="7bc41b5e7cc036b0a3a3d98c278e1d1c7a6a0d58d9fe835d9b47f945597f0ba1" exitCode=0
Sep 14 03:01:57.768033 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:57.767625    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d587dcdd5-bsf8t"
Sep 14 03:01:57.768033 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:57.767659    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d587dcdd5-bsf8t" event={"ID":"57c03020-b4de-419f-9da8-ecc972abb9ae","Type":"ContainerDied","Data":"7bc41b5e7cc036b0a3a3d98c278e1d1c7a6a0d58d9fe835d9b47f945597f0ba1"}
Sep 14 03:01:57.768033 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:57.767705    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d587dcdd5-bsf8t" event={"ID":"57c03020-b4de-419f-9da8-ecc972abb9ae","Type":"ContainerDied","Data":"cd5f20b86b985a9da52fbea45cfef08518a5e6ab06f2ba1a9941189233de50c2"}
Sep 14 03:01:57.768033 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:57.767733    2564 scope.go:117] "RemoveContainer" containerID="7bc41b5e7cc036b0a3a3d98c278e1d1c7a6a0d58d9fe835d9b47f945597f0ba1"
Sep 14 03:01:57.776491 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:57.776481    2564 scope.go:117] "RemoveContainer" containerID="7bc41b5e7cc036b0a3a3d98c278e1d1c7a6a0d58d9fe835d9b47f945597f0ba1"
Sep 14 03:01:57.776761 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:01:57.776743    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bc41b5e7cc036b0a3a3d98c278e1d1c7a6a0d58d9fe835d9b47f945597f0ba1\": container with ID starting with 7bc41b5e7cc036b0a3a3d98c278e1d1c7a6a0d58d9fe835d9b47f945597f0ba1 not found: ID does not exist" containerID="7bc41b5e7cc036b0a3a3d98c278e1d1c7a6a0d58d9fe835d9b47f945597f0ba1"
Sep 14 03:01:57.776799 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:57.776770    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bc41b5e7cc036b0a3a3d98c278e1d1c7a6a0d58d9fe835d9b47f945597f0ba1"} err="failed to get container status \"7bc41b5e7cc036b0a3a3d98c278e1d1c7a6a0d58d9fe835d9b47f945597f0ba1\": rpc error: code = NotFound desc = could not find container \"7bc41b5e7cc036b0a3a3d98c278e1d1c7a6a0d58d9fe835d9b47f945597f0ba1\": container with ID starting with 7bc41b5e7cc036b0a3a3d98c278e1d1c7a6a0d58d9fe835d9b47f945597f0ba1 not found: ID does not exist"
Sep 14 03:01:57.790189 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:57.790165    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-d587dcdd5-bsf8t"]
Sep 14 03:01:57.795633 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:57.795613    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-d587dcdd5-bsf8t"]
Sep 14 03:01:58.201517 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:58.201489    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:01:59.097750 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:01:59.097719    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57c03020-b4de-419f-9da8-ecc972abb9ae" path="/var/lib/kubelet/pods/57c03020-b4de-419f-9da8-ecc972abb9ae/volumes"
Sep 14 03:02:15.850051 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:15.849993    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:02:15.963398 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:15.963354    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:02:17.379830 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:17.379793    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:02:17.503692 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:17.503660    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:02:17.672541 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:17.672452    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-564fdcb8d5-kbxgg"]
Sep 14 03:02:17.672913 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:17.672894    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57c03020-b4de-419f-9da8-ecc972abb9ae" containerName="manager"
Sep 14 03:02:17.673005 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:17.672916    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="57c03020-b4de-419f-9da8-ecc972abb9ae" containerName="manager"
Sep 14 03:02:17.673059 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:17.673005    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="57c03020-b4de-419f-9da8-ecc972abb9ae" containerName="manager"
Sep 14 03:02:17.676316 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:17.676295    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-564fdcb8d5-kbxgg"
Sep 14 03:02:17.679138 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:17.679116    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 14 03:02:17.680022 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:17.680007    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-jf78h\""
Sep 14 03:02:17.680112 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:17.680022    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 14 03:02:17.687717 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:17.687693    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-564fdcb8d5-kbxgg"]
Sep 14 03:02:17.738274 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:17.738244    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2be1f88b-a0b7-4bdf-8dbc-bfeb9db9f8e7-maas-api-tls\") pod \"maas-api-564fdcb8d5-kbxgg\" (UID: \"2be1f88b-a0b7-4bdf-8dbc-bfeb9db9f8e7\") " pod="odh-ai-gateway-infra/maas-api-564fdcb8d5-kbxgg"
Sep 14 03:02:17.738436 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:17.738288    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2be1f88b-a0b7-4bdf-8dbc-bfeb9db9f8e7-openshift-service-ca\") pod \"maas-api-564fdcb8d5-kbxgg\" (UID: \"2be1f88b-a0b7-4bdf-8dbc-bfeb9db9f8e7\") " pod="odh-ai-gateway-infra/maas-api-564fdcb8d5-kbxgg"
Sep 14 03:02:17.738436 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:17.738313    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2be1f88b-a0b7-4bdf-8dbc-bfeb9db9f8e7-metrics-certs\") pod \"maas-api-564fdcb8d5-kbxgg\" (UID: \"2be1f88b-a0b7-4bdf-8dbc-bfeb9db9f8e7\") " pod="odh-ai-gateway-infra/maas-api-564fdcb8d5-kbxgg"
Sep 14 03:02:17.738436 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:17.738348    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfs8z\" (UniqueName: \"kubernetes.io/projected/2be1f88b-a0b7-4bdf-8dbc-bfeb9db9f8e7-kube-api-access-pfs8z\") pod \"maas-api-564fdcb8d5-kbxgg\" (UID: \"2be1f88b-a0b7-4bdf-8dbc-bfeb9db9f8e7\") " pod="odh-ai-gateway-infra/maas-api-564fdcb8d5-kbxgg"
Sep 14 03:02:17.838861 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:17.838824    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2be1f88b-a0b7-4bdf-8dbc-bfeb9db9f8e7-metrics-certs\") pod \"maas-api-564fdcb8d5-kbxgg\" (UID: \"2be1f88b-a0b7-4bdf-8dbc-bfeb9db9f8e7\") " pod="odh-ai-gateway-infra/maas-api-564fdcb8d5-kbxgg"
Sep 14 03:02:17.839047 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:17.838884    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfs8z\" (UniqueName: \"kubernetes.io/projected/2be1f88b-a0b7-4bdf-8dbc-bfeb9db9f8e7-kube-api-access-pfs8z\") pod \"maas-api-564fdcb8d5-kbxgg\" (UID: \"2be1f88b-a0b7-4bdf-8dbc-bfeb9db9f8e7\") " pod="odh-ai-gateway-infra/maas-api-564fdcb8d5-kbxgg"
Sep 14 03:02:17.839047 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:17.838931    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2be1f88b-a0b7-4bdf-8dbc-bfeb9db9f8e7-maas-api-tls\") pod \"maas-api-564fdcb8d5-kbxgg\" (UID: \"2be1f88b-a0b7-4bdf-8dbc-bfeb9db9f8e7\") " pod="odh-ai-gateway-infra/maas-api-564fdcb8d5-kbxgg"
Sep 14 03:02:17.839047 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:17.838972    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2be1f88b-a0b7-4bdf-8dbc-bfeb9db9f8e7-openshift-service-ca\") pod \"maas-api-564fdcb8d5-kbxgg\" (UID: \"2be1f88b-a0b7-4bdf-8dbc-bfeb9db9f8e7\") " pod="odh-ai-gateway-infra/maas-api-564fdcb8d5-kbxgg"
Sep 14 03:02:17.839217 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:02:17.839056    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found
Sep 14 03:02:17.839217 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:02:17.839144    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2be1f88b-a0b7-4bdf-8dbc-bfeb9db9f8e7-maas-api-tls podName:2be1f88b-a0b7-4bdf-8dbc-bfeb9db9f8e7 nodeName:}" failed. No retries permitted until 2026-09-14 03:02:18.339121479 +0000 UTC m=+583.720205674 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2be1f88b-a0b7-4bdf-8dbc-bfeb9db9f8e7-maas-api-tls") pod "maas-api-564fdcb8d5-kbxgg" (UID: "2be1f88b-a0b7-4bdf-8dbc-bfeb9db9f8e7") : secret "maas-api-serving-cert" not found
Sep 14 03:02:17.839665 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:17.839647    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2be1f88b-a0b7-4bdf-8dbc-bfeb9db9f8e7-openshift-service-ca\") pod \"maas-api-564fdcb8d5-kbxgg\" (UID: \"2be1f88b-a0b7-4bdf-8dbc-bfeb9db9f8e7\") " pod="odh-ai-gateway-infra/maas-api-564fdcb8d5-kbxgg"
Sep 14 03:02:17.841156 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:17.841135    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2be1f88b-a0b7-4bdf-8dbc-bfeb9db9f8e7-metrics-certs\") pod \"maas-api-564fdcb8d5-kbxgg\" (UID: \"2be1f88b-a0b7-4bdf-8dbc-bfeb9db9f8e7\") " pod="odh-ai-gateway-infra/maas-api-564fdcb8d5-kbxgg"
Sep 14 03:02:17.848634 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:17.848613    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfs8z\" (UniqueName: \"kubernetes.io/projected/2be1f88b-a0b7-4bdf-8dbc-bfeb9db9f8e7-kube-api-access-pfs8z\") pod \"maas-api-564fdcb8d5-kbxgg\" (UID: \"2be1f88b-a0b7-4bdf-8dbc-bfeb9db9f8e7\") " pod="odh-ai-gateway-infra/maas-api-564fdcb8d5-kbxgg"
Sep 14 03:02:18.249057 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:18.249018    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-265v7"]
Sep 14 03:02:18.251480 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:18.251459    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-265v7"
Sep 14 03:02:18.254852 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:18.254830    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-nxxzg\""
Sep 14 03:02:18.255012 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:18.254994    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 14 03:02:18.270776 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:18.270753    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-265v7"]
Sep 14 03:02:18.282304 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:18.282275    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-27xw5"]
Sep 14 03:02:18.284664 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:18.284647    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-27xw5"
Sep 14 03:02:18.297567 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:18.297540    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-27xw5"]
Sep 14 03:02:18.343135 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:18.343097    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6f51115-9cf5-49ab-97b5-d0a840313cf4-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-265v7\" (UID: \"c6f51115-9cf5-49ab-97b5-d0a840313cf4\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-265v7"
Sep 14 03:02:18.343135 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:18.343135    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk87q\" (UniqueName: \"kubernetes.io/projected/ce48fdb0-6167-4a7e-ac9a-20fffb8e7500-kube-api-access-dk87q\") pod \"payload-pre-processing-59fbc47bf8-27xw5\" (UID: \"ce48fdb0-6167-4a7e-ac9a-20fffb8e7500\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-27xw5"
Sep 14 03:02:18.343393 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:18.343156    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce48fdb0-6167-4a7e-ac9a-20fffb8e7500-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-27xw5\" (UID: \"ce48fdb0-6167-4a7e-ac9a-20fffb8e7500\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-27xw5"
Sep 14 03:02:18.343393 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:18.343174    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p65cx\" (UniqueName: \"kubernetes.io/projected/c6f51115-9cf5-49ab-97b5-d0a840313cf4-kube-api-access-p65cx\") pod \"payload-processing-5b54b7fbb7-265v7\" (UID: \"c6f51115-9cf5-49ab-97b5-d0a840313cf4\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-265v7"
Sep 14 03:02:18.343393 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:18.343269    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2be1f88b-a0b7-4bdf-8dbc-bfeb9db9f8e7-maas-api-tls\") pod \"maas-api-564fdcb8d5-kbxgg\" (UID: \"2be1f88b-a0b7-4bdf-8dbc-bfeb9db9f8e7\") " pod="odh-ai-gateway-infra/maas-api-564fdcb8d5-kbxgg"
Sep 14 03:02:18.345633 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:18.345616    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2be1f88b-a0b7-4bdf-8dbc-bfeb9db9f8e7-maas-api-tls\") pod \"maas-api-564fdcb8d5-kbxgg\" (UID: \"2be1f88b-a0b7-4bdf-8dbc-bfeb9db9f8e7\") " pod="odh-ai-gateway-infra/maas-api-564fdcb8d5-kbxgg"
Sep 14 03:02:18.443920 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:18.443878    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6f51115-9cf5-49ab-97b5-d0a840313cf4-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-265v7\" (UID: \"c6f51115-9cf5-49ab-97b5-d0a840313cf4\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-265v7"
Sep 14 03:02:18.444385 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:18.443936    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk87q\" (UniqueName: \"kubernetes.io/projected/ce48fdb0-6167-4a7e-ac9a-20fffb8e7500-kube-api-access-dk87q\") pod \"payload-pre-processing-59fbc47bf8-27xw5\" (UID: \"ce48fdb0-6167-4a7e-ac9a-20fffb8e7500\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-27xw5"
Sep 14 03:02:18.444385 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:18.443970    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce48fdb0-6167-4a7e-ac9a-20fffb8e7500-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-27xw5\" (UID: \"ce48fdb0-6167-4a7e-ac9a-20fffb8e7500\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-27xw5"
Sep 14 03:02:18.444385 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:18.443999    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p65cx\" (UniqueName: \"kubernetes.io/projected/c6f51115-9cf5-49ab-97b5-d0a840313cf4-kube-api-access-p65cx\") pod \"payload-processing-5b54b7fbb7-265v7\" (UID: \"c6f51115-9cf5-49ab-97b5-d0a840313cf4\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-265v7"
Sep 14 03:02:18.444671 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:18.444650    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6f51115-9cf5-49ab-97b5-d0a840313cf4-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-265v7\" (UID: \"c6f51115-9cf5-49ab-97b5-d0a840313cf4\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-265v7"
Sep 14 03:02:18.444712 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:18.444671    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce48fdb0-6167-4a7e-ac9a-20fffb8e7500-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-27xw5\" (UID: \"ce48fdb0-6167-4a7e-ac9a-20fffb8e7500\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-27xw5"
Sep 14 03:02:18.452757 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:18.452732    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk87q\" (UniqueName: \"kubernetes.io/projected/ce48fdb0-6167-4a7e-ac9a-20fffb8e7500-kube-api-access-dk87q\") pod \"payload-pre-processing-59fbc47bf8-27xw5\" (UID: \"ce48fdb0-6167-4a7e-ac9a-20fffb8e7500\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-27xw5"
Sep 14 03:02:18.453312 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:18.453292    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p65cx\" (UniqueName: \"kubernetes.io/projected/c6f51115-9cf5-49ab-97b5-d0a840313cf4-kube-api-access-p65cx\") pod \"payload-processing-5b54b7fbb7-265v7\" (UID: \"c6f51115-9cf5-49ab-97b5-d0a840313cf4\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-265v7"
Sep 14 03:02:18.562038 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:18.562005    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-265v7"
Sep 14 03:02:18.586453 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:18.586423    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-564fdcb8d5-kbxgg"
Sep 14 03:02:18.595710 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:18.595687    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-27xw5"
Sep 14 03:02:18.851816 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:18.851788    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-265v7"]
Sep 14 03:02:18.854041 ip-10-0-141-4 kubenswrapper[2564]: W0914 03:02:18.854015    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6f51115_9cf5_49ab_97b5_d0a840313cf4.slice/crio-ccc5a7afe3b8821002b2d3bf8d6eb3044b7f87028002edd4c7b0794dfc1f26ad WatchSource:0}: Error finding container ccc5a7afe3b8821002b2d3bf8d6eb3044b7f87028002edd4c7b0794dfc1f26ad: Status 404 returned error can't find the container with id ccc5a7afe3b8821002b2d3bf8d6eb3044b7f87028002edd4c7b0794dfc1f26ad
Sep 14 03:02:18.869718 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:18.869695    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-265v7" event={"ID":"c6f51115-9cf5-49ab-97b5-d0a840313cf4","Type":"ContainerStarted","Data":"ccc5a7afe3b8821002b2d3bf8d6eb3044b7f87028002edd4c7b0794dfc1f26ad"}
Sep 14 03:02:18.892665 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:18.892641    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-564fdcb8d5-kbxgg"]
Sep 14 03:02:18.894760 ip-10-0-141-4 kubenswrapper[2564]: W0914 03:02:18.894574    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2be1f88b_a0b7_4bdf_8dbc_bfeb9db9f8e7.slice/crio-8201d47ecd5917d8721df4cdbf969fd7bef4a73e049ea14bef8941fabcd58107 WatchSource:0}: Error finding container 8201d47ecd5917d8721df4cdbf969fd7bef4a73e049ea14bef8941fabcd58107: Status 404 returned error can't find the container with id 8201d47ecd5917d8721df4cdbf969fd7bef4a73e049ea14bef8941fabcd58107
Sep 14 03:02:18.897054 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:18.897027    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-27xw5"]
Sep 14 03:02:18.900529 ip-10-0-141-4 kubenswrapper[2564]: W0914 03:02:18.900510    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce48fdb0_6167_4a7e_ac9a_20fffb8e7500.slice/crio-a3a9a8afa58eab2caf8bd1f1596edfeaab7b9a318e762bd0e64ba5a1f36b3e1c WatchSource:0}: Error finding container a3a9a8afa58eab2caf8bd1f1596edfeaab7b9a318e762bd0e64ba5a1f36b3e1c: Status 404 returned error can't find the container with id a3a9a8afa58eab2caf8bd1f1596edfeaab7b9a318e762bd0e64ba5a1f36b3e1c
Sep 14 03:02:19.107152 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:19.107064    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:02:19.878724 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:19.878680    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-564fdcb8d5-kbxgg" event={"ID":"2be1f88b-a0b7-4bdf-8dbc-bfeb9db9f8e7","Type":"ContainerStarted","Data":"8201d47ecd5917d8721df4cdbf969fd7bef4a73e049ea14bef8941fabcd58107"}
Sep 14 03:02:19.880317 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:19.880290    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-27xw5" event={"ID":"ce48fdb0-6167-4a7e-ac9a-20fffb8e7500","Type":"ContainerStarted","Data":"a3a9a8afa58eab2caf8bd1f1596edfeaab7b9a318e762bd0e64ba5a1f36b3e1c"}
Sep 14 03:02:20.887741 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:20.887702    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-564fdcb8d5-kbxgg" event={"ID":"2be1f88b-a0b7-4bdf-8dbc-bfeb9db9f8e7","Type":"ContainerStarted","Data":"45bf645935b5d7c5121fbb73df8108d53155e5b9f29b98305d3bb7a114d70d82"}
Sep 14 03:02:20.888219 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:20.887806    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-564fdcb8d5-kbxgg"
Sep 14 03:02:20.889307 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:20.889284    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-265v7" event={"ID":"c6f51115-9cf5-49ab-97b5-d0a840313cf4","Type":"ContainerStarted","Data":"f51dfb390b1602b23496c3df1c95ffb316001c6c87ab2d4f8a0fd89151e69f34"}
Sep 14 03:02:20.889449 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:20.889391    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-265v7"
Sep 14 03:02:20.890829 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:20.890805    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-27xw5" event={"ID":"ce48fdb0-6167-4a7e-ac9a-20fffb8e7500","Type":"ContainerStarted","Data":"753bddc76b729b0bf3b91de65033ee47c6cd3ac9e155c2c29bac71ef2f97489f"}
Sep 14 03:02:20.890911 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:20.890862    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-27xw5"
Sep 14 03:02:20.910119 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:20.910072    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-564fdcb8d5-kbxgg" podStartSLOduration=2.094355948 podStartE2EDuration="3.910057566s" totalImagesPullingTime="1.815701618s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:02:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:02:20.907741093 +0000 UTC m=+586.288825305" watchObservedRunningTime="2026-09-14 03:02:20.910057566 +0000 UTC m=+586.291141823"
Sep 14 03:02:20.934198 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:20.934148    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-265v7" podStartSLOduration=1.600569694 podStartE2EDuration="2.934132129s" totalImagesPullingTime="1.333562435s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:02:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:02:20.932790267 +0000 UTC m=+586.313874468" watchObservedRunningTime="2026-09-14 03:02:20.934132129 +0000 UTC m=+586.315216327"
Sep 14 03:02:20.953086 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:20.953023    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-27xw5" podStartSLOduration=1.668787827 podStartE2EDuration="2.953003647s" totalImagesPullingTime="1.28421582s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:02:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:02:20.951593989 +0000 UTC m=+586.332678189" watchObservedRunningTime="2026-09-14 03:02:20.953003647 +0000 UTC m=+586.334087851"
Sep 14 03:02:26.900528 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:26.900498    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-564fdcb8d5-kbxgg"
Sep 14 03:02:27.916280 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:27.916237    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:02:28.044702 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:28.044661    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:02:31.382881 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:31.382846    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:02:31.897588 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:31.897550    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-27xw5"
Sep 14 03:02:31.897774 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:31.897722    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-265v7"
Sep 14 03:02:34.003912 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:34.003872    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:02:38.843852 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:38.843800    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:02:48.283137 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:48.283100    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:02:57.564051 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:57.563977    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fqms6"]
Sep 14 03:02:57.567100 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:57.567084    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fqms6"
Sep 14 03:02:57.571087 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:57.571063    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-wdpm7\""
Sep 14 03:02:57.571208 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:57.571183    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 14 03:02:57.571261 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:57.571202    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 14 03:02:57.571261 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:57.571223    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 14 03:02:57.586146 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:57.586114    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fqms6"]
Sep 14 03:02:57.604843 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:57.604816    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19776359-d16f-4d5d-a874-fda5a9d0a9a8-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-fqms6\" (UID: \"19776359-d16f-4d5d-a874-fda5a9d0a9a8\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fqms6"
Sep 14 03:02:57.604988 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:57.604877    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19776359-d16f-4d5d-a874-fda5a9d0a9a8-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-fqms6\" (UID: \"19776359-d16f-4d5d-a874-fda5a9d0a9a8\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fqms6"
Sep 14 03:02:57.610636 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:57.604922    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv8md\" (UniqueName: \"kubernetes.io/projected/19776359-d16f-4d5d-a874-fda5a9d0a9a8-kube-api-access-lv8md\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-fqms6\" (UID: \"19776359-d16f-4d5d-a874-fda5a9d0a9a8\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fqms6"
Sep 14 03:02:57.610636 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:57.605348    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19776359-d16f-4d5d-a874-fda5a9d0a9a8-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-fqms6\" (UID: \"19776359-d16f-4d5d-a874-fda5a9d0a9a8\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fqms6"
Sep 14 03:02:57.610636 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:57.605433    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19776359-d16f-4d5d-a874-fda5a9d0a9a8-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-fqms6\" (UID: \"19776359-d16f-4d5d-a874-fda5a9d0a9a8\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fqms6"
Sep 14 03:02:57.610636 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:57.605553    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19776359-d16f-4d5d-a874-fda5a9d0a9a8-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-fqms6\" (UID: \"19776359-d16f-4d5d-a874-fda5a9d0a9a8\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fqms6"
Sep 14 03:02:57.611640 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:57.611605    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:02:57.706622 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:57.706586    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19776359-d16f-4d5d-a874-fda5a9d0a9a8-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-fqms6\" (UID: \"19776359-d16f-4d5d-a874-fda5a9d0a9a8\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fqms6"
Sep 14 03:02:57.706801 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:57.706641    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19776359-d16f-4d5d-a874-fda5a9d0a9a8-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-fqms6\" (UID: \"19776359-d16f-4d5d-a874-fda5a9d0a9a8\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fqms6"
Sep 14 03:02:57.706801 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:57.706687    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19776359-d16f-4d5d-a874-fda5a9d0a9a8-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-fqms6\" (UID: \"19776359-d16f-4d5d-a874-fda5a9d0a9a8\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fqms6"
Sep 14 03:02:57.706801 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:57.706703    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv8md\" (UniqueName: \"kubernetes.io/projected/19776359-d16f-4d5d-a874-fda5a9d0a9a8-kube-api-access-lv8md\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-fqms6\" (UID: \"19776359-d16f-4d5d-a874-fda5a9d0a9a8\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fqms6"
Sep 14 03:02:57.706801 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:57.706722    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19776359-d16f-4d5d-a874-fda5a9d0a9a8-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-fqms6\" (UID: \"19776359-d16f-4d5d-a874-fda5a9d0a9a8\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fqms6"
Sep 14 03:02:57.706801 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:57.706749    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19776359-d16f-4d5d-a874-fda5a9d0a9a8-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-fqms6\" (UID: \"19776359-d16f-4d5d-a874-fda5a9d0a9a8\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fqms6"
Sep 14 03:02:57.707162 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:57.707138    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19776359-d16f-4d5d-a874-fda5a9d0a9a8-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-fqms6\" (UID: \"19776359-d16f-4d5d-a874-fda5a9d0a9a8\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fqms6"
Sep 14 03:02:57.707225 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:57.707193    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19776359-d16f-4d5d-a874-fda5a9d0a9a8-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-fqms6\" (UID: \"19776359-d16f-4d5d-a874-fda5a9d0a9a8\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fqms6"
Sep 14 03:02:57.707261 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:57.707227    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19776359-d16f-4d5d-a874-fda5a9d0a9a8-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-fqms6\" (UID: \"19776359-d16f-4d5d-a874-fda5a9d0a9a8\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fqms6"
Sep 14 03:02:57.708890 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:57.708871    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19776359-d16f-4d5d-a874-fda5a9d0a9a8-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-fqms6\" (UID: \"19776359-d16f-4d5d-a874-fda5a9d0a9a8\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fqms6"
Sep 14 03:02:57.709129 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:57.709109    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19776359-d16f-4d5d-a874-fda5a9d0a9a8-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-fqms6\" (UID: \"19776359-d16f-4d5d-a874-fda5a9d0a9a8\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fqms6"
Sep 14 03:02:57.716781 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:57.716755    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv8md\" (UniqueName: \"kubernetes.io/projected/19776359-d16f-4d5d-a874-fda5a9d0a9a8-kube-api-access-lv8md\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-fqms6\" (UID: \"19776359-d16f-4d5d-a874-fda5a9d0a9a8\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fqms6"
Sep 14 03:02:57.877474 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:57.877359    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fqms6"
Sep 14 03:02:58.150225 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:58.150203    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fqms6"]
Sep 14 03:02:58.152201 ip-10-0-141-4 kubenswrapper[2564]: W0914 03:02:58.152169    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19776359_d16f_4d5d_a874_fda5a9d0a9a8.slice/crio-7e02bf05410891359e5198483797c29f370368e34c3ae8fc02eeeacb77634431 WatchSource:0}: Error finding container 7e02bf05410891359e5198483797c29f370368e34c3ae8fc02eeeacb77634431: Status 404 returned error can't find the container with id 7e02bf05410891359e5198483797c29f370368e34c3ae8fc02eeeacb77634431
Sep 14 03:02:58.153604 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:58.153580    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 14 03:02:59.083290 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:02:59.083259    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fqms6" event={"ID":"19776359-d16f-4d5d-a874-fda5a9d0a9a8","Type":"ContainerStarted","Data":"7e02bf05410891359e5198483797c29f370368e34c3ae8fc02eeeacb77634431"}
Sep 14 03:03:00.089246 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:03:00.089216    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fqms6" event={"ID":"19776359-d16f-4d5d-a874-fda5a9d0a9a8","Type":"ContainerStarted","Data":"b49bef19f58e3ec2f994e2ee2b1cfa8dffb1d1f23fd87a121b1dabeade7d40e4"}
Sep 14 03:03:00.108713 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:03:00.108664    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fqms6" podStartSLOduration=2.14660669 podStartE2EDuration="3.108650154s" totalImagesPullingTime="962.043464ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:02:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:03:00.106811893 +0000 UTC m=+625.487896093" watchObservedRunningTime="2026-09-14 03:03:00.108650154 +0000 UTC m=+625.489734354"
Sep 14 03:03:03.211767 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:03:03.211713    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:03:07.878129 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:03:07.878091    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fqms6"
Sep 14 03:03:07.878596 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:03:07.878155    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fqms6"
Sep 14 03:03:07.890718 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:03:07.890696    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fqms6"
Sep 14 03:03:08.140117 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:03:08.140035    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fqms6"
Sep 14 03:03:09.810200 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:03:09.810156    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:03:18.745889 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:03:18.745857    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f94x6m"]
Sep 14 03:03:18.748367 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:03:18.748346    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f94x6m"
Sep 14 03:03:18.751430 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:03:18.751384    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 14 03:03:18.764582 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:03:18.764556    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f94x6m"]
Sep 14 03:03:18.779637 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:03:18.779609    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:03:18.790959 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:03:18.790934    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e3a9df9-6fa6-499f-a364-ff126aa6ea0c-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f94x6m\" (UID: \"3e3a9df9-6fa6-499f-a364-ff126aa6ea0c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f94x6m"
Sep 14 03:03:18.791056 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:03:18.790979    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28g46\" (UniqueName: \"kubernetes.io/projected/3e3a9df9-6fa6-499f-a364-ff126aa6ea0c-kube-api-access-28g46\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f94x6m\" (UID: \"3e3a9df9-6fa6-499f-a364-ff126aa6ea0c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f94x6m"
Sep 14 03:03:18.791113 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:03:18.791070    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e3a9df9-6fa6-499f-a364-ff126aa6ea0c-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f94x6m\" (UID: \"3e3a9df9-6fa6-499f-a364-ff126aa6ea0c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f94x6m"
Sep 14 03:03:18.791159 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:03:18.791118    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e3a9df9-6fa6-499f-a364-ff126aa6ea0c-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f94x6m\" (UID: \"3e3a9df9-6fa6-499f-a364-ff126aa6ea0c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f94x6m"
Sep 14 03:03:18.791159 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:03:18.791149    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e3a9df9-6fa6-499f-a364-ff126aa6ea0c-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f94x6m\" (UID: \"3e3a9df9-6fa6-499f-a364-ff126aa6ea0c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f94x6m"
Sep 14 03:03:18.791262 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:03:18.791206    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e3a9df9-6fa6-499f-a364-ff126aa6ea0c-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f94x6m\" (UID: \"3e3a9df9-6fa6-499f-a364-ff126aa6ea0c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f94x6m"
Sep 14 03:03:18.891968 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:03:18.891938    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e3a9df9-6fa6-499f-a364-ff126aa6ea0c-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f94x6m\" (UID: \"3e3a9df9-6fa6-499f-a364-ff126aa6ea0c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f94x6m"
Sep 14 03:03:18.892101 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:03:18.891993    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e3a9df9-6fa6-499f-a364-ff126aa6ea0c-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f94x6m\" (UID: \"3e3a9df9-6fa6-499f-a364-ff126aa6ea0c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f94x6m"
Sep 14 03:03:18.892101 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:03:18.892052    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e3a9df9-6fa6-499f-a364-ff126aa6ea0c-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f94x6m\" (UID: \"3e3a9df9-6fa6-499f-a364-ff126aa6ea0c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f94x6m"
Sep 14 03:03:18.892101 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:03:18.892084    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28g46\" (UniqueName: \"kubernetes.io/projected/3e3a9df9-6fa6-499f-a364-ff126aa6ea0c-kube-api-access-28g46\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f94x6m\" (UID: \"3e3a9df9-6fa6-499f-a364-ff126aa6ea0c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f94x6m"
Sep 14 03:03:18.892253 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:03:18.892140    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e3a9df9-6fa6-499f-a364-ff126aa6ea0c-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f94x6m\" (UID: \"3e3a9df9-6fa6-499f-a364-ff126aa6ea0c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f94x6m"
Sep 14 03:03:18.892253 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:03:18.892176    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e3a9df9-6fa6-499f-a364-ff126aa6ea0c-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f94x6m\" (UID: \"3e3a9df9-6fa6-499f-a364-ff126aa6ea0c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f94x6m"
Sep 14 03:03:18.892554 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:03:18.892517    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e3a9df9-6fa6-499f-a364-ff126aa6ea0c-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f94x6m\" (UID: \"3e3a9df9-6fa6-499f-a364-ff126aa6ea0c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f94x6m"
Sep 14 03:03:18.892648 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:03:18.892555    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e3a9df9-6fa6-499f-a364-ff126aa6ea0c-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f94x6m\" (UID: \"3e3a9df9-6fa6-499f-a364-ff126aa6ea0c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f94x6m"
Sep 14 03:03:18.892691 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:03:18.892637    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e3a9df9-6fa6-499f-a364-ff126aa6ea0c-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f94x6m\" (UID: \"3e3a9df9-6fa6-499f-a364-ff126aa6ea0c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f94x6m"
Sep 14 03:03:18.894260 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:03:18.894241    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e3a9df9-6fa6-499f-a364-ff126aa6ea0c-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f94x6m\" (UID: \"3e3a9df9-6fa6-499f-a364-ff126aa6ea0c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f94x6m"
Sep 14 03:03:18.894549 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:03:18.894532    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e3a9df9-6fa6-499f-a364-ff126aa6ea0c-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f94x6m\" (UID: \"3e3a9df9-6fa6-499f-a364-ff126aa6ea0c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f94x6m"
Sep 14 03:03:18.912012 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:03:18.911993    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28g46\" (UniqueName: \"kubernetes.io/projected/3e3a9df9-6fa6-499f-a364-ff126aa6ea0c-kube-api-access-28g46\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f94x6m\" (UID: \"3e3a9df9-6fa6-499f-a364-ff126aa6ea0c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f94x6m"
Sep 14 03:03:19.058955 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:03:19.058917    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f94x6m"
Sep 14 03:03:19.329156 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:03:19.329127    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f94x6m"]
Sep 14 03:03:19.330622 ip-10-0-141-4 kubenswrapper[2564]: W0914 03:03:19.330593    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e3a9df9_6fa6_499f_a364_ff126aa6ea0c.slice/crio-72ccb38d4d7e3bf0c46be2abbb7dd599094448d9be861be6a8e060f2b26a4a53 WatchSource:0}: Error finding container 72ccb38d4d7e3bf0c46be2abbb7dd599094448d9be861be6a8e060f2b26a4a53: Status 404 returned error can't find the container with id 72ccb38d4d7e3bf0c46be2abbb7dd599094448d9be861be6a8e060f2b26a4a53
Sep 14 03:03:20.187708 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:03:20.187672    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f94x6m" event={"ID":"3e3a9df9-6fa6-499f-a364-ff126aa6ea0c","Type":"ContainerStarted","Data":"c139f601b80d8599695bb84ab67fdea89d92163b57be8f96fc21dbcfa85cb213"}
Sep 14 03:03:20.188137 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:03:20.187726    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f94x6m" event={"ID":"3e3a9df9-6fa6-499f-a364-ff126aa6ea0c","Type":"ContainerStarted","Data":"72ccb38d4d7e3bf0c46be2abbb7dd599094448d9be861be6a8e060f2b26a4a53"}
Sep 14 03:03:20.210198 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:03:20.210151    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f94x6m" podStartSLOduration=2.011613763 podStartE2EDuration="2.210135174s" totalImagesPullingTime="198.521411ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:03:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:03:20.207895609 +0000 UTC m=+645.588979808" watchObservedRunningTime="2026-09-14 03:03:20.210135174 +0000 UTC m=+645.591219372"
Sep 14 03:03:29.059170 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:03:29.059135    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f94x6m"
Sep 14 03:03:29.059724 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:03:29.059208    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f94x6m"
Sep 14 03:03:29.071564 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:03:29.071537    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f94x6m"
Sep 14 03:03:29.240352 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:03:29.240325    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f94x6m"
Sep 14 03:03:31.903147 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:03:31.903114    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:03:39.437898 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:03:39.437856    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:03:42.898529 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:03:42.898493    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:04:31.140697 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:04:31.140614    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5f4895b846-pp7ks"]
Sep 14 03:04:31.143932 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:04:31.143915    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f4895b846-pp7ks"
Sep 14 03:04:31.146935 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:04:31.146910    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-9z7kg\""
Sep 14 03:04:31.148028 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:04:31.148009    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 14 03:04:31.155654 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:04:31.155631    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f4895b846-pp7ks"]
Sep 14 03:04:31.248670 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:04:31.248637    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/411c0296-8329-4d08-9902-3255d8463066-tls-cert\") pod \"authorino-5f4895b846-pp7ks\" (UID: \"411c0296-8329-4d08-9902-3255d8463066\") " pod="kuadrant-system/authorino-5f4895b846-pp7ks"
Sep 14 03:04:31.248835 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:04:31.248688    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf2x5\" (UniqueName: \"kubernetes.io/projected/411c0296-8329-4d08-9902-3255d8463066-kube-api-access-gf2x5\") pod \"authorino-5f4895b846-pp7ks\" (UID: \"411c0296-8329-4d08-9902-3255d8463066\") " pod="kuadrant-system/authorino-5f4895b846-pp7ks"
Sep 14 03:04:31.349799 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:04:31.349767    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf2x5\" (UniqueName: \"kubernetes.io/projected/411c0296-8329-4d08-9902-3255d8463066-kube-api-access-gf2x5\") pod \"authorino-5f4895b846-pp7ks\" (UID: \"411c0296-8329-4d08-9902-3255d8463066\") " pod="kuadrant-system/authorino-5f4895b846-pp7ks"
Sep 14 03:04:31.349953 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:04:31.349861    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/411c0296-8329-4d08-9902-3255d8463066-tls-cert\") pod \"authorino-5f4895b846-pp7ks\" (UID: \"411c0296-8329-4d08-9902-3255d8463066\") " pod="kuadrant-system/authorino-5f4895b846-pp7ks"
Sep 14 03:04:31.352241 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:04:31.352215    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/411c0296-8329-4d08-9902-3255d8463066-tls-cert\") pod \"authorino-5f4895b846-pp7ks\" (UID: \"411c0296-8329-4d08-9902-3255d8463066\") " pod="kuadrant-system/authorino-5f4895b846-pp7ks"
Sep 14 03:04:31.359113 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:04:31.359084    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf2x5\" (UniqueName: \"kubernetes.io/projected/411c0296-8329-4d08-9902-3255d8463066-kube-api-access-gf2x5\") pod \"authorino-5f4895b846-pp7ks\" (UID: \"411c0296-8329-4d08-9902-3255d8463066\") " pod="kuadrant-system/authorino-5f4895b846-pp7ks"
Sep 14 03:04:31.453188 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:04:31.453109    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f4895b846-pp7ks"
Sep 14 03:04:31.724324 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:04:31.724239    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f4895b846-pp7ks"]
Sep 14 03:04:31.727425 ip-10-0-141-4 kubenswrapper[2564]: W0914 03:04:31.727367    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod411c0296_8329_4d08_9902_3255d8463066.slice/crio-09afac5581d4791acdb9e7f31fa47da2fab8840e3bac1e7616f81df39142db27 WatchSource:0}: Error finding container 09afac5581d4791acdb9e7f31fa47da2fab8840e3bac1e7616f81df39142db27: Status 404 returned error can't find the container with id 09afac5581d4791acdb9e7f31fa47da2fab8840e3bac1e7616f81df39142db27
Sep 14 03:04:32.538467 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:04:32.538431    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f4895b846-pp7ks" event={"ID":"411c0296-8329-4d08-9902-3255d8463066","Type":"ContainerStarted","Data":"09afac5581d4791acdb9e7f31fa47da2fab8840e3bac1e7616f81df39142db27"}
Sep 14 03:04:43.594708 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:04:43.594669    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f4895b846-pp7ks" event={"ID":"411c0296-8329-4d08-9902-3255d8463066","Type":"ContainerStarted","Data":"b12aabe911cd0e2409ac268a83cbe6ed03376f9e3490af1eabe632abda6ce004"}
Sep 14 03:04:43.616236 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:04:43.616184    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5f4895b846-pp7ks" podStartSLOduration=1.1447162180000001 podStartE2EDuration="12.616167997s" totalImagesPullingTime="11.471451779s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:04:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:04:43.614117769 +0000 UTC m=+728.995201969" watchObservedRunningTime="2026-09-14 03:04:43.616167997 +0000 UTC m=+728.997252195"
Sep 14 03:04:44.423421 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:04:44.423369    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:04:44.550492 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:04:44.550441    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:04:45.467558 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:04:45.467515    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:04:45.558333 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:04:45.558279    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:00.713501 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:00.713462    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:00.823976 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:00.823939    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:00.930034 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:00.929997    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:01.210692 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:01.210654    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:01.491824 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:01.491718    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:01.617797 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:01.617753    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:01.899664 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:01.899620    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:02.016372 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:02.016336    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:02.227535 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:02.227441    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:02.332258 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:02.332214    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:02.738375 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:02.738335    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:04.866765 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:04.866715    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:05.083878 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:05.083842    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:05.218986 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:05.218893    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:05.599194 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:05.599161    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7683-af10df"]
Sep 14 03:05:05.602877 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:05.602854    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7683-af10df"
Sep 14 03:05:05.606582 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:05.606547    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-2qhmh\""
Sep 14 03:05:05.618936 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:05.618889    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7683-af10df"]
Sep 14 03:05:05.679865 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:05.679821    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvjcz\" (UniqueName: \"kubernetes.io/projected/98e9fdbb-defe-4042-ae97-15b895833078-kube-api-access-pvjcz\") pod \"test-curl-7683-af10df\" (UID: \"98e9fdbb-defe-4042-ae97-15b895833078\") " pod="opendatahub/test-curl-7683-af10df"
Sep 14 03:05:05.780366 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:05.780331    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvjcz\" (UniqueName: \"kubernetes.io/projected/98e9fdbb-defe-4042-ae97-15b895833078-kube-api-access-pvjcz\") pod \"test-curl-7683-af10df\" (UID: \"98e9fdbb-defe-4042-ae97-15b895833078\") " pod="opendatahub/test-curl-7683-af10df"
Sep 14 03:05:05.842373 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:05.842343    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvjcz\" (UniqueName: \"kubernetes.io/projected/98e9fdbb-defe-4042-ae97-15b895833078-kube-api-access-pvjcz\") pod \"test-curl-7683-af10df\" (UID: \"98e9fdbb-defe-4042-ae97-15b895833078\") " pod="opendatahub/test-curl-7683-af10df"
Sep 14 03:05:05.912607 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:05.912522    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7683-af10df"
Sep 14 03:05:06.204518 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:06.204445    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7683-af10df"]
Sep 14 03:05:06.207733 ip-10-0-141-4 kubenswrapper[2564]: W0914 03:05:06.207702    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod98e9fdbb_defe_4042_ae97_15b895833078.slice/crio-f8784f0fa5353e9c474cc53d32c2bd85efdd09c61a6b964048e2895fa05f7f44 WatchSource:0}: Error finding container f8784f0fa5353e9c474cc53d32c2bd85efdd09c61a6b964048e2895fa05f7f44: Status 404 returned error can't find the container with id f8784f0fa5353e9c474cc53d32c2bd85efdd09c61a6b964048e2895fa05f7f44
Sep 14 03:05:06.652223 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:06.652188    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-9c9f1e-8497955c59-k5c6b"]
Sep 14 03:05:06.657453 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:06.657429    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9c9f1e-8497955c59-k5c6b"
Sep 14 03:05:06.660924 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:06.660896    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-9c9f1e-serving-cert\""
Sep 14 03:05:06.678753 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:06.678726    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-9c9f1e-8497955c59-k5c6b"]
Sep 14 03:05:06.690340 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:06.690314    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szqxx\" (UniqueName: \"kubernetes.io/projected/0940ece5-1e43-46ce-86d4-3bc31c3777a2-kube-api-access-szqxx\") pod \"maas-api-e2e-worker-w4-9c9f1e-8497955c59-k5c6b\" (UID: \"0940ece5-1e43-46ce-86d4-3bc31c3777a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9c9f1e-8497955c59-k5c6b"
Sep 14 03:05:06.690476 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:06.690376    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0940ece5-1e43-46ce-86d4-3bc31c3777a2-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-9c9f1e-8497955c59-k5c6b\" (UID: \"0940ece5-1e43-46ce-86d4-3bc31c3777a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9c9f1e-8497955c59-k5c6b"
Sep 14 03:05:06.690476 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:06.690448    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0940ece5-1e43-46ce-86d4-3bc31c3777a2-maas-api-tls\") pod \"maas-api-e2e-worker-w4-9c9f1e-8497955c59-k5c6b\" (UID: \"0940ece5-1e43-46ce-86d4-3bc31c3777a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9c9f1e-8497955c59-k5c6b"
Sep 14 03:05:06.690602 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:06.690483    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0940ece5-1e43-46ce-86d4-3bc31c3777a2-metrics-certs\") pod \"maas-api-e2e-worker-w4-9c9f1e-8497955c59-k5c6b\" (UID: \"0940ece5-1e43-46ce-86d4-3bc31c3777a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9c9f1e-8497955c59-k5c6b"
Sep 14 03:05:06.716077 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:06.716044    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7683-af10df" event={"ID":"98e9fdbb-defe-4042-ae97-15b895833078","Type":"ContainerStarted","Data":"f8784f0fa5353e9c474cc53d32c2bd85efdd09c61a6b964048e2895fa05f7f44"}
Sep 14 03:05:06.740370 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:06.740316    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:06.791560 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:06.791514    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szqxx\" (UniqueName: \"kubernetes.io/projected/0940ece5-1e43-46ce-86d4-3bc31c3777a2-kube-api-access-szqxx\") pod \"maas-api-e2e-worker-w4-9c9f1e-8497955c59-k5c6b\" (UID: \"0940ece5-1e43-46ce-86d4-3bc31c3777a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9c9f1e-8497955c59-k5c6b"
Sep 14 03:05:06.791745 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:06.791581    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0940ece5-1e43-46ce-86d4-3bc31c3777a2-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-9c9f1e-8497955c59-k5c6b\" (UID: \"0940ece5-1e43-46ce-86d4-3bc31c3777a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9c9f1e-8497955c59-k5c6b"
Sep 14 03:05:06.791745 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:06.791624    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0940ece5-1e43-46ce-86d4-3bc31c3777a2-maas-api-tls\") pod \"maas-api-e2e-worker-w4-9c9f1e-8497955c59-k5c6b\" (UID: \"0940ece5-1e43-46ce-86d4-3bc31c3777a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9c9f1e-8497955c59-k5c6b"
Sep 14 03:05:06.791745 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:06.791656    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0940ece5-1e43-46ce-86d4-3bc31c3777a2-metrics-certs\") pod \"maas-api-e2e-worker-w4-9c9f1e-8497955c59-k5c6b\" (UID: \"0940ece5-1e43-46ce-86d4-3bc31c3777a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9c9f1e-8497955c59-k5c6b"
Sep 14 03:05:06.792350 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:06.792326    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0940ece5-1e43-46ce-86d4-3bc31c3777a2-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-9c9f1e-8497955c59-k5c6b\" (UID: \"0940ece5-1e43-46ce-86d4-3bc31c3777a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9c9f1e-8497955c59-k5c6b"
Sep 14 03:05:06.794011 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:06.793983    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0940ece5-1e43-46ce-86d4-3bc31c3777a2-maas-api-tls\") pod \"maas-api-e2e-worker-w4-9c9f1e-8497955c59-k5c6b\" (UID: \"0940ece5-1e43-46ce-86d4-3bc31c3777a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9c9f1e-8497955c59-k5c6b"
Sep 14 03:05:06.794156 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:06.794137    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0940ece5-1e43-46ce-86d4-3bc31c3777a2-metrics-certs\") pod \"maas-api-e2e-worker-w4-9c9f1e-8497955c59-k5c6b\" (UID: \"0940ece5-1e43-46ce-86d4-3bc31c3777a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9c9f1e-8497955c59-k5c6b"
Sep 14 03:05:06.812211 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:06.812187    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szqxx\" (UniqueName: \"kubernetes.io/projected/0940ece5-1e43-46ce-86d4-3bc31c3777a2-kube-api-access-szqxx\") pod \"maas-api-e2e-worker-w4-9c9f1e-8497955c59-k5c6b\" (UID: \"0940ece5-1e43-46ce-86d4-3bc31c3777a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9c9f1e-8497955c59-k5c6b"
Sep 14 03:05:06.843267 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:06.843231    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:06.969982 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:06.969893    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9c9f1e-8497955c59-k5c6b"
Sep 14 03:05:07.253714 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:07.253687    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-9c9f1e-8497955c59-k5c6b"]
Sep 14 03:05:07.255090 ip-10-0-141-4 kubenswrapper[2564]: W0914 03:05:07.255059    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0940ece5_1e43_46ce_86d4_3bc31c3777a2.slice/crio-1b12a7ab057325ed0eaed544ae86908d2cf9900dc5b42c6535d58b79432bc9fe WatchSource:0}: Error finding container 1b12a7ab057325ed0eaed544ae86908d2cf9900dc5b42c6535d58b79432bc9fe: Status 404 returned error can't find the container with id 1b12a7ab057325ed0eaed544ae86908d2cf9900dc5b42c6535d58b79432bc9fe
Sep 14 03:05:07.359761 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:07.359726    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:07.451829 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:07.451785    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:07.614631 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:07.614594    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-9c9f1e-785c744797-7hjth"]
Sep 14 03:05:07.618928 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:07.618900    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-9c9f1e-785c744797-7hjth"
Sep 14 03:05:07.624127 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:07.624106    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-9c9f1e\""
Sep 14 03:05:07.624399 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:07.624377    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-9c9f1e-dockercfg-4czcr\""
Sep 14 03:05:07.638363 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:07.638320    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-9c9f1e-785c744797-7hjth"]
Sep 14 03:05:07.702072 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:07.702049    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x964n\" (UniqueName: \"kubernetes.io/projected/cd75830d-dc18-47e2-a0fc-7ea2e0cd21b4-kube-api-access-x964n\") pod \"payload-processing-e2e-worker-w4-9c9f1e-785c744797-7hjth\" (UID: \"cd75830d-dc18-47e2-a0fc-7ea2e0cd21b4\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-9c9f1e-785c744797-7hjth"
Sep 14 03:05:07.702206 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:07.702129    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd75830d-dc18-47e2-a0fc-7ea2e0cd21b4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-9c9f1e-785c744797-7hjth\" (UID: \"cd75830d-dc18-47e2-a0fc-7ea2e0cd21b4\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-9c9f1e-785c744797-7hjth"
Sep 14 03:05:07.724598 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:07.724566    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9c9f1e-8497955c59-k5c6b" event={"ID":"0940ece5-1e43-46ce-86d4-3bc31c3777a2","Type":"ContainerStarted","Data":"1b12a7ab057325ed0eaed544ae86908d2cf9900dc5b42c6535d58b79432bc9fe"}
Sep 14 03:05:07.802792 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:07.802759    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x964n\" (UniqueName: \"kubernetes.io/projected/cd75830d-dc18-47e2-a0fc-7ea2e0cd21b4-kube-api-access-x964n\") pod \"payload-processing-e2e-worker-w4-9c9f1e-785c744797-7hjth\" (UID: \"cd75830d-dc18-47e2-a0fc-7ea2e0cd21b4\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-9c9f1e-785c744797-7hjth"
Sep 14 03:05:07.802964 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:07.802836    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd75830d-dc18-47e2-a0fc-7ea2e0cd21b4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-9c9f1e-785c744797-7hjth\" (UID: \"cd75830d-dc18-47e2-a0fc-7ea2e0cd21b4\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-9c9f1e-785c744797-7hjth"
Sep 14 03:05:07.803556 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:07.803531    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd75830d-dc18-47e2-a0fc-7ea2e0cd21b4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-9c9f1e-785c744797-7hjth\" (UID: \"cd75830d-dc18-47e2-a0fc-7ea2e0cd21b4\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-9c9f1e-785c744797-7hjth"
Sep 14 03:05:07.818004 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:07.817982    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x964n\" (UniqueName: \"kubernetes.io/projected/cd75830d-dc18-47e2-a0fc-7ea2e0cd21b4-kube-api-access-x964n\") pod \"payload-processing-e2e-worker-w4-9c9f1e-785c744797-7hjth\" (UID: \"cd75830d-dc18-47e2-a0fc-7ea2e0cd21b4\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-9c9f1e-785c744797-7hjth"
Sep 14 03:05:07.909952 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:07.909865    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f002a1-7f464f54d5-xntnr"]
Sep 14 03:05:07.916005 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:07.915589    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f002a1-7f464f54d5-xntnr"
Sep 14 03:05:07.920720 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:07.920686    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-f002a1-serving-cert\""
Sep 14 03:05:07.932802 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:07.932642    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-9c9f1e-785c744797-7hjth"
Sep 14 03:05:07.950543 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:07.950514    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f002a1-7f464f54d5-xntnr"]
Sep 14 03:05:08.004957 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:08.004926    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59691064-4c10-4ec6-99f3-5c0c8b30209e-metrics-certs\") pod \"maas-api-e2e-inf-a-f002a1-7f464f54d5-xntnr\" (UID: \"59691064-4c10-4ec6-99f3-5c0c8b30209e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f002a1-7f464f54d5-xntnr"
Sep 14 03:05:08.005390 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:08.005001    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59691064-4c10-4ec6-99f3-5c0c8b30209e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f002a1-7f464f54d5-xntnr\" (UID: \"59691064-4c10-4ec6-99f3-5c0c8b30209e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f002a1-7f464f54d5-xntnr"
Sep 14 03:05:08.005390 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:08.005075    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s74gl\" (UniqueName: \"kubernetes.io/projected/59691064-4c10-4ec6-99f3-5c0c8b30209e-kube-api-access-s74gl\") pod \"maas-api-e2e-inf-a-f002a1-7f464f54d5-xntnr\" (UID: \"59691064-4c10-4ec6-99f3-5c0c8b30209e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f002a1-7f464f54d5-xntnr"
Sep 14 03:05:08.005390 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:08.005110    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59691064-4c10-4ec6-99f3-5c0c8b30209e-maas-api-tls\") pod \"maas-api-e2e-inf-a-f002a1-7f464f54d5-xntnr\" (UID: \"59691064-4c10-4ec6-99f3-5c0c8b30209e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f002a1-7f464f54d5-xntnr"
Sep 14 03:05:08.046745 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:08.046708    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:08.105927 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:08.105875    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59691064-4c10-4ec6-99f3-5c0c8b30209e-metrics-certs\") pod \"maas-api-e2e-inf-a-f002a1-7f464f54d5-xntnr\" (UID: \"59691064-4c10-4ec6-99f3-5c0c8b30209e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f002a1-7f464f54d5-xntnr"
Sep 14 03:05:08.106110 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:08.105994    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59691064-4c10-4ec6-99f3-5c0c8b30209e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f002a1-7f464f54d5-xntnr\" (UID: \"59691064-4c10-4ec6-99f3-5c0c8b30209e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f002a1-7f464f54d5-xntnr"
Sep 14 03:05:08.106110 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:08.106033    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s74gl\" (UniqueName: \"kubernetes.io/projected/59691064-4c10-4ec6-99f3-5c0c8b30209e-kube-api-access-s74gl\") pod \"maas-api-e2e-inf-a-f002a1-7f464f54d5-xntnr\" (UID: \"59691064-4c10-4ec6-99f3-5c0c8b30209e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f002a1-7f464f54d5-xntnr"
Sep 14 03:05:08.106110 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:08.106071    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59691064-4c10-4ec6-99f3-5c0c8b30209e-maas-api-tls\") pod \"maas-api-e2e-inf-a-f002a1-7f464f54d5-xntnr\" (UID: \"59691064-4c10-4ec6-99f3-5c0c8b30209e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f002a1-7f464f54d5-xntnr"
Sep 14 03:05:08.106286 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:05:08.106250    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-f002a1-serving-cert: secret "maas-api-e2e-inf-a-f002a1-serving-cert" not found
Sep 14 03:05:08.106340 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:05:08.106321    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59691064-4c10-4ec6-99f3-5c0c8b30209e-maas-api-tls podName:59691064-4c10-4ec6-99f3-5c0c8b30209e nodeName:}" failed. No retries permitted until 2026-09-14 03:05:08.606299625 +0000 UTC m=+753.987383826 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/59691064-4c10-4ec6-99f3-5c0c8b30209e-maas-api-tls") pod "maas-api-e2e-inf-a-f002a1-7f464f54d5-xntnr" (UID: "59691064-4c10-4ec6-99f3-5c0c8b30209e") : secret "maas-api-e2e-inf-a-f002a1-serving-cert" not found
Sep 14 03:05:08.106836 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:08.106815    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59691064-4c10-4ec6-99f3-5c0c8b30209e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f002a1-7f464f54d5-xntnr\" (UID: \"59691064-4c10-4ec6-99f3-5c0c8b30209e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f002a1-7f464f54d5-xntnr"
Sep 14 03:05:08.108752 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:08.108730    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59691064-4c10-4ec6-99f3-5c0c8b30209e-metrics-certs\") pod \"maas-api-e2e-inf-a-f002a1-7f464f54d5-xntnr\" (UID: \"59691064-4c10-4ec6-99f3-5c0c8b30209e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f002a1-7f464f54d5-xntnr"
Sep 14 03:05:08.116778 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:08.116745    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s74gl\" (UniqueName: \"kubernetes.io/projected/59691064-4c10-4ec6-99f3-5c0c8b30209e-kube-api-access-s74gl\") pod \"maas-api-e2e-inf-a-f002a1-7f464f54d5-xntnr\" (UID: \"59691064-4c10-4ec6-99f3-5c0c8b30209e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f002a1-7f464f54d5-xntnr"
Sep 14 03:05:08.156184 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:08.156065    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:08.249692 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:08.249609    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:08.610955 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:08.610894    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59691064-4c10-4ec6-99f3-5c0c8b30209e-maas-api-tls\") pod \"maas-api-e2e-inf-a-f002a1-7f464f54d5-xntnr\" (UID: \"59691064-4c10-4ec6-99f3-5c0c8b30209e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f002a1-7f464f54d5-xntnr"
Sep 14 03:05:08.613890 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:08.613687    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59691064-4c10-4ec6-99f3-5c0c8b30209e-maas-api-tls\") pod \"maas-api-e2e-inf-a-f002a1-7f464f54d5-xntnr\" (UID: \"59691064-4c10-4ec6-99f3-5c0c8b30209e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f002a1-7f464f54d5-xntnr"
Sep 14 03:05:08.732958 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:08.732918    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9c9f1e-8497955c59-k5c6b" event={"ID":"0940ece5-1e43-46ce-86d4-3bc31c3777a2","Type":"ContainerStarted","Data":"a04e8f1e6fa7b6e849244550691537e271e91d8259d260dd7e0f39f7e42ec75f"}
Sep 14 03:05:08.733127 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:08.733020    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9c9f1e-8497955c59-k5c6b"
Sep 14 03:05:08.734801 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:08.734743    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7683-af10df" event={"ID":"98e9fdbb-defe-4042-ae97-15b895833078","Type":"ContainerStarted","Data":"1c028340904753ae0aa94ad249424fa08fd9d073ad7abf68786c280b47c0bbfe"}
Sep 14 03:05:08.754661 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:08.754613    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9c9f1e-8497955c59-k5c6b" podStartSLOduration=2.343163702 podStartE2EDuration="2.754599663s" totalImagesPullingTime="411.435961ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:05:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:05:08.751820924 +0000 UTC m=+754.132905138" watchObservedRunningTime="2026-09-14 03:05:08.754599663 +0000 UTC m=+754.135683862"
Sep 14 03:05:08.772469 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:08.772374    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7683-af10df" podStartSLOduration=1.345849179 podStartE2EDuration="3.772334135s" totalImagesPullingTime="2.426484956s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:05:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:05:08.771604956 +0000 UTC m=+754.152689152" watchObservedRunningTime="2026-09-14 03:05:08.772334135 +0000 UTC m=+754.153418335"
Sep 14 03:05:08.832803 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:08.832721    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f002a1-7f464f54d5-xntnr"
Sep 14 03:05:08.840605 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:08.840577    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-9c9f1e-785c744797-7hjth"]
Sep 14 03:05:08.843609 ip-10-0-141-4 kubenswrapper[2564]: W0914 03:05:08.843581    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd75830d_dc18_47e2_a0fc_7ea2e0cd21b4.slice/crio-6d8a158647c871e6ba0c12838ef944c1aeb84379515bd3216ccaf30d61e82409 WatchSource:0}: Error finding container 6d8a158647c871e6ba0c12838ef944c1aeb84379515bd3216ccaf30d61e82409: Status 404 returned error can't find the container with id 6d8a158647c871e6ba0c12838ef944c1aeb84379515bd3216ccaf30d61e82409
Sep 14 03:05:08.867522 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:08.867492    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:08.969706 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:08.969647    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:09.165016 ip-10-0-141-4 kubenswrapper[2564]: W0914 03:05:09.164985    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59691064_4c10_4ec6_99f3_5c0c8b30209e.slice/crio-7c5c3d0f916cd9173509029c7189d17c000f2405ee0db2a0dbb196efef8647c9 WatchSource:0}: Error finding container 7c5c3d0f916cd9173509029c7189d17c000f2405ee0db2a0dbb196efef8647c9: Status 404 returned error can't find the container with id 7c5c3d0f916cd9173509029c7189d17c000f2405ee0db2a0dbb196efef8647c9
Sep 14 03:05:09.165846 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:09.165814    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f002a1-7f464f54d5-xntnr"]
Sep 14 03:05:09.508302 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:09.508217    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7683-af10df"]
Sep 14 03:05:09.515819 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:09.515790    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:09.521901 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:09.521876    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7683-af10df"]
Sep 14 03:05:09.741022 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:09.740984    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f002a1-7f464f54d5-xntnr" event={"ID":"59691064-4c10-4ec6-99f3-5c0c8b30209e","Type":"ContainerStarted","Data":"31e9db45970b4a5aace94536f7ab411348df20af6ee9253bbdce99fe7c81dddd"}
Sep 14 03:05:09.741200 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:09.741042    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f002a1-7f464f54d5-xntnr" event={"ID":"59691064-4c10-4ec6-99f3-5c0c8b30209e","Type":"ContainerStarted","Data":"7c5c3d0f916cd9173509029c7189d17c000f2405ee0db2a0dbb196efef8647c9"}
Sep 14 03:05:09.741200 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:09.741098    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f002a1-7f464f54d5-xntnr"
Sep 14 03:05:09.742737 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:09.742707    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-9c9f1e-785c744797-7hjth" event={"ID":"cd75830d-dc18-47e2-a0fc-7ea2e0cd21b4","Type":"ContainerStarted","Data":"d524a37470cb1eedfdefd675865381f05f4565893b661c8a7c532237db1d57d7"}
Sep 14 03:05:09.742885 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:09.742758    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-9c9f1e-785c744797-7hjth" event={"ID":"cd75830d-dc18-47e2-a0fc-7ea2e0cd21b4","Type":"ContainerStarted","Data":"6d8a158647c871e6ba0c12838ef944c1aeb84379515bd3216ccaf30d61e82409"}
Sep 14 03:05:09.742976 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:05:09.742955    2564 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7683-af10df" container="test-curl-7683-af10df"
Sep 14 03:05:09.743047 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:09.743008    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-9c9f1e-785c744797-7hjth"
Sep 14 03:05:09.764774 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:09.764671    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f002a1-7f464f54d5-xntnr" podStartSLOduration=2.31659645 podStartE2EDuration="2.764650973s" totalImagesPullingTime="448.054523ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:05:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:05:09.763735663 +0000 UTC m=+755.144819864" watchObservedRunningTime="2026-09-14 03:05:09.764650973 +0000 UTC m=+755.145735174"
Sep 14 03:05:09.796473 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:09.792789    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-9c9f1e-785c744797-7hjth" podStartSLOduration=2.792768464 podStartE2EDuration="2.792768464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:05:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:05:09.788462092 +0000 UTC m=+755.169546292" watchObservedRunningTime="2026-09-14 03:05:09.792768464 +0000 UTC m=+755.173852664"
Sep 14 03:05:10.057692 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:10.057658    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:10.574143 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:10.574112    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:10.748316 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:10.748273    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7683-af10df" podUID="98e9fdbb-defe-4042-ae97-15b895833078" containerName="test-curl-7683-af10df" containerID="cri-o://1c028340904753ae0aa94ad249424fa08fd9d073ad7abf68786c280b47c0bbfe" gracePeriod=2
Sep 14 03:05:10.750699 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:05:10.750670    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7683-af10df\" is forbidden: User \"system:node:ip-10-0-141-4.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-4.ec2.internal' and this object" podUID="98e9fdbb-defe-4042-ae97-15b895833078" pod="opendatahub/test-curl-7683-af10df"
Sep 14 03:05:10.978332 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:10.978237    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:11.070804 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:11.070773    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:11.481783 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:11.481744    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:11.582351 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:11.582313    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:11.740732 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:11.740638    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:11.966241 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:11.966207    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-3a752f-549955bb4-tbwgt"]
Sep 14 03:05:11.966743 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:11.966726    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98e9fdbb-defe-4042-ae97-15b895833078" containerName="test-curl-7683-af10df"
Sep 14 03:05:11.966795 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:11.966746    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="98e9fdbb-defe-4042-ae97-15b895833078" containerName="test-curl-7683-af10df"
Sep 14 03:05:11.966830 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:11.966818    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="98e9fdbb-defe-4042-ae97-15b895833078" containerName="test-curl-7683-af10df"
Sep 14 03:05:11.970360 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:11.970337    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3a752f-549955bb4-tbwgt"
Sep 14 03:05:11.974775 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:11.974752    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-3a752f-serving-cert\""
Sep 14 03:05:11.991443 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:11.991357    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-3a752f-549955bb4-tbwgt"]
Sep 14 03:05:12.047332 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:12.047301    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac41e03a-54d4-4b59-8e31-563773bd52ba-metrics-certs\") pod \"maas-api-e2e-worker-w0-3a752f-549955bb4-tbwgt\" (UID: \"ac41e03a-54d4-4b59-8e31-563773bd52ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3a752f-549955bb4-tbwgt"
Sep 14 03:05:12.047562 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:12.047363    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac41e03a-54d4-4b59-8e31-563773bd52ba-maas-api-tls\") pod \"maas-api-e2e-worker-w0-3a752f-549955bb4-tbwgt\" (UID: \"ac41e03a-54d4-4b59-8e31-563773bd52ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3a752f-549955bb4-tbwgt"
Sep 14 03:05:12.047562 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:12.047383    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l2md\" (UniqueName: \"kubernetes.io/projected/ac41e03a-54d4-4b59-8e31-563773bd52ba-kube-api-access-2l2md\") pod \"maas-api-e2e-worker-w0-3a752f-549955bb4-tbwgt\" (UID: \"ac41e03a-54d4-4b59-8e31-563773bd52ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3a752f-549955bb4-tbwgt"
Sep 14 03:05:12.047562 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:12.047473    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac41e03a-54d4-4b59-8e31-563773bd52ba-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-3a752f-549955bb4-tbwgt\" (UID: \"ac41e03a-54d4-4b59-8e31-563773bd52ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3a752f-549955bb4-tbwgt"
Sep 14 03:05:12.058941 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:12.058895    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:12.148441 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:12.148382    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac41e03a-54d4-4b59-8e31-563773bd52ba-maas-api-tls\") pod \"maas-api-e2e-worker-w0-3a752f-549955bb4-tbwgt\" (UID: \"ac41e03a-54d4-4b59-8e31-563773bd52ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3a752f-549955bb4-tbwgt"
Sep 14 03:05:12.148628 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:12.148451    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l2md\" (UniqueName: \"kubernetes.io/projected/ac41e03a-54d4-4b59-8e31-563773bd52ba-kube-api-access-2l2md\") pod \"maas-api-e2e-worker-w0-3a752f-549955bb4-tbwgt\" (UID: \"ac41e03a-54d4-4b59-8e31-563773bd52ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3a752f-549955bb4-tbwgt"
Sep 14 03:05:12.148628 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:12.148500    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac41e03a-54d4-4b59-8e31-563773bd52ba-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-3a752f-549955bb4-tbwgt\" (UID: \"ac41e03a-54d4-4b59-8e31-563773bd52ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3a752f-549955bb4-tbwgt"
Sep 14 03:05:12.148628 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:05:12.148534    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-3a752f-serving-cert: secret "maas-api-e2e-worker-w0-3a752f-serving-cert" not found
Sep 14 03:05:12.148628 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:05:12.148607    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac41e03a-54d4-4b59-8e31-563773bd52ba-maas-api-tls podName:ac41e03a-54d4-4b59-8e31-563773bd52ba nodeName:}" failed. No retries permitted until 2026-09-14 03:05:12.648583854 +0000 UTC m=+758.029668046 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ac41e03a-54d4-4b59-8e31-563773bd52ba-maas-api-tls") pod "maas-api-e2e-worker-w0-3a752f-549955bb4-tbwgt" (UID: "ac41e03a-54d4-4b59-8e31-563773bd52ba") : secret "maas-api-e2e-worker-w0-3a752f-serving-cert" not found
Sep 14 03:05:12.148843 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:12.148645    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac41e03a-54d4-4b59-8e31-563773bd52ba-metrics-certs\") pod \"maas-api-e2e-worker-w0-3a752f-549955bb4-tbwgt\" (UID: \"ac41e03a-54d4-4b59-8e31-563773bd52ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3a752f-549955bb4-tbwgt"
Sep 14 03:05:12.149266 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:12.149248    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac41e03a-54d4-4b59-8e31-563773bd52ba-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-3a752f-549955bb4-tbwgt\" (UID: \"ac41e03a-54d4-4b59-8e31-563773bd52ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3a752f-549955bb4-tbwgt"
Sep 14 03:05:12.151144 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:12.151113    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac41e03a-54d4-4b59-8e31-563773bd52ba-metrics-certs\") pod \"maas-api-e2e-worker-w0-3a752f-549955bb4-tbwgt\" (UID: \"ac41e03a-54d4-4b59-8e31-563773bd52ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3a752f-549955bb4-tbwgt"
Sep 14 03:05:12.157501 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:12.157478    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l2md\" (UniqueName: \"kubernetes.io/projected/ac41e03a-54d4-4b59-8e31-563773bd52ba-kube-api-access-2l2md\") pod \"maas-api-e2e-worker-w0-3a752f-549955bb4-tbwgt\" (UID: \"ac41e03a-54d4-4b59-8e31-563773bd52ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3a752f-549955bb4-tbwgt"
Sep 14 03:05:12.165781 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:12.165758    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:12.654519 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:12.654485    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac41e03a-54d4-4b59-8e31-563773bd52ba-maas-api-tls\") pod \"maas-api-e2e-worker-w0-3a752f-549955bb4-tbwgt\" (UID: \"ac41e03a-54d4-4b59-8e31-563773bd52ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3a752f-549955bb4-tbwgt"
Sep 14 03:05:12.655846 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:12.655815    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:12.657222 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:12.657189    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac41e03a-54d4-4b59-8e31-563773bd52ba-maas-api-tls\") pod \"maas-api-e2e-worker-w0-3a752f-549955bb4-tbwgt\" (UID: \"ac41e03a-54d4-4b59-8e31-563773bd52ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3a752f-549955bb4-tbwgt"
Sep 14 03:05:12.798176 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:12.798119    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:12.881253 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:12.881001    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3a752f-549955bb4-tbwgt"
Sep 14 03:05:12.988987 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:12.988959    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7683-af10df"
Sep 14 03:05:12.993555 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:12.993525    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:13.063649 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:13.063615    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pvjcz\" (UniqueName: \"kubernetes.io/projected/98e9fdbb-defe-4042-ae97-15b895833078-kube-api-access-pvjcz\") pod \"98e9fdbb-defe-4042-ae97-15b895833078\" (UID: \"98e9fdbb-defe-4042-ae97-15b895833078\") "
Sep 14 03:05:13.065807 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:13.065774    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98e9fdbb-defe-4042-ae97-15b895833078-kube-api-access-pvjcz" (OuterVolumeSpecName: "kube-api-access-pvjcz") pod "98e9fdbb-defe-4042-ae97-15b895833078" (UID: "98e9fdbb-defe-4042-ae97-15b895833078"). InnerVolumeSpecName "kube-api-access-pvjcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:05:13.100196 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:13.100167    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98e9fdbb-defe-4042-ae97-15b895833078" path="/var/lib/kubelet/pods/98e9fdbb-defe-4042-ae97-15b895833078/volumes"
Sep 14 03:05:13.164523 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:13.164492    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvjcz\" (UniqueName: \"kubernetes.io/projected/98e9fdbb-defe-4042-ae97-15b895833078-kube-api-access-pvjcz\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:05:13.185931 ip-10-0-141-4 kubenswrapper[2564]: W0914 03:05:13.185881    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac41e03a_54d4_4b59_8e31_563773bd52ba.slice/crio-d5e67aab31cae3863a41a0a7ac9a24bef4717c7e6dd6c1eef7729fc693628d31 WatchSource:0}: Error finding container d5e67aab31cae3863a41a0a7ac9a24bef4717c7e6dd6c1eef7729fc693628d31: Status 404 returned error can't find the container with id d5e67aab31cae3863a41a0a7ac9a24bef4717c7e6dd6c1eef7729fc693628d31
Sep 14 03:05:13.186091 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:13.186068    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-3a752f-549955bb4-tbwgt"]
Sep 14 03:05:13.197841 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:13.197814    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-3a752f-69bdf4d6b-gl7qz"]
Sep 14 03:05:13.201801 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:13.201753    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3a752f-69bdf4d6b-gl7qz"
Sep 14 03:05:13.205190 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:13.205173    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-3a752f\""
Sep 14 03:05:13.205783 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:13.205761    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-3a752f-dockercfg-6fqct\""
Sep 14 03:05:13.265899 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:13.265865    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpds2\" (UniqueName: \"kubernetes.io/projected/7a69a896-145c-4365-9444-3ea9f537ae6b-kube-api-access-xpds2\") pod \"payload-pre-processing-e2e-worker-w0-3a752f-69bdf4d6b-gl7qz\" (UID: \"7a69a896-145c-4365-9444-3ea9f537ae6b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3a752f-69bdf4d6b-gl7qz"
Sep 14 03:05:13.266082 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:13.265932    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a69a896-145c-4365-9444-3ea9f537ae6b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-3a752f-69bdf4d6b-gl7qz\" (UID: \"7a69a896-145c-4365-9444-3ea9f537ae6b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3a752f-69bdf4d6b-gl7qz"
Sep 14 03:05:13.290215 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:13.290178    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-3a752f-69bdf4d6b-gl7qz"]
Sep 14 03:05:13.310424 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:13.310379    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-3a752f-776b9b9d9f-42gsk"]
Sep 14 03:05:13.314287 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:13.314260    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-3a752f-776b9b9d9f-42gsk"
Sep 14 03:05:13.330387 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:13.330341    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-3a752f-776b9b9d9f-42gsk"]
Sep 14 03:05:13.367210 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:13.367174    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpds2\" (UniqueName: \"kubernetes.io/projected/7a69a896-145c-4365-9444-3ea9f537ae6b-kube-api-access-xpds2\") pod \"payload-pre-processing-e2e-worker-w0-3a752f-69bdf4d6b-gl7qz\" (UID: \"7a69a896-145c-4365-9444-3ea9f537ae6b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3a752f-69bdf4d6b-gl7qz"
Sep 14 03:05:13.367364 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:13.367245    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a69a896-145c-4365-9444-3ea9f537ae6b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-3a752f-69bdf4d6b-gl7qz\" (UID: \"7a69a896-145c-4365-9444-3ea9f537ae6b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3a752f-69bdf4d6b-gl7qz"
Sep 14 03:05:13.367364 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:13.367282    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srlzf\" (UniqueName: \"kubernetes.io/projected/242753da-219f-47d3-ad2e-f3866269b610-kube-api-access-srlzf\") pod \"payload-processing-e2e-worker-w0-3a752f-776b9b9d9f-42gsk\" (UID: \"242753da-219f-47d3-ad2e-f3866269b610\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-3a752f-776b9b9d9f-42gsk"
Sep 14 03:05:13.367364 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:13.367323    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/242753da-219f-47d3-ad2e-f3866269b610-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-3a752f-776b9b9d9f-42gsk\" (UID: \"242753da-219f-47d3-ad2e-f3866269b610\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-3a752f-776b9b9d9f-42gsk"
Sep 14 03:05:13.368007 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:13.367988    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a69a896-145c-4365-9444-3ea9f537ae6b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-3a752f-69bdf4d6b-gl7qz\" (UID: \"7a69a896-145c-4365-9444-3ea9f537ae6b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3a752f-69bdf4d6b-gl7qz"
Sep 14 03:05:13.378569 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:13.378535    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpds2\" (UniqueName: \"kubernetes.io/projected/7a69a896-145c-4365-9444-3ea9f537ae6b-kube-api-access-xpds2\") pod \"payload-pre-processing-e2e-worker-w0-3a752f-69bdf4d6b-gl7qz\" (UID: \"7a69a896-145c-4365-9444-3ea9f537ae6b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3a752f-69bdf4d6b-gl7qz"
Sep 14 03:05:13.468827 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:13.468737    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srlzf\" (UniqueName: \"kubernetes.io/projected/242753da-219f-47d3-ad2e-f3866269b610-kube-api-access-srlzf\") pod \"payload-processing-e2e-worker-w0-3a752f-776b9b9d9f-42gsk\" (UID: \"242753da-219f-47d3-ad2e-f3866269b610\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-3a752f-776b9b9d9f-42gsk"
Sep 14 03:05:13.468827 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:13.468786    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/242753da-219f-47d3-ad2e-f3866269b610-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-3a752f-776b9b9d9f-42gsk\" (UID: \"242753da-219f-47d3-ad2e-f3866269b610\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-3a752f-776b9b9d9f-42gsk"
Sep 14 03:05:13.469542 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:13.469523    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/242753da-219f-47d3-ad2e-f3866269b610-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-3a752f-776b9b9d9f-42gsk\" (UID: \"242753da-219f-47d3-ad2e-f3866269b610\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-3a752f-776b9b9d9f-42gsk"
Sep 14 03:05:13.481636 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:13.481602    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srlzf\" (UniqueName: \"kubernetes.io/projected/242753da-219f-47d3-ad2e-f3866269b610-kube-api-access-srlzf\") pod \"payload-processing-e2e-worker-w0-3a752f-776b9b9d9f-42gsk\" (UID: \"242753da-219f-47d3-ad2e-f3866269b610\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-3a752f-776b9b9d9f-42gsk"
Sep 14 03:05:13.513575 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:13.513531    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3a752f-69bdf4d6b-gl7qz"
Sep 14 03:05:13.627878 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:13.627845    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-3a752f-776b9b9d9f-42gsk"
Sep 14 03:05:13.767161 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:13.767071    2564 generic.go:356] "Generic (PLEG): container finished" podID="98e9fdbb-defe-4042-ae97-15b895833078" containerID="1c028340904753ae0aa94ad249424fa08fd9d073ad7abf68786c280b47c0bbfe" exitCode=137
Sep 14 03:05:13.767161 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:13.767142    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7683-af10df"
Sep 14 03:05:13.767784 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:13.767134    2564 scope.go:117] "RemoveContainer" containerID="1c028340904753ae0aa94ad249424fa08fd9d073ad7abf68786c280b47c0bbfe"
Sep 14 03:05:13.769685 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:13.769606    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3a752f-549955bb4-tbwgt" event={"ID":"ac41e03a-54d4-4b59-8e31-563773bd52ba","Type":"ContainerStarted","Data":"03af6e426657d52129ffd13ad8924df01509bd4c9ce41c8a1033ada23e2b0f24"}
Sep 14 03:05:13.769807 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:13.769707    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3a752f-549955bb4-tbwgt"
Sep 14 03:05:13.769807 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:13.769734    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3a752f-549955bb4-tbwgt" event={"ID":"ac41e03a-54d4-4b59-8e31-563773bd52ba","Type":"ContainerStarted","Data":"d5e67aab31cae3863a41a0a7ac9a24bef4717c7e6dd6c1eef7729fc693628d31"}
Sep 14 03:05:13.778192 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:13.778168    2564 scope.go:117] "RemoveContainer" containerID="1c028340904753ae0aa94ad249424fa08fd9d073ad7abf68786c280b47c0bbfe"
Sep 14 03:05:13.778512 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:05:13.778492    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c028340904753ae0aa94ad249424fa08fd9d073ad7abf68786c280b47c0bbfe\": container with ID starting with 1c028340904753ae0aa94ad249424fa08fd9d073ad7abf68786c280b47c0bbfe not found: ID does not exist" containerID="1c028340904753ae0aa94ad249424fa08fd9d073ad7abf68786c280b47c0bbfe"
Sep 14 03:05:13.778563 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:13.778522    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c028340904753ae0aa94ad249424fa08fd9d073ad7abf68786c280b47c0bbfe"} err="failed to get container status \"1c028340904753ae0aa94ad249424fa08fd9d073ad7abf68786c280b47c0bbfe\": rpc error: code = NotFound desc = could not find container \"1c028340904753ae0aa94ad249424fa08fd9d073ad7abf68786c280b47c0bbfe\": container with ID starting with 1c028340904753ae0aa94ad249424fa08fd9d073ad7abf68786c280b47c0bbfe not found: ID does not exist"
Sep 14 03:05:13.793722 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:13.793673    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3a752f-549955bb4-tbwgt" podStartSLOduration=2.426527267 podStartE2EDuration="2.793659502s" totalImagesPullingTime="367.132235ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:05:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:05:13.79198659 +0000 UTC m=+759.173070790" watchObservedRunningTime="2026-09-14 03:05:13.793659502 +0000 UTC m=+759.174743729"
Sep 14 03:05:13.813492 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:13.813465    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-3a752f-69bdf4d6b-gl7qz"]
Sep 14 03:05:13.816540 ip-10-0-141-4 kubenswrapper[2564]: W0914 03:05:13.816515    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a69a896_145c_4365_9444_3ea9f537ae6b.slice/crio-f8510908bf627df00199c1ec51187a4c144db322c755149edb65cc63bd79bea4 WatchSource:0}: Error finding container f8510908bf627df00199c1ec51187a4c144db322c755149edb65cc63bd79bea4: Status 404 returned error can't find the container with id f8510908bf627df00199c1ec51187a4c144db322c755149edb65cc63bd79bea4
Sep 14 03:05:13.908091 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:13.908025    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:13.957044 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:13.957015    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-3a752f-776b9b9d9f-42gsk"]
Sep 14 03:05:13.958162 ip-10-0-141-4 kubenswrapper[2564]: W0914 03:05:13.958121    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod242753da_219f_47d3_ad2e_f3866269b610.slice/crio-5ad1ef1afabdb4364281fad4d141be3e63d95b0193bc1dce862965c1c4313c4e WatchSource:0}: Error finding container 5ad1ef1afabdb4364281fad4d141be3e63d95b0193bc1dce862965c1c4313c4e: Status 404 returned error can't find the container with id 5ad1ef1afabdb4364281fad4d141be3e63d95b0193bc1dce862965c1c4313c4e
Sep 14 03:05:14.079027 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:14.078981    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:14.750137 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:14.750105    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9c9f1e-8497955c59-k5c6b"
Sep 14 03:05:14.778875 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:14.778604    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-3a752f-776b9b9d9f-42gsk" event={"ID":"242753da-219f-47d3-ad2e-f3866269b610","Type":"ContainerStarted","Data":"67a125b822416c23bf4c673a47f1b502d9ccf0ae17194e9dcc1c32016d2d5919"}
Sep 14 03:05:14.778875 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:14.778669    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-3a752f-776b9b9d9f-42gsk"
Sep 14 03:05:14.778875 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:14.778696    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-3a752f-776b9b9d9f-42gsk" event={"ID":"242753da-219f-47d3-ad2e-f3866269b610","Type":"ContainerStarted","Data":"5ad1ef1afabdb4364281fad4d141be3e63d95b0193bc1dce862965c1c4313c4e"}
Sep 14 03:05:14.780583 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:14.780555    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3a752f-69bdf4d6b-gl7qz" event={"ID":"7a69a896-145c-4365-9444-3ea9f537ae6b","Type":"ContainerStarted","Data":"59619c12a0230288a092966749ba6752ae76ff009d27e195d20f856702d8657a"}
Sep 14 03:05:14.780702 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:14.780606    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3a752f-69bdf4d6b-gl7qz" event={"ID":"7a69a896-145c-4365-9444-3ea9f537ae6b","Type":"ContainerStarted","Data":"f8510908bf627df00199c1ec51187a4c144db322c755149edb65cc63bd79bea4"}
Sep 14 03:05:14.780702 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:14.780653    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3a752f-69bdf4d6b-gl7qz"
Sep 14 03:05:14.825772 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:14.825713    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-3a752f-776b9b9d9f-42gsk" podStartSLOduration=1.825691697 podStartE2EDuration="1.825691697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:05:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:05:14.817231123 +0000 UTC m=+760.198315334" watchObservedRunningTime="2026-09-14 03:05:14.825691697 +0000 UTC m=+760.206775897"
Sep 14 03:05:14.840140 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:14.840086    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3a752f-69bdf4d6b-gl7qz" podStartSLOduration=1.840068499 podStartE2EDuration="1.840068499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:05:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:05:14.83972286 +0000 UTC m=+760.220807062" watchObservedRunningTime="2026-09-14 03:05:14.840068499 +0000 UTC m=+760.221152697"
Sep 14 03:05:15.206218 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:15.206183    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:15.301548 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:15.301510    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:15.683210 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:15.683169    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:15.754037 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:15.754010    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f002a1-7f464f54d5-xntnr"
Sep 14 03:05:15.785002 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:15.784957    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:15.925550 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:15.925519    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:16.509128 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:16.509092    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:16.600083 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:16.600048    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:16.701691 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:16.701662    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-ab5750-5b6f7d747d-464xq"]
Sep 14 03:05:16.705205 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:16.705188    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-ab5750-5b6f7d747d-464xq"
Sep 14 03:05:16.707976 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:16.707956    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-ab5750\""
Sep 14 03:05:16.708061 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:16.708012    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-ab5750-dockercfg-25x8f\""
Sep 14 03:05:16.717888 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:16.717868    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-ab5750-5b6f7d747d-464xq"]
Sep 14 03:05:16.806092 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:16.806063    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4f5c\" (UniqueName: \"kubernetes.io/projected/2e738de9-f4f0-4ed8-bfb4-c80715863411-kube-api-access-v4f5c\") pod \"payload-pre-processing-e2e-autores-ab5750-5b6f7d747d-464xq\" (UID: \"2e738de9-f4f0-4ed8-bfb4-c80715863411\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-ab5750-5b6f7d747d-464xq"
Sep 14 03:05:16.806478 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:16.806134    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e738de9-f4f0-4ed8-bfb4-c80715863411-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-ab5750-5b6f7d747d-464xq\" (UID: \"2e738de9-f4f0-4ed8-bfb4-c80715863411\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-ab5750-5b6f7d747d-464xq"
Sep 14 03:05:16.907578 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:16.907545    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e738de9-f4f0-4ed8-bfb4-c80715863411-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-ab5750-5b6f7d747d-464xq\" (UID: \"2e738de9-f4f0-4ed8-bfb4-c80715863411\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-ab5750-5b6f7d747d-464xq"
Sep 14 03:05:16.907766 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:16.907637    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4f5c\" (UniqueName: \"kubernetes.io/projected/2e738de9-f4f0-4ed8-bfb4-c80715863411-kube-api-access-v4f5c\") pod \"payload-pre-processing-e2e-autores-ab5750-5b6f7d747d-464xq\" (UID: \"2e738de9-f4f0-4ed8-bfb4-c80715863411\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-ab5750-5b6f7d747d-464xq"
Sep 14 03:05:16.908550 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:16.908529    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e738de9-f4f0-4ed8-bfb4-c80715863411-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-ab5750-5b6f7d747d-464xq\" (UID: \"2e738de9-f4f0-4ed8-bfb4-c80715863411\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-ab5750-5b6f7d747d-464xq"
Sep 14 03:05:16.920456 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:16.920435    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4f5c\" (UniqueName: \"kubernetes.io/projected/2e738de9-f4f0-4ed8-bfb4-c80715863411-kube-api-access-v4f5c\") pod \"payload-pre-processing-e2e-autores-ab5750-5b6f7d747d-464xq\" (UID: \"2e738de9-f4f0-4ed8-bfb4-c80715863411\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-ab5750-5b6f7d747d-464xq"
Sep 14 03:05:16.971537 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:16.971506    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:17.016636 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:17.016603    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-ab5750-5b6f7d747d-464xq"
Sep 14 03:05:17.319220 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:17.319197    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-ab5750-5b6f7d747d-464xq"]
Sep 14 03:05:17.321276 ip-10-0-141-4 kubenswrapper[2564]: W0914 03:05:17.321249    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e738de9_f4f0_4ed8_bfb4_c80715863411.slice/crio-c40a8c736b6d865378254e9c9b119e05c6f7deffa4803de43063e0b320c9886a WatchSource:0}: Error finding container c40a8c736b6d865378254e9c9b119e05c6f7deffa4803de43063e0b320c9886a: Status 404 returned error can't find the container with id c40a8c736b6d865378254e9c9b119e05c6f7deffa4803de43063e0b320c9886a
Sep 14 03:05:17.798042 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:17.798011    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-ab5750-5b6f7d747d-464xq" event={"ID":"2e738de9-f4f0-4ed8-bfb4-c80715863411","Type":"ContainerStarted","Data":"93bc915c7c8e3c3aa37e34582d9b07f47fc02200ad89e671b74a841012314bc5"}
Sep 14 03:05:17.798217 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:17.798059    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-ab5750-5b6f7d747d-464xq" event={"ID":"2e738de9-f4f0-4ed8-bfb4-c80715863411","Type":"ContainerStarted","Data":"c40a8c736b6d865378254e9c9b119e05c6f7deffa4803de43063e0b320c9886a"}
Sep 14 03:05:17.798217 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:17.798155    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-ab5750-5b6f7d747d-464xq"
Sep 14 03:05:17.820703 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:17.820656    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-ab5750-5b6f7d747d-464xq" podStartSLOduration=1.820640839 podStartE2EDuration="1.820640839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:05:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:05:17.817895155 +0000 UTC m=+763.198979358" watchObservedRunningTime="2026-09-14 03:05:17.820640839 +0000 UTC m=+763.201725038"
Sep 14 03:05:18.145999 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:18.145912    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:18.918708 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:18.918673    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:19.435770 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:19.435733    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:19.786286 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:19.786255    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3a752f-549955bb4-tbwgt"
Sep 14 03:05:19.938364 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:19.938327    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:20.054526 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:20.054418    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:20.196057 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:20.196014    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:20.282199 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:20.282160    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:20.703688 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:20.703636    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:20.750453 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:20.750423    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-9c9f1e-785c744797-7hjth"
Sep 14 03:05:20.834030 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:20.833990    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:21.472641 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:21.472606    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:21.648314 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:21.648277    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:21.849184 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:21.849145    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:21.945225 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:21.945185    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:22.041030 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:22.040995    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:22.490469 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:22.490426    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:23.493252 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:23.493218    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:24.347371 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:24.347335    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:24.614164 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:24.614075    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:24.731020 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:24.730982    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:25.037942 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:25.037902    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:25.357628 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:25.357536    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:25.461486 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:25.461448    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:25.575803 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:25.575768    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:25.788793 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:25.788762    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-3a752f-776b9b9d9f-42gsk"
Sep 14 03:05:25.789266 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:25.788877    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3a752f-69bdf4d6b-gl7qz"
Sep 14 03:05:26.339461 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:26.339429    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:26.440481 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:26.440443    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:26.562021 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:26.561986    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:26.660666 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:26.660573    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:26.777047 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:26.777012    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:27.089972 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:27.089938    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:27.411832 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:27.411737    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:27.550604 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:27.550569    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:27.651453 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:27.651419    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:27.753698 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:27.753611    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:28.475751 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:28.475716    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:28.630281 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:28.630249    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:28.805224 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:28.805199    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-ab5750-5b6f7d747d-464xq"
Sep 14 03:05:29.657884 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:29.657848    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:30.546925 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:30.546883    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:31.319150 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:31.319109    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:32.248913 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:32.248881    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:33.415421 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:33.415361    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4eb26b-6bcfc6b59-fwtl2"]
Sep 14 03:05:33.419487 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:33.419467    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4eb26b-6bcfc6b59-fwtl2"
Sep 14 03:05:33.422953 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:33.422935    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-4eb26b-serving-cert\""
Sep 14 03:05:33.443643 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:33.443607    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4eb26b-6bcfc6b59-fwtl2"]
Sep 14 03:05:33.473759 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:33.473725    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:33.579419 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:33.579364    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krp7b\" (UniqueName: \"kubernetes.io/projected/2169752d-69ca-4313-aa66-b4c853c1b255-kube-api-access-krp7b\") pod \"maas-api-e2e-inf-b-4eb26b-6bcfc6b59-fwtl2\" (UID: \"2169752d-69ca-4313-aa66-b4c853c1b255\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4eb26b-6bcfc6b59-fwtl2"
Sep 14 03:05:33.579609 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:33.579455    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2169752d-69ca-4313-aa66-b4c853c1b255-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4eb26b-6bcfc6b59-fwtl2\" (UID: \"2169752d-69ca-4313-aa66-b4c853c1b255\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4eb26b-6bcfc6b59-fwtl2"
Sep 14 03:05:33.579609 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:33.579483    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2169752d-69ca-4313-aa66-b4c853c1b255-metrics-certs\") pod \"maas-api-e2e-inf-b-4eb26b-6bcfc6b59-fwtl2\" (UID: \"2169752d-69ca-4313-aa66-b4c853c1b255\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4eb26b-6bcfc6b59-fwtl2"
Sep 14 03:05:33.579609 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:33.579568    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2169752d-69ca-4313-aa66-b4c853c1b255-maas-api-tls\") pod \"maas-api-e2e-inf-b-4eb26b-6bcfc6b59-fwtl2\" (UID: \"2169752d-69ca-4313-aa66-b4c853c1b255\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4eb26b-6bcfc6b59-fwtl2"
Sep 14 03:05:33.645245 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:33.645208    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:33.680385 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:33.680287    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2169752d-69ca-4313-aa66-b4c853c1b255-maas-api-tls\") pod \"maas-api-e2e-inf-b-4eb26b-6bcfc6b59-fwtl2\" (UID: \"2169752d-69ca-4313-aa66-b4c853c1b255\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4eb26b-6bcfc6b59-fwtl2"
Sep 14 03:05:33.680385 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:33.680344    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krp7b\" (UniqueName: \"kubernetes.io/projected/2169752d-69ca-4313-aa66-b4c853c1b255-kube-api-access-krp7b\") pod \"maas-api-e2e-inf-b-4eb26b-6bcfc6b59-fwtl2\" (UID: \"2169752d-69ca-4313-aa66-b4c853c1b255\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4eb26b-6bcfc6b59-fwtl2"
Sep 14 03:05:33.680638 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:33.680399    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2169752d-69ca-4313-aa66-b4c853c1b255-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4eb26b-6bcfc6b59-fwtl2\" (UID: \"2169752d-69ca-4313-aa66-b4c853c1b255\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4eb26b-6bcfc6b59-fwtl2"
Sep 14 03:05:33.680638 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:33.680451    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2169752d-69ca-4313-aa66-b4c853c1b255-metrics-certs\") pod \"maas-api-e2e-inf-b-4eb26b-6bcfc6b59-fwtl2\" (UID: \"2169752d-69ca-4313-aa66-b4c853c1b255\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4eb26b-6bcfc6b59-fwtl2"
Sep 14 03:05:33.681184 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:33.681157    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2169752d-69ca-4313-aa66-b4c853c1b255-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4eb26b-6bcfc6b59-fwtl2\" (UID: \"2169752d-69ca-4313-aa66-b4c853c1b255\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4eb26b-6bcfc6b59-fwtl2"
Sep 14 03:05:33.682949 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:33.682922    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2169752d-69ca-4313-aa66-b4c853c1b255-maas-api-tls\") pod \"maas-api-e2e-inf-b-4eb26b-6bcfc6b59-fwtl2\" (UID: \"2169752d-69ca-4313-aa66-b4c853c1b255\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4eb26b-6bcfc6b59-fwtl2"
Sep 14 03:05:33.683427 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:33.683380    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2169752d-69ca-4313-aa66-b4c853c1b255-metrics-certs\") pod \"maas-api-e2e-inf-b-4eb26b-6bcfc6b59-fwtl2\" (UID: \"2169752d-69ca-4313-aa66-b4c853c1b255\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4eb26b-6bcfc6b59-fwtl2"
Sep 14 03:05:33.701038 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:33.701013    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krp7b\" (UniqueName: \"kubernetes.io/projected/2169752d-69ca-4313-aa66-b4c853c1b255-kube-api-access-krp7b\") pod \"maas-api-e2e-inf-b-4eb26b-6bcfc6b59-fwtl2\" (UID: \"2169752d-69ca-4313-aa66-b4c853c1b255\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4eb26b-6bcfc6b59-fwtl2"
Sep 14 03:05:33.730224 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:33.730186    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4eb26b-6bcfc6b59-fwtl2"
Sep 14 03:05:33.925059 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:33.925015    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:34.045933 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:34.045894    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4eb26b-6bcfc6b59-fwtl2"]
Sep 14 03:05:34.048571 ip-10-0-141-4 kubenswrapper[2564]: W0914 03:05:34.048536    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2169752d_69ca_4313_aa66_b4c853c1b255.slice/crio-a0291458d1bbc9e9dec7a96015b4170639d49caf3cd8eb7dda88116258488041 WatchSource:0}: Error finding container a0291458d1bbc9e9dec7a96015b4170639d49caf3cd8eb7dda88116258488041: Status 404 returned error can't find the container with id a0291458d1bbc9e9dec7a96015b4170639d49caf3cd8eb7dda88116258488041
Sep 14 03:05:34.776767 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:34.776733    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:34.906872 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:34.906833    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4eb26b-6bcfc6b59-fwtl2" event={"ID":"2169752d-69ca-4313-aa66-b4c853c1b255","Type":"ContainerStarted","Data":"f954504db1f5aeacc09d0a0c95d2b25db58a7de9dc7187134f9fef95307d535f"}
Sep 14 03:05:34.907065 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:34.906908    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4eb26b-6bcfc6b59-fwtl2" event={"ID":"2169752d-69ca-4313-aa66-b4c853c1b255","Type":"ContainerStarted","Data":"a0291458d1bbc9e9dec7a96015b4170639d49caf3cd8eb7dda88116258488041"}
Sep 14 03:05:34.907065 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:34.906974    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4eb26b-6bcfc6b59-fwtl2"
Sep 14 03:05:34.907584 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:34.907556    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:34.946371 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:34.946309    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4eb26b-6bcfc6b59-fwtl2" podStartSLOduration=1.585977639 podStartE2EDuration="1.946284354s" totalImagesPullingTime="360.306715ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:05:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:05:34.944689181 +0000 UTC m=+780.325773379" watchObservedRunningTime="2026-09-14 03:05:34.946284354 +0000 UTC m=+780.327368553"
Sep 14 03:05:35.477309 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:35.477209    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:36.437573 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:36.437526    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:36.638126 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:36.638090    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:36.856670 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:36.856627    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:36.982851 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:36.982820    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:37.578274 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:37.578223    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:38.540905 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:38.540871    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:39.647031 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:39.646993    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:40.539929 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:40.539887    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:40.917139 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:40.917052    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4eb26b-6bcfc6b59-fwtl2"
Sep 14 03:05:41.268868 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:41.268772    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:42.274370 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:42.274332    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:43.015545 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:43.015515    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:43.221584 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:43.221546    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:43.916074 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:43.916032    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:44.047045 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:44.047015    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:44.149046 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:44.149016    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:44.803345 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:44.803302    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:45.548791 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:45.548745    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:46.514399 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:46.514365    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:47.276960 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:47.276918    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:48.077306 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:48.077267    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:48.694271 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:48.694238    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:48.767129 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:48.767086    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:48.831971 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:48.831928    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:48.905264 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:48.905222    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:49.064383 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:49.064344    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:49.347790 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:49.347689    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:49.522211 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:49.522178    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:49.619920 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:49.619835    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:49.720017 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:49.719987    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:50.229297 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:50.229266    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:50.511776 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:50.511683    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:50.757054 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:50.757025    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:51.341061 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:51.341019    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:52.140594 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:52.140557    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:52.590833 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:52.590795    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:52.705064 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:52.705023    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:53.091578 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:53.091541    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:53.926471 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:53.926388    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:54.680066 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:54.680010    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:54.752099 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:54.752069    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:54.960541 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:54.960452    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:55.224612 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:55.224530    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:55.326149 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:55.326113    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:55.703513 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:55.703477    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:56.642758 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:56.642720    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:57.050069 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:57.050016    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:57.332846 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:57.332752    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:58.062730 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:58.062686    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:05:58.851665 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:05:58.851630    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:06:16.829137 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:06:16.829076    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:06:17.595426 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:06:17.595373    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:06:17.814837 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:06:17.814796    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:06:18.588084 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:06:18.588038    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:06:19.551767 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:06:19.551736    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:06:20.417224 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:06:20.417189    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:06:21.362945 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:06:21.362903    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:06:22.141252 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:06:22.141214    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:06:23.143931 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:06:23.143885    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:06:24.048545 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:06:24.048513    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:06:33.716637 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:06:33.716597    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:06:34.456764 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:06:34.456726    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:06:34.555517 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:06:34.555477    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:06:34.654145 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:06:34.654108    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:06:34.754918 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:06:34.754815    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:06:34.874788 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:06:34.874742    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:06:35.136827 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:06:35.136792    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:06:35.326254 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:06:35.326219    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:06:35.638037 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:06:35.637996    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:06:35.801321 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:06:35.801282    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:06:36.096820 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:06:36.096774    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:06:36.957834 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:06:36.957799    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:06:37.910477 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:06:37.910438    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:06:38.636086 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:06:38.636051    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:06:38.739728 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:06:38.739687    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:06:39.543266 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:06:39.543227    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:06:40.351661 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:06:40.351619    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:06:40.429920 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:06:40.429885    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:06:40.560808 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:06:40.560766    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:06:40.661250 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:06:40.661159    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:06:40.998659 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:06:40.998576    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:06:41.089991 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:06:41.089937    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:06:41.099253 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:06:41.099225    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-65e77699-7f54ddccc8-r78hn"]
Sep 14 03:06:41.103235 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:06:41.103215    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-65e77699-7f54ddccc8-r78hn"
Sep 14 03:06:41.105848 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:06:41.105823    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-65e77699\""
Sep 14 03:06:41.111569 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:06:41.111275    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-65e77699-7f54ddccc8-r78hn"]
Sep 14 03:06:41.273456 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:06:41.273397    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/228451a7-e549-4cdb-8d76-30b44a13f81b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-65e77699-7f54ddccc8-r78hn\" (UID: \"228451a7-e549-4cdb-8d76-30b44a13f81b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-65e77699-7f54ddccc8-r78hn"
Sep 14 03:06:41.273652 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:06:41.273459    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlxz4\" (UniqueName: \"kubernetes.io/projected/228451a7-e549-4cdb-8d76-30b44a13f81b-kube-api-access-hlxz4\") pod \"payload-pre-processing-e2e-ait-65e77699-7f54ddccc8-r78hn\" (UID: \"228451a7-e549-4cdb-8d76-30b44a13f81b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-65e77699-7f54ddccc8-r78hn"
Sep 14 03:06:41.374466 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:06:41.374424    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/228451a7-e549-4cdb-8d76-30b44a13f81b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-65e77699-7f54ddccc8-r78hn\" (UID: \"228451a7-e549-4cdb-8d76-30b44a13f81b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-65e77699-7f54ddccc8-r78hn"
Sep 14 03:06:41.374466 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:06:41.374462    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlxz4\" (UniqueName: \"kubernetes.io/projected/228451a7-e549-4cdb-8d76-30b44a13f81b-kube-api-access-hlxz4\") pod \"payload-pre-processing-e2e-ait-65e77699-7f54ddccc8-r78hn\" (UID: \"228451a7-e549-4cdb-8d76-30b44a13f81b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-65e77699-7f54ddccc8-r78hn"
Sep 14 03:06:41.375173 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:06:41.375151    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/228451a7-e549-4cdb-8d76-30b44a13f81b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-65e77699-7f54ddccc8-r78hn\" (UID: \"228451a7-e549-4cdb-8d76-30b44a13f81b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-65e77699-7f54ddccc8-r78hn"
Sep 14 03:06:41.384004 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:06:41.383969    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlxz4\" (UniqueName: \"kubernetes.io/projected/228451a7-e549-4cdb-8d76-30b44a13f81b-kube-api-access-hlxz4\") pod \"payload-pre-processing-e2e-ait-65e77699-7f54ddccc8-r78hn\" (UID: \"228451a7-e549-4cdb-8d76-30b44a13f81b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-65e77699-7f54ddccc8-r78hn"
Sep 14 03:06:41.415131 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:06:41.415098    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-65e77699-7f54ddccc8-r78hn"
Sep 14 03:06:41.430101 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:06:41.430073    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:06:41.692341 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:06:41.692314    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-65e77699-7f54ddccc8-r78hn"]
Sep 14 03:06:41.694898 ip-10-0-141-4 kubenswrapper[2564]: W0914 03:06:41.694861    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod228451a7_e549_4cdb_8d76_30b44a13f81b.slice/crio-595cfa39f65a40e61f873c0086ce325117b91172f881c9b819255f0187bad1c0 WatchSource:0}: Error finding container 595cfa39f65a40e61f873c0086ce325117b91172f881c9b819255f0187bad1c0: Status 404 returned error can't find the container with id 595cfa39f65a40e61f873c0086ce325117b91172f881c9b819255f0187bad1c0
Sep 14 03:06:42.236264 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:06:42.236228    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:06:42.293382 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:06:42.293351    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-65e77699-7f54ddccc8-r78hn" event={"ID":"228451a7-e549-4cdb-8d76-30b44a13f81b","Type":"ContainerStarted","Data":"2f4ee112c86271c6c224f1c7710283e9f9a4ee68264b10a9a8dc127a6b62f320"}
Sep 14 03:06:42.293576 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:06:42.293452    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-65e77699-7f54ddccc8-r78hn" event={"ID":"228451a7-e549-4cdb-8d76-30b44a13f81b","Type":"ContainerStarted","Data":"595cfa39f65a40e61f873c0086ce325117b91172f881c9b819255f0187bad1c0"}
Sep 14 03:06:42.293576 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:06:42.293508    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-65e77699-7f54ddccc8-r78hn"
Sep 14 03:06:42.314620 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:06:42.314564    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-65e77699-7f54ddccc8-r78hn" podStartSLOduration=1.314543484 podStartE2EDuration="1.314543484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:06:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:06:42.31292329 +0000 UTC m=+847.694007490" watchObservedRunningTime="2026-09-14 03:06:42.314543484 +0000 UTC m=+847.695627685"
Sep 14 03:06:43.196605 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:06:43.196568    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:06:43.921835 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:06:43.921800    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:06:44.727493 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:06:44.727460    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:06:53.300863 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:06:53.300823    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-65e77699-7f54ddccc8-r78hn"
Sep 14 03:07:02.656093 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:02.656055    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:03.765270 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:03.765234    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:04.701934 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:04.701897    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:05.432460 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:05.432427    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:06.192671 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:06.192608    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:06.861159 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:06.861117    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:07.505543 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:07.505503    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:07.759002 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:07.758908    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:08.532915 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:08.532864    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:09.256624 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:09.256588    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:10.054095 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:10.054060    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:10.835789 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:10.835733    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:10.862082 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:10.862045    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-65e77699-7f54ddccc8-r78hn"]
Sep 14 03:07:10.862342 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:10.862288    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-65e77699-7f54ddccc8-r78hn" podUID="228451a7-e549-4cdb-8d76-30b44a13f81b" containerName="payload-pre-processing" containerID="cri-o://2f4ee112c86271c6c224f1c7710283e9f9a4ee68264b10a9a8dc127a6b62f320" gracePeriod=30
Sep 14 03:07:10.910969 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:10.910932    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:11.010930 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:11.010896    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:11.078355 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:11.078316    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:11.120160 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:11.120131    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-65e77699-7f54ddccc8-r78hn"
Sep 14 03:07:11.165497 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:11.165457    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hlxz4\" (UniqueName: \"kubernetes.io/projected/228451a7-e549-4cdb-8d76-30b44a13f81b-kube-api-access-hlxz4\") pod \"228451a7-e549-4cdb-8d76-30b44a13f81b\" (UID: \"228451a7-e549-4cdb-8d76-30b44a13f81b\") "
Sep 14 03:07:11.165712 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:11.165679    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/228451a7-e549-4cdb-8d76-30b44a13f81b-plugins-config-volume\") pod \"228451a7-e549-4cdb-8d76-30b44a13f81b\" (UID: \"228451a7-e549-4cdb-8d76-30b44a13f81b\") "
Sep 14 03:07:11.166153 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:11.166121    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/228451a7-e549-4cdb-8d76-30b44a13f81b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "228451a7-e549-4cdb-8d76-30b44a13f81b" (UID: "228451a7-e549-4cdb-8d76-30b44a13f81b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 03:07:11.168167 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:11.168140    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/228451a7-e549-4cdb-8d76-30b44a13f81b-kube-api-access-hlxz4" (OuterVolumeSpecName: "kube-api-access-hlxz4") pod "228451a7-e549-4cdb-8d76-30b44a13f81b" (UID: "228451a7-e549-4cdb-8d76-30b44a13f81b"). InnerVolumeSpecName "kube-api-access-hlxz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:07:11.267627 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:11.267590    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/228451a7-e549-4cdb-8d76-30b44a13f81b-plugins-config-volume\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:07:11.267627 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:11.267621    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlxz4\" (UniqueName: \"kubernetes.io/projected/228451a7-e549-4cdb-8d76-30b44a13f81b-kube-api-access-hlxz4\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:07:11.427534 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:11.427431    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:11.461334 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:11.461298    2564 generic.go:356] "Generic (PLEG): container finished" podID="228451a7-e549-4cdb-8d76-30b44a13f81b" containerID="2f4ee112c86271c6c224f1c7710283e9f9a4ee68264b10a9a8dc127a6b62f320" exitCode=0
Sep 14 03:07:11.461539 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:11.461347    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-65e77699-7f54ddccc8-r78hn"
Sep 14 03:07:11.461539 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:11.461388    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-65e77699-7f54ddccc8-r78hn" event={"ID":"228451a7-e549-4cdb-8d76-30b44a13f81b","Type":"ContainerDied","Data":"2f4ee112c86271c6c224f1c7710283e9f9a4ee68264b10a9a8dc127a6b62f320"}
Sep 14 03:07:11.461539 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:11.461458    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-65e77699-7f54ddccc8-r78hn" event={"ID":"228451a7-e549-4cdb-8d76-30b44a13f81b","Type":"ContainerDied","Data":"595cfa39f65a40e61f873c0086ce325117b91172f881c9b819255f0187bad1c0"}
Sep 14 03:07:11.461539 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:11.461488    2564 scope.go:117] "RemoveContainer" containerID="2f4ee112c86271c6c224f1c7710283e9f9a4ee68264b10a9a8dc127a6b62f320"
Sep 14 03:07:11.470899 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:11.470877    2564 scope.go:117] "RemoveContainer" containerID="2f4ee112c86271c6c224f1c7710283e9f9a4ee68264b10a9a8dc127a6b62f320"
Sep 14 03:07:11.471229 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:07:11.471209    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f4ee112c86271c6c224f1c7710283e9f9a4ee68264b10a9a8dc127a6b62f320\": container with ID starting with 2f4ee112c86271c6c224f1c7710283e9f9a4ee68264b10a9a8dc127a6b62f320 not found: ID does not exist" containerID="2f4ee112c86271c6c224f1c7710283e9f9a4ee68264b10a9a8dc127a6b62f320"
Sep 14 03:07:11.471280 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:11.471238    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f4ee112c86271c6c224f1c7710283e9f9a4ee68264b10a9a8dc127a6b62f320"} err="failed to get container status \"2f4ee112c86271c6c224f1c7710283e9f9a4ee68264b10a9a8dc127a6b62f320\": rpc error: code = NotFound desc = could not find container \"2f4ee112c86271c6c224f1c7710283e9f9a4ee68264b10a9a8dc127a6b62f320\": container with ID starting with 2f4ee112c86271c6c224f1c7710283e9f9a4ee68264b10a9a8dc127a6b62f320 not found: ID does not exist"
Sep 14 03:07:11.487047 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:11.487003    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-65e77699-7f54ddccc8-r78hn"]
Sep 14 03:07:11.489013 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:11.488991    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-65e77699-7f54ddccc8-r78hn"]
Sep 14 03:07:12.161530 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:12.161477    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:12.269638 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:12.269579    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:12.624618 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:12.624583    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:12.915877 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:12.915787    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:13.099039 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:13.099006    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="228451a7-e549-4cdb-8d76-30b44a13f81b" path="/var/lib/kubelet/pods/228451a7-e549-4cdb-8d76-30b44a13f81b/volumes"
Sep 14 03:07:13.138519 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:13.138484    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:13.339557 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:13.339512    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:13.976902 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:13.976872    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:14.867070 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:14.867029    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:15.368742 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:15.368707    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:15.462701 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:15.462663    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:15.767788 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:15.767691    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:16.746768 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:16.746733    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:17.592885 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:17.592848    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:18.472308 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:18.472271    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:18.570566 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:18.570528    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:18.868393 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:18.868359    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:18.971713 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:18.971685    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:19.078076 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:19.078037    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:19.080524 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:19.080484    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-fb76f4c2-6cbd998d6-wppld"]
Sep 14 03:07:19.081007 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:19.080993    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="228451a7-e549-4cdb-8d76-30b44a13f81b" containerName="payload-pre-processing"
Sep 14 03:07:19.081053 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:19.081008    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="228451a7-e549-4cdb-8d76-30b44a13f81b" containerName="payload-pre-processing"
Sep 14 03:07:19.081088 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:19.081079    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="228451a7-e549-4cdb-8d76-30b44a13f81b" containerName="payload-pre-processing"
Sep 14 03:07:19.085798 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:19.085781    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-fb76f4c2-6cbd998d6-wppld"
Sep 14 03:07:19.088853 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:19.088833    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-fb76f4c2-dockercfg-hmncl\""
Sep 14 03:07:19.088942 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:19.088840    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-fb76f4c2\""
Sep 14 03:07:19.103194 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:19.103169    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-fb76f4c2-6cbd998d6-wppld"]
Sep 14 03:07:19.154514 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:19.154395    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d6682c1-8385-4737-bff7-39f95c49453b-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-fb76f4c2-6cbd998d6-wppld\" (UID: \"9d6682c1-8385-4737-bff7-39f95c49453b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-fb76f4c2-6cbd998d6-wppld"
Sep 14 03:07:19.154663 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:19.154529    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c5fv\" (UniqueName: \"kubernetes.io/projected/9d6682c1-8385-4737-bff7-39f95c49453b-kube-api-access-7c5fv\") pod \"payload-pre-processing-e2e-derive-fb76f4c2-6cbd998d6-wppld\" (UID: \"9d6682c1-8385-4737-bff7-39f95c49453b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-fb76f4c2-6cbd998d6-wppld"
Sep 14 03:07:19.256139 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:19.256094    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d6682c1-8385-4737-bff7-39f95c49453b-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-fb76f4c2-6cbd998d6-wppld\" (UID: \"9d6682c1-8385-4737-bff7-39f95c49453b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-fb76f4c2-6cbd998d6-wppld"
Sep 14 03:07:19.256331 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:19.256158    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c5fv\" (UniqueName: \"kubernetes.io/projected/9d6682c1-8385-4737-bff7-39f95c49453b-kube-api-access-7c5fv\") pod \"payload-pre-processing-e2e-derive-fb76f4c2-6cbd998d6-wppld\" (UID: \"9d6682c1-8385-4737-bff7-39f95c49453b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-fb76f4c2-6cbd998d6-wppld"
Sep 14 03:07:19.256855 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:19.256832    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d6682c1-8385-4737-bff7-39f95c49453b-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-fb76f4c2-6cbd998d6-wppld\" (UID: \"9d6682c1-8385-4737-bff7-39f95c49453b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-fb76f4c2-6cbd998d6-wppld"
Sep 14 03:07:19.265573 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:19.265548    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c5fv\" (UniqueName: \"kubernetes.io/projected/9d6682c1-8385-4737-bff7-39f95c49453b-kube-api-access-7c5fv\") pod \"payload-pre-processing-e2e-derive-fb76f4c2-6cbd998d6-wppld\" (UID: \"9d6682c1-8385-4737-bff7-39f95c49453b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-fb76f4c2-6cbd998d6-wppld"
Sep 14 03:07:19.397128 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:19.397090    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-fb76f4c2-6cbd998d6-wppld"
Sep 14 03:07:19.670966 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:19.670935    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-fb76f4c2-6cbd998d6-wppld"]
Sep 14 03:07:19.673618 ip-10-0-141-4 kubenswrapper[2564]: W0914 03:07:19.673589    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d6682c1_8385_4737_bff7_39f95c49453b.slice/crio-5f78fbb8534ccaf959e72f642d3c12bf6c9732be5d26941a722d737764d5e5e0 WatchSource:0}: Error finding container 5f78fbb8534ccaf959e72f642d3c12bf6c9732be5d26941a722d737764d5e5e0: Status 404 returned error can't find the container with id 5f78fbb8534ccaf959e72f642d3c12bf6c9732be5d26941a722d737764d5e5e0
Sep 14 03:07:20.401080 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:20.401023    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:20.513434 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:20.513360    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-fb76f4c2-6cbd998d6-wppld" event={"ID":"9d6682c1-8385-4737-bff7-39f95c49453b","Type":"ContainerStarted","Data":"cb1a382037086b89623cf344dd7288af9f68c14c07bbc0ff1e8ac137d019ca04"}
Sep 14 03:07:20.513601 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:20.513539    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-fb76f4c2-6cbd998d6-wppld"
Sep 14 03:07:20.513601 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:20.513573    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-fb76f4c2-6cbd998d6-wppld" event={"ID":"9d6682c1-8385-4737-bff7-39f95c49453b","Type":"ContainerStarted","Data":"5f78fbb8534ccaf959e72f642d3c12bf6c9732be5d26941a722d737764d5e5e0"}
Sep 14 03:07:20.531543 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:20.531491    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-fb76f4c2-6cbd998d6-wppld" podStartSLOduration=1.531475602 podStartE2EDuration="1.531475602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:07:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:07:20.529881253 +0000 UTC m=+885.910965454" watchObservedRunningTime="2026-09-14 03:07:20.531475602 +0000 UTC m=+885.912559901"
Sep 14 03:07:21.155707 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:21.155657    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:22.425961 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:22.425920    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:23.091865 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:23.091819    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:23.121787 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:23.121756    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-ab5750-5b6f7d747d-464xq"]
Sep 14 03:07:23.122026 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:23.121987    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-ab5750-5b6f7d747d-464xq" podUID="2e738de9-f4f0-4ed8-bfb4-c80715863411" containerName="payload-pre-processing" containerID="cri-o://93bc915c7c8e3c3aa37e34582d9b07f47fc02200ad89e671b74a841012314bc5" gracePeriod=30
Sep 14 03:07:23.191860 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:23.191817    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:23.295290 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:23.295175    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:23.381050 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:23.381025    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-ab5750-5b6f7d747d-464xq"
Sep 14 03:07:23.394395 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:23.394360    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:23.499853 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:23.499812    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v4f5c\" (UniqueName: \"kubernetes.io/projected/2e738de9-f4f0-4ed8-bfb4-c80715863411-kube-api-access-v4f5c\") pod \"2e738de9-f4f0-4ed8-bfb4-c80715863411\" (UID: \"2e738de9-f4f0-4ed8-bfb4-c80715863411\") "
Sep 14 03:07:23.500287 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:23.499877    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e738de9-f4f0-4ed8-bfb4-c80715863411-plugins-config-volume\") pod \"2e738de9-f4f0-4ed8-bfb4-c80715863411\" (UID: \"2e738de9-f4f0-4ed8-bfb4-c80715863411\") "
Sep 14 03:07:23.500350 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:23.500306    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e738de9-f4f0-4ed8-bfb4-c80715863411-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2e738de9-f4f0-4ed8-bfb4-c80715863411" (UID: "2e738de9-f4f0-4ed8-bfb4-c80715863411"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 03:07:23.502070 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:23.502045    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e738de9-f4f0-4ed8-bfb4-c80715863411-kube-api-access-v4f5c" (OuterVolumeSpecName: "kube-api-access-v4f5c") pod "2e738de9-f4f0-4ed8-bfb4-c80715863411" (UID: "2e738de9-f4f0-4ed8-bfb4-c80715863411"). InnerVolumeSpecName "kube-api-access-v4f5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:07:23.533158 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:23.533128    2564 generic.go:356] "Generic (PLEG): container finished" podID="2e738de9-f4f0-4ed8-bfb4-c80715863411" containerID="93bc915c7c8e3c3aa37e34582d9b07f47fc02200ad89e671b74a841012314bc5" exitCode=0
Sep 14 03:07:23.533318 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:23.533185    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-ab5750-5b6f7d747d-464xq"
Sep 14 03:07:23.533318 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:23.533221    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-ab5750-5b6f7d747d-464xq" event={"ID":"2e738de9-f4f0-4ed8-bfb4-c80715863411","Type":"ContainerDied","Data":"93bc915c7c8e3c3aa37e34582d9b07f47fc02200ad89e671b74a841012314bc5"}
Sep 14 03:07:23.533318 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:23.533296    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-ab5750-5b6f7d747d-464xq" event={"ID":"2e738de9-f4f0-4ed8-bfb4-c80715863411","Type":"ContainerDied","Data":"c40a8c736b6d865378254e9c9b119e05c6f7deffa4803de43063e0b320c9886a"}
Sep 14 03:07:23.533318 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:23.533318    2564 scope.go:117] "RemoveContainer" containerID="93bc915c7c8e3c3aa37e34582d9b07f47fc02200ad89e671b74a841012314bc5"
Sep 14 03:07:23.543199 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:23.543181    2564 scope.go:117] "RemoveContainer" containerID="93bc915c7c8e3c3aa37e34582d9b07f47fc02200ad89e671b74a841012314bc5"
Sep 14 03:07:23.543551 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:07:23.543524    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93bc915c7c8e3c3aa37e34582d9b07f47fc02200ad89e671b74a841012314bc5\": container with ID starting with 93bc915c7c8e3c3aa37e34582d9b07f47fc02200ad89e671b74a841012314bc5 not found: ID does not exist" containerID="93bc915c7c8e3c3aa37e34582d9b07f47fc02200ad89e671b74a841012314bc5"
Sep 14 03:07:23.543652 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:23.543563    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93bc915c7c8e3c3aa37e34582d9b07f47fc02200ad89e671b74a841012314bc5"} err="failed to get container status \"93bc915c7c8e3c3aa37e34582d9b07f47fc02200ad89e671b74a841012314bc5\": rpc error: code = NotFound desc = could not find container \"93bc915c7c8e3c3aa37e34582d9b07f47fc02200ad89e671b74a841012314bc5\": container with ID starting with 93bc915c7c8e3c3aa37e34582d9b07f47fc02200ad89e671b74a841012314bc5 not found: ID does not exist"
Sep 14 03:07:23.563528 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:23.563494    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-ab5750-5b6f7d747d-464xq"]
Sep 14 03:07:23.571464 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:23.571431    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-ab5750-5b6f7d747d-464xq"]
Sep 14 03:07:23.601988 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:23.601955    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4f5c\" (UniqueName: \"kubernetes.io/projected/2e738de9-f4f0-4ed8-bfb4-c80715863411-kube-api-access-v4f5c\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:07:23.601988 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:23.601991    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e738de9-f4f0-4ed8-bfb4-c80715863411-plugins-config-volume\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:07:23.699220 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:23.699108    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:23.798360 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:23.798302    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:24.569859 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:24.569823    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:25.098581 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:25.098553    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e738de9-f4f0-4ed8-bfb4-c80715863411" path="/var/lib/kubelet/pods/2e738de9-f4f0-4ed8-bfb4-c80715863411/volumes"
Sep 14 03:07:25.309658 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:25.309616    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:26.143093 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:26.143054    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:26.942250 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:26.942208    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:27.666247 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:27.666146    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:28.752346 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:28.752304    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:28.942646 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:28.942604    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:29.752011 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:29.751976    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:30.665866 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:30.665826    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:31.522791 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:31.522760    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-fb76f4c2-6cbd998d6-wppld"
Sep 14 03:07:31.551487 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:31.551454    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:32.343532 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:32.343504    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:33.373140 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:33.373103    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:34.431185 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:34.431147    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:35.330333 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:35.330292    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:39.394534 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:39.394494    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:39.979373 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:39.979337    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:40.033525 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:40.033489    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-fb76f4c2-6cbd998d6-wppld"]
Sep 14 03:07:40.033774 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:40.033748    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-fb76f4c2-6cbd998d6-wppld" podUID="9d6682c1-8385-4737-bff7-39f95c49453b" containerName="payload-pre-processing" containerID="cri-o://cb1a382037086b89623cf344dd7288af9f68c14c07bbc0ff1e8ac137d019ca04" gracePeriod=30
Sep 14 03:07:40.075880 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:40.075850    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:40.176824 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:40.176789    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:40.279947 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:40.279914    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:40.300885 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:40.300858    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-fb76f4c2-6cbd998d6-wppld"
Sep 14 03:07:40.376191 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:40.376159    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:40.387185 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:40.387156    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d6682c1-8385-4737-bff7-39f95c49453b-plugins-config-volume\") pod \"9d6682c1-8385-4737-bff7-39f95c49453b\" (UID: \"9d6682c1-8385-4737-bff7-39f95c49453b\") "
Sep 14 03:07:40.387377 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:40.387255    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7c5fv\" (UniqueName: \"kubernetes.io/projected/9d6682c1-8385-4737-bff7-39f95c49453b-kube-api-access-7c5fv\") pod \"9d6682c1-8385-4737-bff7-39f95c49453b\" (UID: \"9d6682c1-8385-4737-bff7-39f95c49453b\") "
Sep 14 03:07:40.387610 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:40.387586    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d6682c1-8385-4737-bff7-39f95c49453b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9d6682c1-8385-4737-bff7-39f95c49453b" (UID: "9d6682c1-8385-4737-bff7-39f95c49453b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 03:07:40.389488 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:40.389464    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d6682c1-8385-4737-bff7-39f95c49453b-kube-api-access-7c5fv" (OuterVolumeSpecName: "kube-api-access-7c5fv") pod "9d6682c1-8385-4737-bff7-39f95c49453b" (UID: "9d6682c1-8385-4737-bff7-39f95c49453b"). InnerVolumeSpecName "kube-api-access-7c5fv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:07:40.488607 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:40.488569    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d6682c1-8385-4737-bff7-39f95c49453b-plugins-config-volume\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:07:40.488607 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:40.488600    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7c5fv\" (UniqueName: \"kubernetes.io/projected/9d6682c1-8385-4737-bff7-39f95c49453b-kube-api-access-7c5fv\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:07:40.629711 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:40.629611    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:40.635647 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:40.635620    2564 generic.go:356] "Generic (PLEG): container finished" podID="9d6682c1-8385-4737-bff7-39f95c49453b" containerID="cb1a382037086b89623cf344dd7288af9f68c14c07bbc0ff1e8ac137d019ca04" exitCode=0
Sep 14 03:07:40.635816 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:40.635669    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-fb76f4c2-6cbd998d6-wppld"
Sep 14 03:07:40.635816 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:40.635716    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-fb76f4c2-6cbd998d6-wppld" event={"ID":"9d6682c1-8385-4737-bff7-39f95c49453b","Type":"ContainerDied","Data":"cb1a382037086b89623cf344dd7288af9f68c14c07bbc0ff1e8ac137d019ca04"}
Sep 14 03:07:40.635816 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:40.635776    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-fb76f4c2-6cbd998d6-wppld" event={"ID":"9d6682c1-8385-4737-bff7-39f95c49453b","Type":"ContainerDied","Data":"5f78fbb8534ccaf959e72f642d3c12bf6c9732be5d26941a722d737764d5e5e0"}
Sep 14 03:07:40.635816 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:40.635807    2564 scope.go:117] "RemoveContainer" containerID="cb1a382037086b89623cf344dd7288af9f68c14c07bbc0ff1e8ac137d019ca04"
Sep 14 03:07:40.645359 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:40.645340    2564 scope.go:117] "RemoveContainer" containerID="cb1a382037086b89623cf344dd7288af9f68c14c07bbc0ff1e8ac137d019ca04"
Sep 14 03:07:40.645661 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:07:40.645644    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb1a382037086b89623cf344dd7288af9f68c14c07bbc0ff1e8ac137d019ca04\": container with ID starting with cb1a382037086b89623cf344dd7288af9f68c14c07bbc0ff1e8ac137d019ca04 not found: ID does not exist" containerID="cb1a382037086b89623cf344dd7288af9f68c14c07bbc0ff1e8ac137d019ca04"
Sep 14 03:07:40.645713 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:40.645669    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb1a382037086b89623cf344dd7288af9f68c14c07bbc0ff1e8ac137d019ca04"} err="failed to get container status \"cb1a382037086b89623cf344dd7288af9f68c14c07bbc0ff1e8ac137d019ca04\": rpc error: code = NotFound desc = could not find container \"cb1a382037086b89623cf344dd7288af9f68c14c07bbc0ff1e8ac137d019ca04\": container with ID starting with cb1a382037086b89623cf344dd7288af9f68c14c07bbc0ff1e8ac137d019ca04 not found: ID does not exist"
Sep 14 03:07:40.658529 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:40.658499    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-fb76f4c2-6cbd998d6-wppld"]
Sep 14 03:07:40.662825 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:40.662798    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-fb76f4c2-6cbd998d6-wppld"]
Sep 14 03:07:41.029843 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:41.029805    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:41.098669 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:41.098636    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d6682c1-8385-4737-bff7-39f95c49453b" path="/var/lib/kubelet/pods/9d6682c1-8385-4737-bff7-39f95c49453b/volumes"
Sep 14 03:07:41.155599 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:41.155561    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:41.294632 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:41.294541    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:42.108393 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:42.108359    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:42.696387 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:42.696350    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:42.929667 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:42.929625    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:43.629892 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:43.629851    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:44.477251 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:44.477207    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:47.544239 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:47.544196    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:48.818989 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:48.818953    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:48.882183 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:48.882145    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:48.955563 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:48.955526    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:49.057435 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:49.057366    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:49.157398 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:49.157312    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:07:49.540933 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:07:49.540901    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:08:02.270179 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:08:02.270148    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:08:05.727676 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:08:05.727603    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:08:06.031639 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:08:06.031603    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:08:06.418506 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:08:06.418391    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:08:06.716366 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:08:06.716273    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:08:07.091837 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:08:07.091803    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:08:09.433116 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:08:09.433080    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:08:09.533660 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:08:09.533626    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:08:10.037543 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:08:10.037499    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:08:10.247817 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:08:10.247780    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:08:10.346465 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:08:10.346378    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:08:10.599815 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:08:10.599718    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:08:10.855115 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:08:10.855022    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:08:11.836055 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:08:11.836022    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:08:12.176813 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:08:12.176723    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:08:12.569783 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:08:12.569748    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:08:13.476787 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:08:13.476698    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:08:13.575593 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:08:13.575555    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:08:13.979455 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:08:13.979397    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:08:14.076278 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:08:14.076247    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:08:14.169044 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:08:14.169014    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-8364a0-5f5d45748b-tckmm"]
Sep 14 03:08:14.169514 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:08:14.169499    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e738de9-f4f0-4ed8-bfb4-c80715863411" containerName="payload-pre-processing"
Sep 14 03:08:14.169567 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:08:14.169516    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e738de9-f4f0-4ed8-bfb4-c80715863411" containerName="payload-pre-processing"
Sep 14 03:08:14.169567 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:08:14.169533    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d6682c1-8385-4737-bff7-39f95c49453b" containerName="payload-pre-processing"
Sep 14 03:08:14.169567 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:08:14.169539    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d6682c1-8385-4737-bff7-39f95c49453b" containerName="payload-pre-processing"
Sep 14 03:08:14.169696 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:08:14.169632    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d6682c1-8385-4737-bff7-39f95c49453b" containerName="payload-pre-processing"
Sep 14 03:08:14.169696 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:08:14.169641    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e738de9-f4f0-4ed8-bfb4-c80715863411" containerName="payload-pre-processing"
Sep 14 03:08:14.173036 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:08:14.173018    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-8364a0-5f5d45748b-tckmm"
Sep 14 03:08:14.176628 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:08:14.176608    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-8364a0-dockercfg-5j9rf\""
Sep 14 03:08:14.176731 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:08:14.176644    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-8364a0\""
Sep 14 03:08:14.190476 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:08:14.190449    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-8364a0-5f5d45748b-tckmm"]
Sep 14 03:08:14.227039 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:08:14.227008    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2m5v\" (UniqueName: \"kubernetes.io/projected/93cd1971-a8b0-4650-bd95-efafdbf49267-kube-api-access-n2m5v\") pod \"payload-pre-processing-e2e-explicit-8364a0-5f5d45748b-tckmm\" (UID: \"93cd1971-a8b0-4650-bd95-efafdbf49267\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-8364a0-5f5d45748b-tckmm"
Sep 14 03:08:14.227209 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:08:14.227075    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93cd1971-a8b0-4650-bd95-efafdbf49267-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-8364a0-5f5d45748b-tckmm\" (UID: \"93cd1971-a8b0-4650-bd95-efafdbf49267\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-8364a0-5f5d45748b-tckmm"
Sep 14 03:08:14.327786 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:08:14.327746    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2m5v\" (UniqueName: \"kubernetes.io/projected/93cd1971-a8b0-4650-bd95-efafdbf49267-kube-api-access-n2m5v\") pod \"payload-pre-processing-e2e-explicit-8364a0-5f5d45748b-tckmm\" (UID: \"93cd1971-a8b0-4650-bd95-efafdbf49267\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-8364a0-5f5d45748b-tckmm"
Sep 14 03:08:14.327993 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:08:14.327836    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93cd1971-a8b0-4650-bd95-efafdbf49267-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-8364a0-5f5d45748b-tckmm\" (UID: \"93cd1971-a8b0-4650-bd95-efafdbf49267\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-8364a0-5f5d45748b-tckmm"
Sep 14 03:08:14.328543 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:08:14.328525    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93cd1971-a8b0-4650-bd95-efafdbf49267-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-8364a0-5f5d45748b-tckmm\" (UID: \"93cd1971-a8b0-4650-bd95-efafdbf49267\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-8364a0-5f5d45748b-tckmm"
Sep 14 03:08:14.338004 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:08:14.337982    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2m5v\" (UniqueName: \"kubernetes.io/projected/93cd1971-a8b0-4650-bd95-efafdbf49267-kube-api-access-n2m5v\") pod \"payload-pre-processing-e2e-explicit-8364a0-5f5d45748b-tckmm\" (UID: \"93cd1971-a8b0-4650-bd95-efafdbf49267\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-8364a0-5f5d45748b-tckmm"
Sep 14 03:08:14.483353 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:08:14.483314    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-8364a0-5f5d45748b-tckmm"
Sep 14 03:08:14.756354 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:08:14.756327    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-8364a0-5f5d45748b-tckmm"]
Sep 14 03:08:14.757312 ip-10-0-141-4 kubenswrapper[2564]: W0914 03:08:14.757287    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93cd1971_a8b0_4650_bd95_efafdbf49267.slice/crio-ee3446266e5ace5c5c2dd5fe0a2892ed2f71fbce1ddeb6e445d4db3e4569856d WatchSource:0}: Error finding container ee3446266e5ace5c5c2dd5fe0a2892ed2f71fbce1ddeb6e445d4db3e4569856d: Status 404 returned error can't find the container with id ee3446266e5ace5c5c2dd5fe0a2892ed2f71fbce1ddeb6e445d4db3e4569856d
Sep 14 03:08:14.823256 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:08:14.823229    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-8364a0-5f5d45748b-tckmm" event={"ID":"93cd1971-a8b0-4650-bd95-efafdbf49267","Type":"ContainerStarted","Data":"ee3446266e5ace5c5c2dd5fe0a2892ed2f71fbce1ddeb6e445d4db3e4569856d"}
Sep 14 03:08:15.458076 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:08:15.458042    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:08:15.830878 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:08:15.830833    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-8364a0-5f5d45748b-tckmm" event={"ID":"93cd1971-a8b0-4650-bd95-efafdbf49267","Type":"ContainerStarted","Data":"2ebf03c2454c340f57741e9bef1cc68823a36be65116f33aa5f2fcf71422aca2"}
Sep 14 03:08:15.831296 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:08:15.830984    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-8364a0-5f5d45748b-tckmm"
Sep 14 03:08:15.850715 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:08:15.850651    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-8364a0-5f5d45748b-tckmm" podStartSLOduration=1.85062863 podStartE2EDuration="1.85062863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:08:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:08:15.847011414 +0000 UTC m=+941.228095616" watchObservedRunningTime="2026-09-14 03:08:15.85062863 +0000 UTC m=+941.231712832"
Sep 14 03:08:21.345057 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:08:21.345013    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:08:21.461299 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:08:21.461251    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:08:21.582820 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:08:21.582784    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:08:26.845019 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:08:26.844986    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-8364a0-5f5d45748b-tckmm"
Sep 14 03:08:27.256145 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:08:27.256049    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:08:35.621351 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:08:35.621313    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:08:36.330046 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:08:36.330012    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:08:37.070595 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:08:37.070558    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:08:37.426493 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:08:37.426386    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:08:37.964810 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:08:37.964774    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:08:38.641515 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:08:38.641477    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:08:39.441075 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:08:39.441039    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:08:40.147731 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:08:40.147689    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:08:43.755031 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:08:43.754973    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:08:43.864885 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:08:43.864848    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:08:43.960503 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:08:43.960466    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:08:50.394181 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:08:50.394144    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:08:54.470951 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:08:54.470915    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:08:55.205786 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:08:55.205753    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:08:57.487155 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:08:57.487077    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:09:01.609701 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:01.609640    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:09:01.721100 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:01.721065    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:09:01.826040 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:01.826006    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:09:02.577354 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:02.577315    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:09:02.782462 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:02.782425    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:09:04.269475 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:04.269441    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:09:06.783814 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:06.783779    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:09:10.710248 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:10.710217    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:09:10.812690 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:10.812658    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:09:11.409807 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:11.409770    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:09:18.000792 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:18.000758    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:09:18.101097 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:18.101059    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:09:18.258089 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:18.257989    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:09:18.601325 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:18.601291    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:09:18.798624 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:18.798592    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:09:19.420417 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:19.420367    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-8364a0/explicit-model-8364a0-kserve-7c5c649f65-wwszz"]
Sep 14 03:09:19.424347 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:19.424329    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-8364a0/explicit-model-8364a0-kserve-7c5c649f65-wwszz"
Sep 14 03:09:19.428475 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:19.428451    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-8364a0\"/\"default-dockercfg-mmrcq\""
Sep 14 03:09:19.428475 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:19.428460    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-8364a0\"/\"explicit-model-8364a0-kserve-self-signed-certs\""
Sep 14 03:09:19.428475 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:19.428457    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-8364a0\"/\"kube-root-ca.crt\""
Sep 14 03:09:19.428695 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:19.428499    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-8364a0\"/\"openshift-service-ca.crt\""
Sep 14 03:09:19.441921 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:19.441896    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-8364a0/explicit-model-8364a0-kserve-7c5c649f65-wwszz"]
Sep 14 03:09:19.469994 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:19.469957    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:09:19.562888 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:19.562854    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3cd1027-6f90-41ff-81e5-844f9063c493-model-cache\") pod \"explicit-model-8364a0-kserve-7c5c649f65-wwszz\" (UID: \"c3cd1027-6f90-41ff-81e5-844f9063c493\") " pod="ai-tenant-e2e-explicit-8364a0/explicit-model-8364a0-kserve-7c5c649f65-wwszz"
Sep 14 03:09:19.562888 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:19.562888    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kngmw\" (UniqueName: \"kubernetes.io/projected/c3cd1027-6f90-41ff-81e5-844f9063c493-kube-api-access-kngmw\") pod \"explicit-model-8364a0-kserve-7c5c649f65-wwszz\" (UID: \"c3cd1027-6f90-41ff-81e5-844f9063c493\") " pod="ai-tenant-e2e-explicit-8364a0/explicit-model-8364a0-kserve-7c5c649f65-wwszz"
Sep 14 03:09:19.563143 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:19.562912    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3cd1027-6f90-41ff-81e5-844f9063c493-home\") pod \"explicit-model-8364a0-kserve-7c5c649f65-wwszz\" (UID: \"c3cd1027-6f90-41ff-81e5-844f9063c493\") " pod="ai-tenant-e2e-explicit-8364a0/explicit-model-8364a0-kserve-7c5c649f65-wwszz"
Sep 14 03:09:19.563143 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:19.563018    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3cd1027-6f90-41ff-81e5-844f9063c493-tmp-dir\") pod \"explicit-model-8364a0-kserve-7c5c649f65-wwszz\" (UID: \"c3cd1027-6f90-41ff-81e5-844f9063c493\") " pod="ai-tenant-e2e-explicit-8364a0/explicit-model-8364a0-kserve-7c5c649f65-wwszz"
Sep 14 03:09:19.563143 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:19.563088    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3cd1027-6f90-41ff-81e5-844f9063c493-tls-certs\") pod \"explicit-model-8364a0-kserve-7c5c649f65-wwszz\" (UID: \"c3cd1027-6f90-41ff-81e5-844f9063c493\") " pod="ai-tenant-e2e-explicit-8364a0/explicit-model-8364a0-kserve-7c5c649f65-wwszz"
Sep 14 03:09:19.563283 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:19.563188    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3cd1027-6f90-41ff-81e5-844f9063c493-dshm\") pod \"explicit-model-8364a0-kserve-7c5c649f65-wwszz\" (UID: \"c3cd1027-6f90-41ff-81e5-844f9063c493\") " pod="ai-tenant-e2e-explicit-8364a0/explicit-model-8364a0-kserve-7c5c649f65-wwszz"
Sep 14 03:09:19.664063 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:19.664029    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3cd1027-6f90-41ff-81e5-844f9063c493-dshm\") pod \"explicit-model-8364a0-kserve-7c5c649f65-wwszz\" (UID: \"c3cd1027-6f90-41ff-81e5-844f9063c493\") " pod="ai-tenant-e2e-explicit-8364a0/explicit-model-8364a0-kserve-7c5c649f65-wwszz"
Sep 14 03:09:19.664231 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:19.664096    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3cd1027-6f90-41ff-81e5-844f9063c493-model-cache\") pod \"explicit-model-8364a0-kserve-7c5c649f65-wwszz\" (UID: \"c3cd1027-6f90-41ff-81e5-844f9063c493\") " pod="ai-tenant-e2e-explicit-8364a0/explicit-model-8364a0-kserve-7c5c649f65-wwszz"
Sep 14 03:09:19.664231 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:19.664126    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kngmw\" (UniqueName: \"kubernetes.io/projected/c3cd1027-6f90-41ff-81e5-844f9063c493-kube-api-access-kngmw\") pod \"explicit-model-8364a0-kserve-7c5c649f65-wwszz\" (UID: \"c3cd1027-6f90-41ff-81e5-844f9063c493\") " pod="ai-tenant-e2e-explicit-8364a0/explicit-model-8364a0-kserve-7c5c649f65-wwszz"
Sep 14 03:09:19.664231 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:19.664158    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3cd1027-6f90-41ff-81e5-844f9063c493-home\") pod \"explicit-model-8364a0-kserve-7c5c649f65-wwszz\" (UID: \"c3cd1027-6f90-41ff-81e5-844f9063c493\") " pod="ai-tenant-e2e-explicit-8364a0/explicit-model-8364a0-kserve-7c5c649f65-wwszz"
Sep 14 03:09:19.664231 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:19.664195    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3cd1027-6f90-41ff-81e5-844f9063c493-tmp-dir\") pod \"explicit-model-8364a0-kserve-7c5c649f65-wwszz\" (UID: \"c3cd1027-6f90-41ff-81e5-844f9063c493\") " pod="ai-tenant-e2e-explicit-8364a0/explicit-model-8364a0-kserve-7c5c649f65-wwszz"
Sep 14 03:09:19.664445 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:19.664251    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3cd1027-6f90-41ff-81e5-844f9063c493-tls-certs\") pod \"explicit-model-8364a0-kserve-7c5c649f65-wwszz\" (UID: \"c3cd1027-6f90-41ff-81e5-844f9063c493\") " pod="ai-tenant-e2e-explicit-8364a0/explicit-model-8364a0-kserve-7c5c649f65-wwszz"
Sep 14 03:09:19.664606 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:19.664580    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3cd1027-6f90-41ff-81e5-844f9063c493-model-cache\") pod \"explicit-model-8364a0-kserve-7c5c649f65-wwszz\" (UID: \"c3cd1027-6f90-41ff-81e5-844f9063c493\") " pod="ai-tenant-e2e-explicit-8364a0/explicit-model-8364a0-kserve-7c5c649f65-wwszz"
Sep 14 03:09:19.664674 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:19.664605    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3cd1027-6f90-41ff-81e5-844f9063c493-home\") pod \"explicit-model-8364a0-kserve-7c5c649f65-wwszz\" (UID: \"c3cd1027-6f90-41ff-81e5-844f9063c493\") " pod="ai-tenant-e2e-explicit-8364a0/explicit-model-8364a0-kserve-7c5c649f65-wwszz"
Sep 14 03:09:19.664674 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:19.664653    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3cd1027-6f90-41ff-81e5-844f9063c493-tmp-dir\") pod \"explicit-model-8364a0-kserve-7c5c649f65-wwszz\" (UID: \"c3cd1027-6f90-41ff-81e5-844f9063c493\") " pod="ai-tenant-e2e-explicit-8364a0/explicit-model-8364a0-kserve-7c5c649f65-wwszz"
Sep 14 03:09:19.666607 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:19.666585    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3cd1027-6f90-41ff-81e5-844f9063c493-dshm\") pod \"explicit-model-8364a0-kserve-7c5c649f65-wwszz\" (UID: \"c3cd1027-6f90-41ff-81e5-844f9063c493\") " pod="ai-tenant-e2e-explicit-8364a0/explicit-model-8364a0-kserve-7c5c649f65-wwszz"
Sep 14 03:09:19.666758 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:19.666731    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3cd1027-6f90-41ff-81e5-844f9063c493-tls-certs\") pod \"explicit-model-8364a0-kserve-7c5c649f65-wwszz\" (UID: \"c3cd1027-6f90-41ff-81e5-844f9063c493\") " pod="ai-tenant-e2e-explicit-8364a0/explicit-model-8364a0-kserve-7c5c649f65-wwszz"
Sep 14 03:09:19.672874 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:19.672814    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kngmw\" (UniqueName: \"kubernetes.io/projected/c3cd1027-6f90-41ff-81e5-844f9063c493-kube-api-access-kngmw\") pod \"explicit-model-8364a0-kserve-7c5c649f65-wwszz\" (UID: \"c3cd1027-6f90-41ff-81e5-844f9063c493\") " pod="ai-tenant-e2e-explicit-8364a0/explicit-model-8364a0-kserve-7c5c649f65-wwszz"
Sep 14 03:09:19.735330 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:19.735296    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-8364a0/explicit-model-8364a0-kserve-7c5c649f65-wwszz"
Sep 14 03:09:20.012661 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:20.012631    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-8364a0/explicit-model-8364a0-kserve-7c5c649f65-wwszz"]
Sep 14 03:09:20.013711 ip-10-0-141-4 kubenswrapper[2564]: W0914 03:09:20.013689    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc3cd1027_6f90_41ff_81e5_844f9063c493.slice/crio-b16cfab1bf6f14d816b39c19e1b7f4178dae7e72fc46bc91d76e8496adbe4f7b WatchSource:0}: Error finding container b16cfab1bf6f14d816b39c19e1b7f4178dae7e72fc46bc91d76e8496adbe4f7b: Status 404 returned error can't find the container with id b16cfab1bf6f14d816b39c19e1b7f4178dae7e72fc46bc91d76e8496adbe4f7b
Sep 14 03:09:20.179196 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:20.179156    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-8364a0/explicit-model-8364a0-kserve-7c5c649f65-wwszz" event={"ID":"c3cd1027-6f90-41ff-81e5-844f9063c493","Type":"ContainerStarted","Data":"a8b7f05e8ece646e838be8b5cd6e9a877f134e3c1f8cf84b4ea800445a7b6511"}
Sep 14 03:09:20.179341 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:20.179205    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-8364a0/explicit-model-8364a0-kserve-7c5c649f65-wwszz" event={"ID":"c3cd1027-6f90-41ff-81e5-844f9063c493","Type":"ContainerStarted","Data":"b16cfab1bf6f14d816b39c19e1b7f4178dae7e72fc46bc91d76e8496adbe4f7b"}
Sep 14 03:09:20.212337 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:20.212290    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-8364a0/explicit-model-8364a0-kserve-7c5c649f65-wwszz" podStartSLOduration=1.212275658 podStartE2EDuration="1.212275658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:09:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:09:20.209924705 +0000 UTC m=+1005.591008900" watchObservedRunningTime="2026-09-14 03:09:20.212275658 +0000 UTC m=+1005.593359857"
Sep 14 03:09:24.644203 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:24.644169    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:09:25.325799 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:25.325760    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:09:25.428152 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:25.428116    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:09:26.174129 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:26.174062    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:09:26.965752 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:26.965718    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:09:27.672112 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:27.672079    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:09:28.350858 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:28.350810    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:09:29.249434 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:29.249396    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:09:29.312878 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:29.312829    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:09:29.654253 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:29.654211    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:09:29.736104 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:29.736069    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-8364a0/explicit-model-8364a0-kserve-7c5c649f65-wwszz"
Sep 14 03:09:29.736272 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:29.736129    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-8364a0/explicit-model-8364a0-kserve-7c5c649f65-wwszz"
Sep 14 03:09:29.752159 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:29.752129    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:09:29.753655 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:29.753631    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-8364a0/explicit-model-8364a0-kserve-7c5c649f65-wwszz"
Sep 14 03:09:29.835870 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:29.835839    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:09:30.246277 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:30.246246    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-8364a0/explicit-model-8364a0-kserve-7c5c649f65-wwszz"
Sep 14 03:09:46.966373 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:46.966338    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-a-7a5ebdfc-kserve-f7f5cb5c6-8clfl"]
Sep 14 03:09:46.971041 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:46.971018    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-a-7a5ebdfc-kserve-f7f5cb5c6-8clfl"
Sep 14 03:09:46.975252 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:46.975224    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-9c9f1e\"/\"openshift-service-ca.crt\""
Sep 14 03:09:46.976462 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:46.976439    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-9c9f1e\"/\"e2e-conflict-a-7a5ebdfc-kserve-self-signed-certs\""
Sep 14 03:09:46.976764 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:46.976744    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-9c9f1e\"/\"kube-root-ca.crt\""
Sep 14 03:09:46.976857 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:46.976772    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-9c9f1e\"/\"default-dockercfg-5qjfv\""
Sep 14 03:09:46.987992 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:46.987964    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-a-7a5ebdfc-kserve-f7f5cb5c6-8clfl"]
Sep 14 03:09:47.033336 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:47.033305    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a524982-7e25-4967-9efd-121d9903d4e3-tmp-dir\") pod \"e2e-conflict-a-7a5ebdfc-kserve-f7f5cb5c6-8clfl\" (UID: \"5a524982-7e25-4967-9efd-121d9903d4e3\") " pod="e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-a-7a5ebdfc-kserve-f7f5cb5c6-8clfl"
Sep 14 03:09:47.033522 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:47.033354    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a524982-7e25-4967-9efd-121d9903d4e3-home\") pod \"e2e-conflict-a-7a5ebdfc-kserve-f7f5cb5c6-8clfl\" (UID: \"5a524982-7e25-4967-9efd-121d9903d4e3\") " pod="e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-a-7a5ebdfc-kserve-f7f5cb5c6-8clfl"
Sep 14 03:09:47.033598 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:47.033516    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwlbt\" (UniqueName: \"kubernetes.io/projected/5a524982-7e25-4967-9efd-121d9903d4e3-kube-api-access-nwlbt\") pod \"e2e-conflict-a-7a5ebdfc-kserve-f7f5cb5c6-8clfl\" (UID: \"5a524982-7e25-4967-9efd-121d9903d4e3\") " pod="e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-a-7a5ebdfc-kserve-f7f5cb5c6-8clfl"
Sep 14 03:09:47.033598 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:47.033572    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a524982-7e25-4967-9efd-121d9903d4e3-dshm\") pod \"e2e-conflict-a-7a5ebdfc-kserve-f7f5cb5c6-8clfl\" (UID: \"5a524982-7e25-4967-9efd-121d9903d4e3\") " pod="e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-a-7a5ebdfc-kserve-f7f5cb5c6-8clfl"
Sep 14 03:09:47.033690 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:47.033638    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a524982-7e25-4967-9efd-121d9903d4e3-model-cache\") pod \"e2e-conflict-a-7a5ebdfc-kserve-f7f5cb5c6-8clfl\" (UID: \"5a524982-7e25-4967-9efd-121d9903d4e3\") " pod="e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-a-7a5ebdfc-kserve-f7f5cb5c6-8clfl"
Sep 14 03:09:47.033690 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:47.033672    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a524982-7e25-4967-9efd-121d9903d4e3-tls-certs\") pod \"e2e-conflict-a-7a5ebdfc-kserve-f7f5cb5c6-8clfl\" (UID: \"5a524982-7e25-4967-9efd-121d9903d4e3\") " pod="e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-a-7a5ebdfc-kserve-f7f5cb5c6-8clfl"
Sep 14 03:09:47.038748 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:47.038720    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:09:47.134371 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:47.134339    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwlbt\" (UniqueName: \"kubernetes.io/projected/5a524982-7e25-4967-9efd-121d9903d4e3-kube-api-access-nwlbt\") pod \"e2e-conflict-a-7a5ebdfc-kserve-f7f5cb5c6-8clfl\" (UID: \"5a524982-7e25-4967-9efd-121d9903d4e3\") " pod="e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-a-7a5ebdfc-kserve-f7f5cb5c6-8clfl"
Sep 14 03:09:47.134552 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:47.134387    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a524982-7e25-4967-9efd-121d9903d4e3-dshm\") pod \"e2e-conflict-a-7a5ebdfc-kserve-f7f5cb5c6-8clfl\" (UID: \"5a524982-7e25-4967-9efd-121d9903d4e3\") " pod="e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-a-7a5ebdfc-kserve-f7f5cb5c6-8clfl"
Sep 14 03:09:47.134552 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:47.134454    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a524982-7e25-4967-9efd-121d9903d4e3-model-cache\") pod \"e2e-conflict-a-7a5ebdfc-kserve-f7f5cb5c6-8clfl\" (UID: \"5a524982-7e25-4967-9efd-121d9903d4e3\") " pod="e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-a-7a5ebdfc-kserve-f7f5cb5c6-8clfl"
Sep 14 03:09:47.134552 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:47.134475    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a524982-7e25-4967-9efd-121d9903d4e3-tls-certs\") pod \"e2e-conflict-a-7a5ebdfc-kserve-f7f5cb5c6-8clfl\" (UID: \"5a524982-7e25-4967-9efd-121d9903d4e3\") " pod="e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-a-7a5ebdfc-kserve-f7f5cb5c6-8clfl"
Sep 14 03:09:47.134552 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:47.134546    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a524982-7e25-4967-9efd-121d9903d4e3-tmp-dir\") pod \"e2e-conflict-a-7a5ebdfc-kserve-f7f5cb5c6-8clfl\" (UID: \"5a524982-7e25-4967-9efd-121d9903d4e3\") " pod="e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-a-7a5ebdfc-kserve-f7f5cb5c6-8clfl"
Sep 14 03:09:47.134770 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:47.134573    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a524982-7e25-4967-9efd-121d9903d4e3-home\") pod \"e2e-conflict-a-7a5ebdfc-kserve-f7f5cb5c6-8clfl\" (UID: \"5a524982-7e25-4967-9efd-121d9903d4e3\") " pod="e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-a-7a5ebdfc-kserve-f7f5cb5c6-8clfl"
Sep 14 03:09:47.134968 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:47.134923    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a524982-7e25-4967-9efd-121d9903d4e3-model-cache\") pod \"e2e-conflict-a-7a5ebdfc-kserve-f7f5cb5c6-8clfl\" (UID: \"5a524982-7e25-4967-9efd-121d9903d4e3\") " pod="e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-a-7a5ebdfc-kserve-f7f5cb5c6-8clfl"
Sep 14 03:09:47.135088 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:47.135048    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a524982-7e25-4967-9efd-121d9903d4e3-tmp-dir\") pod \"e2e-conflict-a-7a5ebdfc-kserve-f7f5cb5c6-8clfl\" (UID: \"5a524982-7e25-4967-9efd-121d9903d4e3\") " pod="e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-a-7a5ebdfc-kserve-f7f5cb5c6-8clfl"
Sep 14 03:09:47.135088 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:47.135070    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a524982-7e25-4967-9efd-121d9903d4e3-home\") pod \"e2e-conflict-a-7a5ebdfc-kserve-f7f5cb5c6-8clfl\" (UID: \"5a524982-7e25-4967-9efd-121d9903d4e3\") " pod="e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-a-7a5ebdfc-kserve-f7f5cb5c6-8clfl"
Sep 14 03:09:47.136834 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:47.136814    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a524982-7e25-4967-9efd-121d9903d4e3-dshm\") pod \"e2e-conflict-a-7a5ebdfc-kserve-f7f5cb5c6-8clfl\" (UID: \"5a524982-7e25-4967-9efd-121d9903d4e3\") " pod="e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-a-7a5ebdfc-kserve-f7f5cb5c6-8clfl"
Sep 14 03:09:47.137080 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:47.137064    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a524982-7e25-4967-9efd-121d9903d4e3-tls-certs\") pod \"e2e-conflict-a-7a5ebdfc-kserve-f7f5cb5c6-8clfl\" (UID: \"5a524982-7e25-4967-9efd-121d9903d4e3\") " pod="e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-a-7a5ebdfc-kserve-f7f5cb5c6-8clfl"
Sep 14 03:09:47.145725 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:47.145700    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwlbt\" (UniqueName: \"kubernetes.io/projected/5a524982-7e25-4967-9efd-121d9903d4e3-kube-api-access-nwlbt\") pod \"e2e-conflict-a-7a5ebdfc-kserve-f7f5cb5c6-8clfl\" (UID: \"5a524982-7e25-4967-9efd-121d9903d4e3\") " pod="e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-a-7a5ebdfc-kserve-f7f5cb5c6-8clfl"
Sep 14 03:09:47.281562 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:47.281536    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-a-7a5ebdfc-kserve-f7f5cb5c6-8clfl"
Sep 14 03:09:47.566315 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:47.566290    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-a-7a5ebdfc-kserve-f7f5cb5c6-8clfl"]
Sep 14 03:09:47.569040 ip-10-0-141-4 kubenswrapper[2564]: W0914 03:09:47.569010    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a524982_7e25_4967_9efd_121d9903d4e3.slice/crio-3538f76112c63ab7eb3ed8c9cda549a5bc32ec1c2f36d3f8c8b170298a5f3c3a WatchSource:0}: Error finding container 3538f76112c63ab7eb3ed8c9cda549a5bc32ec1c2f36d3f8c8b170298a5f3c3a: Status 404 returned error can't find the container with id 3538f76112c63ab7eb3ed8c9cda549a5bc32ec1c2f36d3f8c8b170298a5f3c3a
Sep 14 03:09:48.337789 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:48.337752    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-a-7a5ebdfc-kserve-f7f5cb5c6-8clfl" event={"ID":"5a524982-7e25-4967-9efd-121d9903d4e3","Type":"ContainerStarted","Data":"c92fd205d10c49f6b883330493e3ef7251fce6e6feea592e4da9bf306ced030b"}
Sep 14 03:09:48.338169 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:48.337799    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-a-7a5ebdfc-kserve-f7f5cb5c6-8clfl" event={"ID":"5a524982-7e25-4967-9efd-121d9903d4e3","Type":"ContainerStarted","Data":"3538f76112c63ab7eb3ed8c9cda549a5bc32ec1c2f36d3f8c8b170298a5f3c3a"}
Sep 14 03:09:48.359545 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:48.359499    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-a-7a5ebdfc-kserve-f7f5cb5c6-8clfl" podStartSLOduration=2.359485356 podStartE2EDuration="2.359485356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:09:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:09:48.356938835 +0000 UTC m=+1033.738023034" watchObservedRunningTime="2026-09-14 03:09:48.359485356 +0000 UTC m=+1033.740569555"
Sep 14 03:09:49.932529 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:49.932494    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-8364a0/explicit-model-8364a0-kserve-7c5c649f65-wwszz"]
Sep 14 03:09:49.956072 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:49.956041    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:09:50.396861 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:50.396828    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:09:51.239003 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:51.238955    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:09:51.936898 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:51.936852    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:09:52.670676 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:52.670640    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:09:53.368790 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:53.368758    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:09:54.255869 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:54.255835    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:09:54.914972 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:54.914929    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:09:55.499077 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:55.499035    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:09:55.740951 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:55.740920    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:09:56.206137 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:56.206087    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:09:56.301330 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:56.301282    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:09:56.648836 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:56.648804    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:09:56.758105 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:56.758060    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:09:57.281984 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:57.281954    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-a-7a5ebdfc-kserve-f7f5cb5c6-8clfl"
Sep 14 03:09:57.282153 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:57.282003    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-a-7a5ebdfc-kserve-f7f5cb5c6-8clfl"
Sep 14 03:09:57.295686 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:57.295659    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-a-7a5ebdfc-kserve-f7f5cb5c6-8clfl"
Sep 14 03:09:57.399219 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:57.399193    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-a-7a5ebdfc-kserve-f7f5cb5c6-8clfl"
Sep 14 03:09:57.551452 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:09:57.551360    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:10:00.229576 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:00.229539    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:10:00.985250 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:00.985208    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:10:01.026645 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:01.026614    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-8364a0-5f5d45748b-tckmm"]
Sep 14 03:10:01.026871 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:01.026841    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-8364a0-5f5d45748b-tckmm" podUID="93cd1971-a8b0-4650-bd95-efafdbf49267" containerName="payload-pre-processing" containerID="cri-o://2ebf03c2454c340f57741e9bef1cc68823a36be65116f33aa5f2fcf71422aca2" gracePeriod=30
Sep 14 03:10:01.085989 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:01.085933    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:10:01.186667 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:01.186617    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:10:01.289291 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:01.289265    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-8364a0-5f5d45748b-tckmm"
Sep 14 03:10:01.291099 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:01.291071    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:10:01.385262 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:01.385231    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93cd1971-a8b0-4650-bd95-efafdbf49267-plugins-config-volume\") pod \"93cd1971-a8b0-4650-bd95-efafdbf49267\" (UID: \"93cd1971-a8b0-4650-bd95-efafdbf49267\") "
Sep 14 03:10:01.385439 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:01.385280    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n2m5v\" (UniqueName: \"kubernetes.io/projected/93cd1971-a8b0-4650-bd95-efafdbf49267-kube-api-access-n2m5v\") pod \"93cd1971-a8b0-4650-bd95-efafdbf49267\" (UID: \"93cd1971-a8b0-4650-bd95-efafdbf49267\") "
Sep 14 03:10:01.385574 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:01.385553    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93cd1971-a8b0-4650-bd95-efafdbf49267-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "93cd1971-a8b0-4650-bd95-efafdbf49267" (UID: "93cd1971-a8b0-4650-bd95-efafdbf49267"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 03:10:01.387221 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:01.387198    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93cd1971-a8b0-4650-bd95-efafdbf49267-kube-api-access-n2m5v" (OuterVolumeSpecName: "kube-api-access-n2m5v") pod "93cd1971-a8b0-4650-bd95-efafdbf49267" (UID: "93cd1971-a8b0-4650-bd95-efafdbf49267"). InnerVolumeSpecName "kube-api-access-n2m5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:10:01.410062 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:01.410028    2564 generic.go:356] "Generic (PLEG): container finished" podID="93cd1971-a8b0-4650-bd95-efafdbf49267" containerID="2ebf03c2454c340f57741e9bef1cc68823a36be65116f33aa5f2fcf71422aca2" exitCode=0
Sep 14 03:10:01.410239 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:01.410088    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-8364a0-5f5d45748b-tckmm"
Sep 14 03:10:01.410239 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:01.410110    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-8364a0-5f5d45748b-tckmm" event={"ID":"93cd1971-a8b0-4650-bd95-efafdbf49267","Type":"ContainerDied","Data":"2ebf03c2454c340f57741e9bef1cc68823a36be65116f33aa5f2fcf71422aca2"}
Sep 14 03:10:01.410239 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:01.410170    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-8364a0-5f5d45748b-tckmm" event={"ID":"93cd1971-a8b0-4650-bd95-efafdbf49267","Type":"ContainerDied","Data":"ee3446266e5ace5c5c2dd5fe0a2892ed2f71fbce1ddeb6e445d4db3e4569856d"}
Sep 14 03:10:01.410239 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:01.410196    2564 scope.go:117] "RemoveContainer" containerID="2ebf03c2454c340f57741e9bef1cc68823a36be65116f33aa5f2fcf71422aca2"
Sep 14 03:10:01.419522 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:01.419490    2564 scope.go:117] "RemoveContainer" containerID="2ebf03c2454c340f57741e9bef1cc68823a36be65116f33aa5f2fcf71422aca2"
Sep 14 03:10:01.419909 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:10:01.419883    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ebf03c2454c340f57741e9bef1cc68823a36be65116f33aa5f2fcf71422aca2\": container with ID starting with 2ebf03c2454c340f57741e9bef1cc68823a36be65116f33aa5f2fcf71422aca2 not found: ID does not exist" containerID="2ebf03c2454c340f57741e9bef1cc68823a36be65116f33aa5f2fcf71422aca2"
Sep 14 03:10:01.420022 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:01.419919    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ebf03c2454c340f57741e9bef1cc68823a36be65116f33aa5f2fcf71422aca2"} err="failed to get container status \"2ebf03c2454c340f57741e9bef1cc68823a36be65116f33aa5f2fcf71422aca2\": rpc error: code = NotFound desc = could not find container \"2ebf03c2454c340f57741e9bef1cc68823a36be65116f33aa5f2fcf71422aca2\": container with ID starting with 2ebf03c2454c340f57741e9bef1cc68823a36be65116f33aa5f2fcf71422aca2 not found: ID does not exist"
Sep 14 03:10:01.435488 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:01.435438    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-8364a0-5f5d45748b-tckmm"]
Sep 14 03:10:01.437682 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:01.437659    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-8364a0-5f5d45748b-tckmm"]
Sep 14 03:10:01.486422 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:01.486378    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93cd1971-a8b0-4650-bd95-efafdbf49267-plugins-config-volume\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:10:01.486588 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:01.486447    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2m5v\" (UniqueName: \"kubernetes.io/projected/93cd1971-a8b0-4650-bd95-efafdbf49267-kube-api-access-n2m5v\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:10:01.594032 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:01.593996    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:10:01.725980 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:01.725944    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:10:02.575631 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:02.575597    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:10:03.098736 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:03.098706    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93cd1971-a8b0-4650-bd95-efafdbf49267" path="/var/lib/kubelet/pods/93cd1971-a8b0-4650-bd95-efafdbf49267/volumes"
Sep 14 03:10:03.687200 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:03.687161    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:10:04.317189 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:04.317152    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:10:04.498853 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:04.498820    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:10:04.598276 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:04.598174    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:10:04.952504 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:04.952369    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-8364a0/explicit-model-8364a0-kserve-7c5c649f65-wwszz" podUID="c3cd1027-6f90-41ff-81e5-844f9063c493" containerName="main" containerID="cri-o://a8b7f05e8ece646e838be8b5cd6e9a877f134e3c1f8cf84b4ea800445a7b6511" gracePeriod=45
Sep 14 03:10:05.209734 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:05.209670    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-8364a0/explicit-model-8364a0-kserve-7c5c649f65-wwszz"
Sep 14 03:10:05.223728 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:05.223700    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3cd1027-6f90-41ff-81e5-844f9063c493-model-cache\") pod \"c3cd1027-6f90-41ff-81e5-844f9063c493\" (UID: \"c3cd1027-6f90-41ff-81e5-844f9063c493\") "
Sep 14 03:10:05.223892 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:05.223739    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3cd1027-6f90-41ff-81e5-844f9063c493-home\") pod \"c3cd1027-6f90-41ff-81e5-844f9063c493\" (UID: \"c3cd1027-6f90-41ff-81e5-844f9063c493\") "
Sep 14 03:10:05.223892 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:05.223835    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3cd1027-6f90-41ff-81e5-844f9063c493-tmp-dir\") pod \"c3cd1027-6f90-41ff-81e5-844f9063c493\" (UID: \"c3cd1027-6f90-41ff-81e5-844f9063c493\") "
Sep 14 03:10:05.223892 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:05.223859    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3cd1027-6f90-41ff-81e5-844f9063c493-dshm\") pod \"c3cd1027-6f90-41ff-81e5-844f9063c493\" (UID: \"c3cd1027-6f90-41ff-81e5-844f9063c493\") "
Sep 14 03:10:05.223892 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:05.223878    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3cd1027-6f90-41ff-81e5-844f9063c493-tls-certs\") pod \"c3cd1027-6f90-41ff-81e5-844f9063c493\" (UID: \"c3cd1027-6f90-41ff-81e5-844f9063c493\") "
Sep 14 03:10:05.224069 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:05.223899    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kngmw\" (UniqueName: \"kubernetes.io/projected/c3cd1027-6f90-41ff-81e5-844f9063c493-kube-api-access-kngmw\") pod \"c3cd1027-6f90-41ff-81e5-844f9063c493\" (UID: \"c3cd1027-6f90-41ff-81e5-844f9063c493\") "
Sep 14 03:10:05.224069 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:05.223969    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3cd1027-6f90-41ff-81e5-844f9063c493-model-cache" (OuterVolumeSpecName: "model-cache") pod "c3cd1027-6f90-41ff-81e5-844f9063c493" (UID: "c3cd1027-6f90-41ff-81e5-844f9063c493"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:10:05.224177 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:05.224063    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3cd1027-6f90-41ff-81e5-844f9063c493-home" (OuterVolumeSpecName: "home") pod "c3cd1027-6f90-41ff-81e5-844f9063c493" (UID: "c3cd1027-6f90-41ff-81e5-844f9063c493"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:10:05.224177 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:05.224147    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3cd1027-6f90-41ff-81e5-844f9063c493-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c3cd1027-6f90-41ff-81e5-844f9063c493" (UID: "c3cd1027-6f90-41ff-81e5-844f9063c493"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:10:05.224281 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:05.224250    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3cd1027-6f90-41ff-81e5-844f9063c493-model-cache\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:10:05.224281 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:05.224264    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3cd1027-6f90-41ff-81e5-844f9063c493-home\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:10:05.224281 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:05.224272    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3cd1027-6f90-41ff-81e5-844f9063c493-tmp-dir\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:10:05.226204 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:05.226182    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3cd1027-6f90-41ff-81e5-844f9063c493-dshm" (OuterVolumeSpecName: "dshm") pod "c3cd1027-6f90-41ff-81e5-844f9063c493" (UID: "c3cd1027-6f90-41ff-81e5-844f9063c493"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:10:05.226204 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:05.226192    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3cd1027-6f90-41ff-81e5-844f9063c493-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c3cd1027-6f90-41ff-81e5-844f9063c493" (UID: "c3cd1027-6f90-41ff-81e5-844f9063c493"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 03:10:05.226349 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:05.226219    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3cd1027-6f90-41ff-81e5-844f9063c493-kube-api-access-kngmw" (OuterVolumeSpecName: "kube-api-access-kngmw") pod "c3cd1027-6f90-41ff-81e5-844f9063c493" (UID: "c3cd1027-6f90-41ff-81e5-844f9063c493"). InnerVolumeSpecName "kube-api-access-kngmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:10:05.324843 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:05.324815    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3cd1027-6f90-41ff-81e5-844f9063c493-dshm\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:10:05.324843 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:05.324841    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3cd1027-6f90-41ff-81e5-844f9063c493-tls-certs\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:10:05.324843 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:05.324851    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kngmw\" (UniqueName: \"kubernetes.io/projected/c3cd1027-6f90-41ff-81e5-844f9063c493-kube-api-access-kngmw\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:10:05.436281 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:05.436248    2564 generic.go:356] "Generic (PLEG): container finished" podID="c3cd1027-6f90-41ff-81e5-844f9063c493" containerID="a8b7f05e8ece646e838be8b5cd6e9a877f134e3c1f8cf84b4ea800445a7b6511" exitCode=0
Sep 14 03:10:05.436504 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:05.436302    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-8364a0/explicit-model-8364a0-kserve-7c5c649f65-wwszz"
Sep 14 03:10:05.436504 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:05.436335    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-8364a0/explicit-model-8364a0-kserve-7c5c649f65-wwszz" event={"ID":"c3cd1027-6f90-41ff-81e5-844f9063c493","Type":"ContainerDied","Data":"a8b7f05e8ece646e838be8b5cd6e9a877f134e3c1f8cf84b4ea800445a7b6511"}
Sep 14 03:10:05.436504 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:05.436377    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-8364a0/explicit-model-8364a0-kserve-7c5c649f65-wwszz" event={"ID":"c3cd1027-6f90-41ff-81e5-844f9063c493","Type":"ContainerDied","Data":"b16cfab1bf6f14d816b39c19e1b7f4178dae7e72fc46bc91d76e8496adbe4f7b"}
Sep 14 03:10:05.436504 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:05.436393    2564 scope.go:117] "RemoveContainer" containerID="a8b7f05e8ece646e838be8b5cd6e9a877f134e3c1f8cf84b4ea800445a7b6511"
Sep 14 03:10:05.446318 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:05.446302    2564 scope.go:117] "RemoveContainer" containerID="a8b7f05e8ece646e838be8b5cd6e9a877f134e3c1f8cf84b4ea800445a7b6511"
Sep 14 03:10:05.446655 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:10:05.446634    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8b7f05e8ece646e838be8b5cd6e9a877f134e3c1f8cf84b4ea800445a7b6511\": container with ID starting with a8b7f05e8ece646e838be8b5cd6e9a877f134e3c1f8cf84b4ea800445a7b6511 not found: ID does not exist" containerID="a8b7f05e8ece646e838be8b5cd6e9a877f134e3c1f8cf84b4ea800445a7b6511"
Sep 14 03:10:05.446749 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:05.446661    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8b7f05e8ece646e838be8b5cd6e9a877f134e3c1f8cf84b4ea800445a7b6511"} err="failed to get container status \"a8b7f05e8ece646e838be8b5cd6e9a877f134e3c1f8cf84b4ea800445a7b6511\": rpc error: code = NotFound desc = could not find container \"a8b7f05e8ece646e838be8b5cd6e9a877f134e3c1f8cf84b4ea800445a7b6511\": container with ID starting with a8b7f05e8ece646e838be8b5cd6e9a877f134e3c1f8cf84b4ea800445a7b6511 not found: ID does not exist"
Sep 14 03:10:05.453262 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:05.453237    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:10:05.462296 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:05.462237    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-8364a0/explicit-model-8364a0-kserve-7c5c649f65-wwszz"]
Sep 14 03:10:05.470354 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:05.470326    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-8364a0/explicit-model-8364a0-kserve-7c5c649f65-wwszz"]
Sep 14 03:10:06.187802 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:06.187762    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:10:06.930565 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:06.930527    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:10:07.099009 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:07.098974    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3cd1027-6f90-41ff-81e5-844f9063c493" path="/var/lib/kubelet/pods/c3cd1027-6f90-41ff-81e5-844f9063c493/volumes"
Sep 14 03:10:07.627294 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:07.627250    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:10:08.039973 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:08.039937    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:10:08.158511 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:08.158462    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:10:08.422324 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:08.422224    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:10:08.514302 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:08.514249    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:10:08.811329 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:08.811280    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:10:09.152497 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:09.152381    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:10:09.247720 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:09.247687    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:10:10.116873 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:10.116837    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:10:10.217704 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:10.217671    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:10:10.375050 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:10.374960    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:10:14.926003 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:14.925968    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:10:15.495502 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:15.495470    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:10:15.596891 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:15.596855    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:10:15.675144 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:15.675106    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:10:15.748549 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:15.748449    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:10:15.943851 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:15.943795    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:10:16.051679 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:16.051648    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:10:16.743974 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:16.743931    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:10:17.639455 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:17.639391    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:10:18.448518 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:18.448486    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:10:19.147189 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:19.147157    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:10:22.145097 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:22.145058    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:10:22.251780 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:22.251730    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:10:22.441016 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:22.440925    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:10:22.805221 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:22.805188    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:10:22.907355 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:22.907322    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:10:23.037488 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:23.037451    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:10:26.765094 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:26.765060    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:10:28.796369 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:28.796286    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:10:28.858616 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:28.858585    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:10:28.947029 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:28.946994    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:10:29.045547 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:29.045508    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:10:29.159066 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:29.158976    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:10:32.345015 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:32.344984    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:10:35.234557 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:35.234519    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:10:35.300132 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:35.300095    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:10:36.143800 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:36.143767    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:10:36.458377 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:36.458289    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:10:36.545832 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:36.545801    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:10:38.428239 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:38.428208    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:10:39.336914 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:39.336882    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:10:39.939367 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:39.939331    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:10:40.030854 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:40.030818    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:10:40.130703 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:40.130668    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:10:41.040667 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:41.040631    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:10:41.770270 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:41.770238    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:10:42.462245 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:42.462214    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:10:43.229198 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:43.229161    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:10:43.331765 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:43.331732    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:10:43.429516 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:43.429480    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:10:43.711592 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:43.711561    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:10:43.808660 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:43.808627    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:10:43.817517 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:43.817492    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ce5b9b-8f77c865d-d2qkn"]
Sep 14 03:10:43.817955 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:43.817943    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93cd1971-a8b0-4650-bd95-efafdbf49267" containerName="payload-pre-processing"
Sep 14 03:10:43.818008 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:43.817957    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="93cd1971-a8b0-4650-bd95-efafdbf49267" containerName="payload-pre-processing"
Sep 14 03:10:43.818008 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:43.817977    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3cd1027-6f90-41ff-81e5-844f9063c493" containerName="main"
Sep 14 03:10:43.818008 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:43.817984    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3cd1027-6f90-41ff-81e5-844f9063c493" containerName="main"
Sep 14 03:10:43.818108 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:43.818055    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3cd1027-6f90-41ff-81e5-844f9063c493" containerName="main"
Sep 14 03:10:43.818108 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:43.818063    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="93cd1971-a8b0-4650-bd95-efafdbf49267" containerName="payload-pre-processing"
Sep 14 03:10:43.822594 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:43.822575    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ce5b9b-8f77c865d-d2qkn"
Sep 14 03:10:43.825210 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:43.825191    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-ce5b9b\""
Sep 14 03:10:43.825347 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:43.825329    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-ce5b9b-dockercfg-mwrcw\""
Sep 14 03:10:43.837052 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:43.837030    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ce5b9b-8f77c865d-d2qkn"]
Sep 14 03:10:43.909636 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:43.909602    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3eed608f-4d31-4708-ac7c-9a29fe254a1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-ce5b9b-8f77c865d-d2qkn\" (UID: \"3eed608f-4d31-4708-ac7c-9a29fe254a1b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ce5b9b-8f77c865d-d2qkn"
Sep 14 03:10:43.909787 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:43.909658    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wghk2\" (UniqueName: \"kubernetes.io/projected/3eed608f-4d31-4708-ac7c-9a29fe254a1b-kube-api-access-wghk2\") pod \"payload-pre-processing-e2e-shared-a-w2-ce5b9b-8f77c865d-d2qkn\" (UID: \"3eed608f-4d31-4708-ac7c-9a29fe254a1b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ce5b9b-8f77c865d-d2qkn"
Sep 14 03:10:44.010895 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:44.010813    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3eed608f-4d31-4708-ac7c-9a29fe254a1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-ce5b9b-8f77c865d-d2qkn\" (UID: \"3eed608f-4d31-4708-ac7c-9a29fe254a1b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ce5b9b-8f77c865d-d2qkn"
Sep 14 03:10:44.010895 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:44.010871    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wghk2\" (UniqueName: \"kubernetes.io/projected/3eed608f-4d31-4708-ac7c-9a29fe254a1b-kube-api-access-wghk2\") pod \"payload-pre-processing-e2e-shared-a-w2-ce5b9b-8f77c865d-d2qkn\" (UID: \"3eed608f-4d31-4708-ac7c-9a29fe254a1b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ce5b9b-8f77c865d-d2qkn"
Sep 14 03:10:44.011532 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:44.011512    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3eed608f-4d31-4708-ac7c-9a29fe254a1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-ce5b9b-8f77c865d-d2qkn\" (UID: \"3eed608f-4d31-4708-ac7c-9a29fe254a1b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ce5b9b-8f77c865d-d2qkn"
Sep 14 03:10:44.020539 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:44.020515    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wghk2\" (UniqueName: \"kubernetes.io/projected/3eed608f-4d31-4708-ac7c-9a29fe254a1b-kube-api-access-wghk2\") pod \"payload-pre-processing-e2e-shared-a-w2-ce5b9b-8f77c865d-d2qkn\" (UID: \"3eed608f-4d31-4708-ac7c-9a29fe254a1b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ce5b9b-8f77c865d-d2qkn"
Sep 14 03:10:44.132871 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:44.132841    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ce5b9b-8f77c865d-d2qkn"
Sep 14 03:10:44.426437 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:44.426395    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ce5b9b-8f77c865d-d2qkn"]
Sep 14 03:10:44.428433 ip-10-0-141-4 kubenswrapper[2564]: W0914 03:10:44.428390    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eed608f_4d31_4708_ac7c_9a29fe254a1b.slice/crio-49eb5766c4e12b0dcd884a83f6f7261f1a9c6516765aa42fb450c741264edbcf WatchSource:0}: Error finding container 49eb5766c4e12b0dcd884a83f6f7261f1a9c6516765aa42fb450c741264edbcf: Status 404 returned error can't find the container with id 49eb5766c4e12b0dcd884a83f6f7261f1a9c6516765aa42fb450c741264edbcf
Sep 14 03:10:44.642932 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:44.642898    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ce5b9b-8f77c865d-d2qkn" event={"ID":"3eed608f-4d31-4708-ac7c-9a29fe254a1b","Type":"ContainerStarted","Data":"95b4f0a0090ddf7f23472f5e6c8139c329d246acaba401dd67d59f66ac91a806"}
Sep 14 03:10:44.643124 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:44.642948    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ce5b9b-8f77c865d-d2qkn" event={"ID":"3eed608f-4d31-4708-ac7c-9a29fe254a1b","Type":"ContainerStarted","Data":"49eb5766c4e12b0dcd884a83f6f7261f1a9c6516765aa42fb450c741264edbcf"}
Sep 14 03:10:44.643124 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:44.642978    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ce5b9b-8f77c865d-d2qkn"
Sep 14 03:10:44.661975 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:44.661918    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ce5b9b-8f77c865d-d2qkn" podStartSLOduration=1.66189422 podStartE2EDuration="1.66189422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:10:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:10:44.658586471 +0000 UTC m=+1090.039670671" watchObservedRunningTime="2026-09-14 03:10:44.66189422 +0000 UTC m=+1090.042978423"
Sep 14 03:10:50.647213 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:50.647172    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:10:55.041743 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:55.041711    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-b-7a5ebdfc-kserve-77998f997d-dl52h"]
Sep 14 03:10:55.045503 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:55.045487    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-b-7a5ebdfc-kserve-77998f997d-dl52h"
Sep 14 03:10:55.048315 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:55.048293    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-9c9f1e\"/\"e2e-conflict-b-7a5ebdfc-kserve-self-signed-certs\""
Sep 14 03:10:55.056383 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:55.056361    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-b-7a5ebdfc-kserve-77998f997d-dl52h"]
Sep 14 03:10:55.132977 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:55.132944    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f-tls-certs\") pod \"e2e-conflict-b-7a5ebdfc-kserve-77998f997d-dl52h\" (UID: \"0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f\") " pod="e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-b-7a5ebdfc-kserve-77998f997d-dl52h"
Sep 14 03:10:55.133130 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:55.133024    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f-dshm\") pod \"e2e-conflict-b-7a5ebdfc-kserve-77998f997d-dl52h\" (UID: \"0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f\") " pod="e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-b-7a5ebdfc-kserve-77998f997d-dl52h"
Sep 14 03:10:55.133130 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:55.133042    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f-home\") pod \"e2e-conflict-b-7a5ebdfc-kserve-77998f997d-dl52h\" (UID: \"0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f\") " pod="e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-b-7a5ebdfc-kserve-77998f997d-dl52h"
Sep 14 03:10:55.133130 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:55.133071    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f-model-cache\") pod \"e2e-conflict-b-7a5ebdfc-kserve-77998f997d-dl52h\" (UID: \"0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f\") " pod="e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-b-7a5ebdfc-kserve-77998f997d-dl52h"
Sep 14 03:10:55.133130 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:55.133093    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f-tmp-dir\") pod \"e2e-conflict-b-7a5ebdfc-kserve-77998f997d-dl52h\" (UID: \"0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f\") " pod="e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-b-7a5ebdfc-kserve-77998f997d-dl52h"
Sep 14 03:10:55.133283 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:55.133165    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7ftm\" (UniqueName: \"kubernetes.io/projected/0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f-kube-api-access-m7ftm\") pod \"e2e-conflict-b-7a5ebdfc-kserve-77998f997d-dl52h\" (UID: \"0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f\") " pod="e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-b-7a5ebdfc-kserve-77998f997d-dl52h"
Sep 14 03:10:55.234012 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:55.233979    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f-model-cache\") pod \"e2e-conflict-b-7a5ebdfc-kserve-77998f997d-dl52h\" (UID: \"0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f\") " pod="e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-b-7a5ebdfc-kserve-77998f997d-dl52h"
Sep 14 03:10:55.234161 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:55.234019    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f-tmp-dir\") pod \"e2e-conflict-b-7a5ebdfc-kserve-77998f997d-dl52h\" (UID: \"0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f\") " pod="e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-b-7a5ebdfc-kserve-77998f997d-dl52h"
Sep 14 03:10:55.234161 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:55.234065    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7ftm\" (UniqueName: \"kubernetes.io/projected/0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f-kube-api-access-m7ftm\") pod \"e2e-conflict-b-7a5ebdfc-kserve-77998f997d-dl52h\" (UID: \"0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f\") " pod="e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-b-7a5ebdfc-kserve-77998f997d-dl52h"
Sep 14 03:10:55.234161 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:55.234100    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f-tls-certs\") pod \"e2e-conflict-b-7a5ebdfc-kserve-77998f997d-dl52h\" (UID: \"0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f\") " pod="e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-b-7a5ebdfc-kserve-77998f997d-dl52h"
Sep 14 03:10:55.234295 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:55.234250    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f-dshm\") pod \"e2e-conflict-b-7a5ebdfc-kserve-77998f997d-dl52h\" (UID: \"0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f\") " pod="e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-b-7a5ebdfc-kserve-77998f997d-dl52h"
Sep 14 03:10:55.234346 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:55.234293    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f-home\") pod \"e2e-conflict-b-7a5ebdfc-kserve-77998f997d-dl52h\" (UID: \"0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f\") " pod="e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-b-7a5ebdfc-kserve-77998f997d-dl52h"
Sep 14 03:10:55.234465 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:55.234442    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f-model-cache\") pod \"e2e-conflict-b-7a5ebdfc-kserve-77998f997d-dl52h\" (UID: \"0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f\") " pod="e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-b-7a5ebdfc-kserve-77998f997d-dl52h"
Sep 14 03:10:55.234539 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:55.234489    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f-tmp-dir\") pod \"e2e-conflict-b-7a5ebdfc-kserve-77998f997d-dl52h\" (UID: \"0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f\") " pod="e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-b-7a5ebdfc-kserve-77998f997d-dl52h"
Sep 14 03:10:55.234602 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:55.234576    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f-home\") pod \"e2e-conflict-b-7a5ebdfc-kserve-77998f997d-dl52h\" (UID: \"0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f\") " pod="e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-b-7a5ebdfc-kserve-77998f997d-dl52h"
Sep 14 03:10:55.236364 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:55.236344    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f-dshm\") pod \"e2e-conflict-b-7a5ebdfc-kserve-77998f997d-dl52h\" (UID: \"0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f\") " pod="e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-b-7a5ebdfc-kserve-77998f997d-dl52h"
Sep 14 03:10:55.236558 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:55.236540    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f-tls-certs\") pod \"e2e-conflict-b-7a5ebdfc-kserve-77998f997d-dl52h\" (UID: \"0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f\") " pod="e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-b-7a5ebdfc-kserve-77998f997d-dl52h"
Sep 14 03:10:55.242727 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:55.242709    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7ftm\" (UniqueName: \"kubernetes.io/projected/0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f-kube-api-access-m7ftm\") pod \"e2e-conflict-b-7a5ebdfc-kserve-77998f997d-dl52h\" (UID: \"0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f\") " pod="e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-b-7a5ebdfc-kserve-77998f997d-dl52h"
Sep 14 03:10:55.354183 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:55.354102    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-b-7a5ebdfc-kserve-77998f997d-dl52h"
Sep 14 03:10:55.452630 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:55.452600    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:10:55.627477 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:55.627445    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-b-7a5ebdfc-kserve-77998f997d-dl52h"]
Sep 14 03:10:55.628918 ip-10-0-141-4 kubenswrapper[2564]: W0914 03:10:55.628894    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0c6142b7_9d3e_4b1c_97a8_ed957fea4c1f.slice/crio-f9253aa5b9af7280e99d058ba095248a13325cf0ec9e2ee05d05c61e292f2ab8 WatchSource:0}: Error finding container f9253aa5b9af7280e99d058ba095248a13325cf0ec9e2ee05d05c61e292f2ab8: Status 404 returned error can't find the container with id f9253aa5b9af7280e99d058ba095248a13325cf0ec9e2ee05d05c61e292f2ab8
Sep 14 03:10:55.651669 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:55.651642    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ce5b9b-8f77c865d-d2qkn"
Sep 14 03:10:55.701818 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:55.701786    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-b-7a5ebdfc-kserve-77998f997d-dl52h" event={"ID":"0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f","Type":"ContainerStarted","Data":"3591d13aaffff55dad566c7833116fe88c0267af3c9c4bab778ce8eb8f2d4c44"}
Sep 14 03:10:55.701939 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:55.701831    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-b-7a5ebdfc-kserve-77998f997d-dl52h" event={"ID":"0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f","Type":"ContainerStarted","Data":"f9253aa5b9af7280e99d058ba095248a13325cf0ec9e2ee05d05c61e292f2ab8"}
Sep 14 03:10:55.720957 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:55.720898    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-b-7a5ebdfc-kserve-77998f997d-dl52h" podStartSLOduration=0.720875968 podStartE2EDuration="720.875968ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:10:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:10:55.719620683 +0000 UTC m=+1101.100704881" watchObservedRunningTime="2026-09-14 03:10:55.720875968 +0000 UTC m=+1101.101960168"
Sep 14 03:10:58.740892 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:10:58.740853    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:11:04.238657 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:04.238625    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:11:05.354179 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:05.354153    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-b-7a5ebdfc-kserve-77998f997d-dl52h"
Sep 14 03:11:05.354762 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:05.354195    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-b-7a5ebdfc-kserve-77998f997d-dl52h"
Sep 14 03:11:05.368630 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:05.368607    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-b-7a5ebdfc-kserve-77998f997d-dl52h"
Sep 14 03:11:05.434122 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:05.434079    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:11:05.766375 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:05.766294    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-b-7a5ebdfc-kserve-77998f997d-dl52h"
Sep 14 03:11:06.094546 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:06.094505    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:11:06.838128 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:06.838096    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:11:07.525682 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:07.525647    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:11:08.338101 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:08.338069    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:11:09.037993 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:09.037959    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:11:09.979274 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:09.979213    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:11:10.295130 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:10.295094    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:11:10.697461 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:10.697345    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:11:11.153205 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:11.153168    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:11:12.600550 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:12.600512    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:11:13.562917 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:13.562880    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:11:14.342432 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:14.342378    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:11:14.434729 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:14.434684    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:11:15.375525 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:15.375495    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:11:16.241131 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:16.241094    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:11:16.848566 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:16.848534    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-f16030/noten-model-f16030-kserve-86d665dbc-snqc4"]
Sep 14 03:11:16.852292 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:16.852272    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-f16030/noten-model-f16030-kserve-86d665dbc-snqc4"
Sep 14 03:11:16.856207 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:16.856186    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-f16030\"/\"openshift-service-ca.crt\""
Sep 14 03:11:16.856333 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:16.856186    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-f16030\"/\"default-dockercfg-s7pfx\""
Sep 14 03:11:16.856333 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:16.856186    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-f16030\"/\"kube-root-ca.crt\""
Sep 14 03:11:16.856333 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:16.856186    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-f16030\"/\"noten-model-f16030-kserve-self-signed-certs\""
Sep 14 03:11:16.863815 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:16.863791    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-f16030/noten-model-f16030-kserve-86d665dbc-snqc4"]
Sep 14 03:11:16.952749 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:16.952716    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f345be18-091e-48e6-bbf4-d4581301b800-dshm\") pod \"noten-model-f16030-kserve-86d665dbc-snqc4\" (UID: \"f345be18-091e-48e6-bbf4-d4581301b800\") " pod="ai-tenant-e2e-noten-f16030/noten-model-f16030-kserve-86d665dbc-snqc4"
Sep 14 03:11:16.952916 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:16.952797    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f345be18-091e-48e6-bbf4-d4581301b800-tmp-dir\") pod \"noten-model-f16030-kserve-86d665dbc-snqc4\" (UID: \"f345be18-091e-48e6-bbf4-d4581301b800\") " pod="ai-tenant-e2e-noten-f16030/noten-model-f16030-kserve-86d665dbc-snqc4"
Sep 14 03:11:16.952916 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:16.952834    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f345be18-091e-48e6-bbf4-d4581301b800-tls-certs\") pod \"noten-model-f16030-kserve-86d665dbc-snqc4\" (UID: \"f345be18-091e-48e6-bbf4-d4581301b800\") " pod="ai-tenant-e2e-noten-f16030/noten-model-f16030-kserve-86d665dbc-snqc4"
Sep 14 03:11:16.952916 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:16.952871    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8xl2\" (UniqueName: \"kubernetes.io/projected/f345be18-091e-48e6-bbf4-d4581301b800-kube-api-access-q8xl2\") pod \"noten-model-f16030-kserve-86d665dbc-snqc4\" (UID: \"f345be18-091e-48e6-bbf4-d4581301b800\") " pod="ai-tenant-e2e-noten-f16030/noten-model-f16030-kserve-86d665dbc-snqc4"
Sep 14 03:11:16.953031 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:16.952939    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f345be18-091e-48e6-bbf4-d4581301b800-model-cache\") pod \"noten-model-f16030-kserve-86d665dbc-snqc4\" (UID: \"f345be18-091e-48e6-bbf4-d4581301b800\") " pod="ai-tenant-e2e-noten-f16030/noten-model-f16030-kserve-86d665dbc-snqc4"
Sep 14 03:11:16.953031 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:16.952962    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f345be18-091e-48e6-bbf4-d4581301b800-home\") pod \"noten-model-f16030-kserve-86d665dbc-snqc4\" (UID: \"f345be18-091e-48e6-bbf4-d4581301b800\") " pod="ai-tenant-e2e-noten-f16030/noten-model-f16030-kserve-86d665dbc-snqc4"
Sep 14 03:11:17.053671 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:17.053641    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f345be18-091e-48e6-bbf4-d4581301b800-tmp-dir\") pod \"noten-model-f16030-kserve-86d665dbc-snqc4\" (UID: \"f345be18-091e-48e6-bbf4-d4581301b800\") " pod="ai-tenant-e2e-noten-f16030/noten-model-f16030-kserve-86d665dbc-snqc4"
Sep 14 03:11:17.053850 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:17.053684    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f345be18-091e-48e6-bbf4-d4581301b800-tls-certs\") pod \"noten-model-f16030-kserve-86d665dbc-snqc4\" (UID: \"f345be18-091e-48e6-bbf4-d4581301b800\") " pod="ai-tenant-e2e-noten-f16030/noten-model-f16030-kserve-86d665dbc-snqc4"
Sep 14 03:11:17.053850 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:17.053714    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8xl2\" (UniqueName: \"kubernetes.io/projected/f345be18-091e-48e6-bbf4-d4581301b800-kube-api-access-q8xl2\") pod \"noten-model-f16030-kserve-86d665dbc-snqc4\" (UID: \"f345be18-091e-48e6-bbf4-d4581301b800\") " pod="ai-tenant-e2e-noten-f16030/noten-model-f16030-kserve-86d665dbc-snqc4"
Sep 14 03:11:17.053850 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:17.053769    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f345be18-091e-48e6-bbf4-d4581301b800-model-cache\") pod \"noten-model-f16030-kserve-86d665dbc-snqc4\" (UID: \"f345be18-091e-48e6-bbf4-d4581301b800\") " pod="ai-tenant-e2e-noten-f16030/noten-model-f16030-kserve-86d665dbc-snqc4"
Sep 14 03:11:17.053850 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:17.053804    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f345be18-091e-48e6-bbf4-d4581301b800-home\") pod \"noten-model-f16030-kserve-86d665dbc-snqc4\" (UID: \"f345be18-091e-48e6-bbf4-d4581301b800\") " pod="ai-tenant-e2e-noten-f16030/noten-model-f16030-kserve-86d665dbc-snqc4"
Sep 14 03:11:17.053850 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:17.053834    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f345be18-091e-48e6-bbf4-d4581301b800-dshm\") pod \"noten-model-f16030-kserve-86d665dbc-snqc4\" (UID: \"f345be18-091e-48e6-bbf4-d4581301b800\") " pod="ai-tenant-e2e-noten-f16030/noten-model-f16030-kserve-86d665dbc-snqc4"
Sep 14 03:11:17.054160 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:17.054132    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f345be18-091e-48e6-bbf4-d4581301b800-tmp-dir\") pod \"noten-model-f16030-kserve-86d665dbc-snqc4\" (UID: \"f345be18-091e-48e6-bbf4-d4581301b800\") " pod="ai-tenant-e2e-noten-f16030/noten-model-f16030-kserve-86d665dbc-snqc4"
Sep 14 03:11:17.054232 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:17.054157    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f345be18-091e-48e6-bbf4-d4581301b800-model-cache\") pod \"noten-model-f16030-kserve-86d665dbc-snqc4\" (UID: \"f345be18-091e-48e6-bbf4-d4581301b800\") " pod="ai-tenant-e2e-noten-f16030/noten-model-f16030-kserve-86d665dbc-snqc4"
Sep 14 03:11:17.054232 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:17.054219    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f345be18-091e-48e6-bbf4-d4581301b800-home\") pod \"noten-model-f16030-kserve-86d665dbc-snqc4\" (UID: \"f345be18-091e-48e6-bbf4-d4581301b800\") " pod="ai-tenant-e2e-noten-f16030/noten-model-f16030-kserve-86d665dbc-snqc4"
Sep 14 03:11:17.056120 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:17.056097    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f345be18-091e-48e6-bbf4-d4581301b800-dshm\") pod \"noten-model-f16030-kserve-86d665dbc-snqc4\" (UID: \"f345be18-091e-48e6-bbf4-d4581301b800\") " pod="ai-tenant-e2e-noten-f16030/noten-model-f16030-kserve-86d665dbc-snqc4"
Sep 14 03:11:17.056279 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:17.056263    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f345be18-091e-48e6-bbf4-d4581301b800-tls-certs\") pod \"noten-model-f16030-kserve-86d665dbc-snqc4\" (UID: \"f345be18-091e-48e6-bbf4-d4581301b800\") " pod="ai-tenant-e2e-noten-f16030/noten-model-f16030-kserve-86d665dbc-snqc4"
Sep 14 03:11:17.067999 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:17.067967    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8xl2\" (UniqueName: \"kubernetes.io/projected/f345be18-091e-48e6-bbf4-d4581301b800-kube-api-access-q8xl2\") pod \"noten-model-f16030-kserve-86d665dbc-snqc4\" (UID: \"f345be18-091e-48e6-bbf4-d4581301b800\") " pod="ai-tenant-e2e-noten-f16030/noten-model-f16030-kserve-86d665dbc-snqc4"
Sep 14 03:11:17.145098 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:17.144999    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:11:17.162827 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:17.162787    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-f16030/noten-model-f16030-kserve-86d665dbc-snqc4"
Sep 14 03:11:17.264968 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:17.264937    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:11:17.443024 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:17.442990    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-f16030/noten-model-f16030-kserve-86d665dbc-snqc4"]
Sep 14 03:11:17.443993 ip-10-0-141-4 kubenswrapper[2564]: W0914 03:11:17.443962    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf345be18_091e_48e6_bbf4_d4581301b800.slice/crio-54340071ca6415e0d6768b09287a90ac1e82d83dd2880471e0cd2dcd7152f00d WatchSource:0}: Error finding container 54340071ca6415e0d6768b09287a90ac1e82d83dd2880471e0cd2dcd7152f00d: Status 404 returned error can't find the container with id 54340071ca6415e0d6768b09287a90ac1e82d83dd2880471e0cd2dcd7152f00d
Sep 14 03:11:17.823678 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:17.823641    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-f16030/noten-model-f16030-kserve-86d665dbc-snqc4" event={"ID":"f345be18-091e-48e6-bbf4-d4581301b800","Type":"ContainerStarted","Data":"78f2cf32c8d1bf3d3b3d3535cd52fd8781ee19bae589c1760772a65e0295df71"}
Sep 14 03:11:17.823856 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:17.823688    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-f16030/noten-model-f16030-kserve-86d665dbc-snqc4" event={"ID":"f345be18-091e-48e6-bbf4-d4581301b800","Type":"ContainerStarted","Data":"54340071ca6415e0d6768b09287a90ac1e82d83dd2880471e0cd2dcd7152f00d"}
Sep 14 03:11:17.844008 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:17.843955    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-f16030/noten-model-f16030-kserve-86d665dbc-snqc4" podStartSLOduration=1.843939864 podStartE2EDuration="1.843939864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:11:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:11:17.842314151 +0000 UTC m=+1123.223398366" watchObservedRunningTime="2026-09-14 03:11:17.843939864 +0000 UTC m=+1123.225024062"
Sep 14 03:11:18.187605 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:18.187514    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:11:18.286818 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:18.286782    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:11:18.385628 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:18.385596    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:11:18.683819 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:18.683784    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:11:18.805183 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:18.805149    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:11:19.193990 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:19.193928    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:11:20.012473 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:20.012429    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:11:20.676865 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:20.676829    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:11:21.515692 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:21.515653    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:11:21.979345 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:21.979310    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:11:22.406967 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:22.406935    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:11:23.050645 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:23.050610    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f002a1-7f464f54d5-xntnr"]
Sep 14 03:11:23.051104 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:23.050860    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f002a1-7f464f54d5-xntnr" podUID="59691064-4c10-4ec6-99f3-5c0c8b30209e" containerName="maas-api" containerID="cri-o://31e9db45970b4a5aace94536f7ab411348df20af6ee9253bbdce99fe7c81dddd" gracePeriod=30
Sep 14 03:11:23.088159 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:23.088125    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:11:23.165903 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:23.165867    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:11:23.267781 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:23.267747    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:11:23.310318 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:23.310259    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f002a1-7f464f54d5-xntnr"
Sep 14 03:11:23.371173 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:23.371102    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:11:23.422654 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:23.422616    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59691064-4c10-4ec6-99f3-5c0c8b30209e-metrics-certs\") pod \"59691064-4c10-4ec6-99f3-5c0c8b30209e\" (UID: \"59691064-4c10-4ec6-99f3-5c0c8b30209e\") "
Sep 14 03:11:23.422837 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:23.422727    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59691064-4c10-4ec6-99f3-5c0c8b30209e-openshift-service-ca\") pod \"59691064-4c10-4ec6-99f3-5c0c8b30209e\" (UID: \"59691064-4c10-4ec6-99f3-5c0c8b30209e\") "
Sep 14 03:11:23.422906 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:23.422847    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s74gl\" (UniqueName: \"kubernetes.io/projected/59691064-4c10-4ec6-99f3-5c0c8b30209e-kube-api-access-s74gl\") pod \"59691064-4c10-4ec6-99f3-5c0c8b30209e\" (UID: \"59691064-4c10-4ec6-99f3-5c0c8b30209e\") "
Sep 14 03:11:23.422906 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:23.422880    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59691064-4c10-4ec6-99f3-5c0c8b30209e-maas-api-tls\") pod \"59691064-4c10-4ec6-99f3-5c0c8b30209e\" (UID: \"59691064-4c10-4ec6-99f3-5c0c8b30209e\") "
Sep 14 03:11:23.423146 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:23.423114    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59691064-4c10-4ec6-99f3-5c0c8b30209e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "59691064-4c10-4ec6-99f3-5c0c8b30209e" (UID: "59691064-4c10-4ec6-99f3-5c0c8b30209e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 03:11:23.425165 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:23.425139    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59691064-4c10-4ec6-99f3-5c0c8b30209e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "59691064-4c10-4ec6-99f3-5c0c8b30209e" (UID: "59691064-4c10-4ec6-99f3-5c0c8b30209e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 03:11:23.425387 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:23.425367    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59691064-4c10-4ec6-99f3-5c0c8b30209e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "59691064-4c10-4ec6-99f3-5c0c8b30209e" (UID: "59691064-4c10-4ec6-99f3-5c0c8b30209e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 03:11:23.426164 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:23.426144    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59691064-4c10-4ec6-99f3-5c0c8b30209e-kube-api-access-s74gl" (OuterVolumeSpecName: "kube-api-access-s74gl") pod "59691064-4c10-4ec6-99f3-5c0c8b30209e" (UID: "59691064-4c10-4ec6-99f3-5c0c8b30209e"). InnerVolumeSpecName "kube-api-access-s74gl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:11:23.524148 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:23.524108    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s74gl\" (UniqueName: \"kubernetes.io/projected/59691064-4c10-4ec6-99f3-5c0c8b30209e-kube-api-access-s74gl\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:11:23.524148 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:23.524141    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59691064-4c10-4ec6-99f3-5c0c8b30209e-maas-api-tls\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:11:23.524148 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:23.524151    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59691064-4c10-4ec6-99f3-5c0c8b30209e-metrics-certs\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:11:23.524391 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:23.524161    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59691064-4c10-4ec6-99f3-5c0c8b30209e-openshift-service-ca\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:11:23.735545 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:23.735454    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:11:23.857823 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:23.857788    2564 generic.go:356] "Generic (PLEG): container finished" podID="59691064-4c10-4ec6-99f3-5c0c8b30209e" containerID="31e9db45970b4a5aace94536f7ab411348df20af6ee9253bbdce99fe7c81dddd" exitCode=0
Sep 14 03:11:23.858024 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:23.857838    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f002a1-7f464f54d5-xntnr" event={"ID":"59691064-4c10-4ec6-99f3-5c0c8b30209e","Type":"ContainerDied","Data":"31e9db45970b4a5aace94536f7ab411348df20af6ee9253bbdce99fe7c81dddd"}
Sep 14 03:11:23.858024 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:23.857847    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f002a1-7f464f54d5-xntnr"
Sep 14 03:11:23.858024 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:23.857924    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f002a1-7f464f54d5-xntnr" event={"ID":"59691064-4c10-4ec6-99f3-5c0c8b30209e","Type":"ContainerDied","Data":"7c5c3d0f916cd9173509029c7189d17c000f2405ee0db2a0dbb196efef8647c9"}
Sep 14 03:11:23.858024 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:23.857956    2564 scope.go:117] "RemoveContainer" containerID="31e9db45970b4a5aace94536f7ab411348df20af6ee9253bbdce99fe7c81dddd"
Sep 14 03:11:23.868184 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:23.868167    2564 scope.go:117] "RemoveContainer" containerID="31e9db45970b4a5aace94536f7ab411348df20af6ee9253bbdce99fe7c81dddd"
Sep 14 03:11:23.868488 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:11:23.868468    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31e9db45970b4a5aace94536f7ab411348df20af6ee9253bbdce99fe7c81dddd\": container with ID starting with 31e9db45970b4a5aace94536f7ab411348df20af6ee9253bbdce99fe7c81dddd not found: ID does not exist" containerID="31e9db45970b4a5aace94536f7ab411348df20af6ee9253bbdce99fe7c81dddd"
Sep 14 03:11:23.868590 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:23.868493    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31e9db45970b4a5aace94536f7ab411348df20af6ee9253bbdce99fe7c81dddd"} err="failed to get container status \"31e9db45970b4a5aace94536f7ab411348df20af6ee9253bbdce99fe7c81dddd\": rpc error: code = NotFound desc = could not find container \"31e9db45970b4a5aace94536f7ab411348df20af6ee9253bbdce99fe7c81dddd\": container with ID starting with 31e9db45970b4a5aace94536f7ab411348df20af6ee9253bbdce99fe7c81dddd not found: ID does not exist"
Sep 14 03:11:23.883895 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:23.883874    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f002a1-7f464f54d5-xntnr"]
Sep 14 03:11:23.886587 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:23.886568    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f002a1-7f464f54d5-xntnr"]
Sep 14 03:11:23.899444 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:23.899422    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:11:24.658726 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:24.658691    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:11:25.098599 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:25.098571    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59691064-4c10-4ec6-99f3-5c0c8b30209e" path="/var/lib/kubelet/pods/59691064-4c10-4ec6-99f3-5c0c8b30209e/volumes"
Sep 14 03:11:25.457249 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:25.457161    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:11:26.195003 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:26.194965    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:11:26.853493 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:26.853457    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:11:27.163867 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:27.163780    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-f16030/noten-model-f16030-kserve-86d665dbc-snqc4"
Sep 14 03:11:27.164028 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:27.163929    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-f16030/noten-model-f16030-kserve-86d665dbc-snqc4"
Sep 14 03:11:27.176773 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:27.176740    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-f16030/noten-model-f16030-kserve-86d665dbc-snqc4"
Sep 14 03:11:27.707116 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:27.707085    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:11:27.895791 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:27.895754    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-f16030/noten-model-f16030-kserve-86d665dbc-snqc4"
Sep 14 03:11:28.758202 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:28.758166    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:11:28.927552 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:28.927520    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:11:29.050117 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:29.050073    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:11:29.638960 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:29.638927    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7671-e60d47"]
Sep 14 03:11:29.639423 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:29.639393    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59691064-4c10-4ec6-99f3-5c0c8b30209e" containerName="maas-api"
Sep 14 03:11:29.639423 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:29.639420    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="59691064-4c10-4ec6-99f3-5c0c8b30209e" containerName="maas-api"
Sep 14 03:11:29.639536 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:29.639527    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="59691064-4c10-4ec6-99f3-5c0c8b30209e" containerName="maas-api"
Sep 14 03:11:29.644183 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:29.644167    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7671-e60d47"
Sep 14 03:11:29.647272 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:29.647232    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-2qhmh\""
Sep 14 03:11:29.649056 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:29.649032    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7671-e60d47"]
Sep 14 03:11:29.741498 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:29.741459    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:11:29.787175 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:29.787144    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czswp\" (UniqueName: \"kubernetes.io/projected/45b8e89b-b746-4ff9-84b4-f11c8a7c5aeb-kube-api-access-czswp\") pod \"test-curl-7671-e60d47\" (UID: \"45b8e89b-b746-4ff9-84b4-f11c8a7c5aeb\") " pod="opendatahub/test-curl-7671-e60d47"
Sep 14 03:11:29.887963 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:29.887927    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czswp\" (UniqueName: \"kubernetes.io/projected/45b8e89b-b746-4ff9-84b4-f11c8a7c5aeb-kube-api-access-czswp\") pod \"test-curl-7671-e60d47\" (UID: \"45b8e89b-b746-4ff9-84b4-f11c8a7c5aeb\") " pod="opendatahub/test-curl-7671-e60d47"
Sep 14 03:11:29.899373 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:29.899298    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czswp\" (UniqueName: \"kubernetes.io/projected/45b8e89b-b746-4ff9-84b4-f11c8a7c5aeb-kube-api-access-czswp\") pod \"test-curl-7671-e60d47\" (UID: \"45b8e89b-b746-4ff9-84b4-f11c8a7c5aeb\") " pod="opendatahub/test-curl-7671-e60d47"
Sep 14 03:11:29.955709 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:29.955655    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7671-e60d47"
Sep 14 03:11:30.251297 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:30.251218    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7671-e60d47"]
Sep 14 03:11:30.254533 ip-10-0-141-4 kubenswrapper[2564]: W0914 03:11:30.254506    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod45b8e89b_b746_4ff9_84b4_f11c8a7c5aeb.slice/crio-74976527d5ad5a4609da722e382b3e41a9f6b44b71032b97965a91026e9c9d5c WatchSource:0}: Error finding container 74976527d5ad5a4609da722e382b3e41a9f6b44b71032b97965a91026e9c9d5c: Status 404 returned error can't find the container with id 74976527d5ad5a4609da722e382b3e41a9f6b44b71032b97965a91026e9c9d5c
Sep 14 03:11:30.656202 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:30.656155    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:11:30.897378 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:30.897342    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7671-e60d47" event={"ID":"45b8e89b-b746-4ff9-84b4-f11c8a7c5aeb","Type":"ContainerStarted","Data":"240346ac0205f5c45ef1c1403a14b6cb0b77e63a793e53b4fed94b8a91f115e1"}
Sep 14 03:11:30.897378 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:30.897381    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7671-e60d47" event={"ID":"45b8e89b-b746-4ff9-84b4-f11c8a7c5aeb","Type":"ContainerStarted","Data":"74976527d5ad5a4609da722e382b3e41a9f6b44b71032b97965a91026e9c9d5c"}
Sep 14 03:11:30.920133 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:30.920035    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7671-e60d47" podStartSLOduration=1.920017951 podStartE2EDuration="1.920017951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:11:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:11:30.917783751 +0000 UTC m=+1136.298867951" watchObservedRunningTime="2026-09-14 03:11:30.920017951 +0000 UTC m=+1136.301102191"
Sep 14 03:11:31.472646 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:31.472607    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:11:31.621234 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:31.621197    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7671-e60d47"]
Sep 14 03:11:31.625279 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:31.625246    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7671-e60d47"]
Sep 14 03:11:31.761262 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:31.761171    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7671-9eebf3"]
Sep 14 03:11:31.761892 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:31.761873    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45b8e89b-b746-4ff9-84b4-f11c8a7c5aeb" containerName="test-curl-7671-e60d47"
Sep 14 03:11:31.761959 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:31.761896    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="45b8e89b-b746-4ff9-84b4-f11c8a7c5aeb" containerName="test-curl-7671-e60d47"
Sep 14 03:11:31.762027 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:31.762015    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="45b8e89b-b746-4ff9-84b4-f11c8a7c5aeb" containerName="test-curl-7671-e60d47"
Sep 14 03:11:31.765732 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:31.765716    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7671-9eebf3"
Sep 14 03:11:31.769382 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:31.769356    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7671-9eebf3"]
Sep 14 03:11:31.902697 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:11:31.902668    2564 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7671-e60d47" container="test-curl-7671-e60d47"
Sep 14 03:11:31.909170 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:31.909147    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wggm\" (UniqueName: \"kubernetes.io/projected/8e1c1eeb-792d-462e-90ce-a81133803da5-kube-api-access-8wggm\") pod \"test-curl-7671-9eebf3\" (UID: \"8e1c1eeb-792d-462e-90ce-a81133803da5\") " pod="opendatahub/test-curl-7671-9eebf3"
Sep 14 03:11:32.010246 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:32.010210    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wggm\" (UniqueName: \"kubernetes.io/projected/8e1c1eeb-792d-462e-90ce-a81133803da5-kube-api-access-8wggm\") pod \"test-curl-7671-9eebf3\" (UID: \"8e1c1eeb-792d-462e-90ce-a81133803da5\") " pod="opendatahub/test-curl-7671-9eebf3"
Sep 14 03:11:32.019994 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:32.019923    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wggm\" (UniqueName: \"kubernetes.io/projected/8e1c1eeb-792d-462e-90ce-a81133803da5-kube-api-access-8wggm\") pod \"test-curl-7671-9eebf3\" (UID: \"8e1c1eeb-792d-462e-90ce-a81133803da5\") " pod="opendatahub/test-curl-7671-9eebf3"
Sep 14 03:11:32.076304 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:32.076269    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7671-9eebf3"
Sep 14 03:11:32.349935 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:32.349903    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7671-9eebf3"]
Sep 14 03:11:32.352128 ip-10-0-141-4 kubenswrapper[2564]: W0914 03:11:32.352098    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e1c1eeb_792d_462e_90ce_a81133803da5.slice/crio-74c082f6726866641eb6a9de9febdb3d0bba03316e6d5049e1a93206c9604135 WatchSource:0}: Error finding container 74c082f6726866641eb6a9de9febdb3d0bba03316e6d5049e1a93206c9604135: Status 404 returned error can't find the container with id 74c082f6726866641eb6a9de9febdb3d0bba03316e6d5049e1a93206c9604135
Sep 14 03:11:32.470593 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:32.470556    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:11:32.909285 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:32.909246    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7671-9eebf3" event={"ID":"8e1c1eeb-792d-462e-90ce-a81133803da5","Type":"ContainerStarted","Data":"72ac32937c05f06b0253820d11fae222f711840effc6e7ca9af65e541c3bec3f"}
Sep 14 03:11:32.909736 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:32.909297    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7671-9eebf3" event={"ID":"8e1c1eeb-792d-462e-90ce-a81133803da5","Type":"ContainerStarted","Data":"74c082f6726866641eb6a9de9febdb3d0bba03316e6d5049e1a93206c9604135"}
Sep 14 03:11:32.909736 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:32.909397    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7671-e60d47" podUID="45b8e89b-b746-4ff9-84b4-f11c8a7c5aeb" containerName="test-curl-7671-e60d47" containerID="cri-o://240346ac0205f5c45ef1c1403a14b6cb0b77e63a793e53b4fed94b8a91f115e1" gracePeriod=2
Sep 14 03:11:32.913441 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:11:32.913391    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7671-e60d47\" is forbidden: User \"system:node:ip-10-0-141-4.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-4.ec2.internal' and this object" podUID="45b8e89b-b746-4ff9-84b4-f11c8a7c5aeb" pod="opendatahub/test-curl-7671-e60d47"
Sep 14 03:11:32.946178 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:32.946132    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7671-9eebf3" podStartSLOduration=1.946115405 podStartE2EDuration="1.946115405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:11:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:11:32.94587299 +0000 UTC m=+1138.326957189" watchObservedRunningTime="2026-09-14 03:11:32.946115405 +0000 UTC m=+1138.327199603"
Sep 14 03:11:32.948359 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:11:32.948336    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7671-e60d47\" is forbidden: User \"system:node:ip-10-0-141-4.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-4.ec2.internal' and this object" podUID="45b8e89b-b746-4ff9-84b4-f11c8a7c5aeb" pod="opendatahub/test-curl-7671-e60d47"
Sep 14 03:11:33.269658 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:33.269564    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:11:33.652164 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:33.652129    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7671-9eebf3"]
Sep 14 03:11:33.663732 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:33.663693    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7671-9eebf3"]
Sep 14 03:11:33.804650 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:33.804613    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7671-d16c76"]
Sep 14 03:11:33.805101 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:33.805088    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e1c1eeb-792d-462e-90ce-a81133803da5" containerName="test-curl-7671-9eebf3"
Sep 14 03:11:33.805142 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:33.805102    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e1c1eeb-792d-462e-90ce-a81133803da5" containerName="test-curl-7671-9eebf3"
Sep 14 03:11:33.805208 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:33.805199    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e1c1eeb-792d-462e-90ce-a81133803da5" containerName="test-curl-7671-9eebf3"
Sep 14 03:11:33.808396 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:33.808380    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7671-d16c76"
Sep 14 03:11:33.826953 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:33.826924    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7671-d16c76"]
Sep 14 03:11:33.915479 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:11:33.915370    2564 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7671-9eebf3" container="test-curl-7671-9eebf3"
Sep 14 03:11:33.930553 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:33.930525    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwxpr\" (UniqueName: \"kubernetes.io/projected/4d545046-40ad-46a8-ae37-aaf244f091c9-kube-api-access-qwxpr\") pod \"test-curl-7671-d16c76\" (UID: \"4d545046-40ad-46a8-ae37-aaf244f091c9\") " pod="opendatahub/test-curl-7671-d16c76"
Sep 14 03:11:34.031808 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:34.031774    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwxpr\" (UniqueName: \"kubernetes.io/projected/4d545046-40ad-46a8-ae37-aaf244f091c9-kube-api-access-qwxpr\") pod \"test-curl-7671-d16c76\" (UID: \"4d545046-40ad-46a8-ae37-aaf244f091c9\") " pod="opendatahub/test-curl-7671-d16c76"
Sep 14 03:11:34.041805 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:34.041780    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwxpr\" (UniqueName: \"kubernetes.io/projected/4d545046-40ad-46a8-ae37-aaf244f091c9-kube-api-access-qwxpr\") pod \"test-curl-7671-d16c76\" (UID: \"4d545046-40ad-46a8-ae37-aaf244f091c9\") " pod="opendatahub/test-curl-7671-d16c76"
Sep 14 03:11:34.118968 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:34.118938    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7671-d16c76"
Sep 14 03:11:34.390821 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:34.390798    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7671-d16c76"]
Sep 14 03:11:34.392292 ip-10-0-141-4 kubenswrapper[2564]: W0914 03:11:34.392250    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d545046_40ad_46a8_ae37_aaf244f091c9.slice/crio-7c44457613460b0c42e46aeeb77fc4e23ff39c2d679e1126828734a20769620d WatchSource:0}: Error finding container 7c44457613460b0c42e46aeeb77fc4e23ff39c2d679e1126828734a20769620d: Status 404 returned error can't find the container with id 7c44457613460b0c42e46aeeb77fc4e23ff39c2d679e1126828734a20769620d
Sep 14 03:11:34.925243 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:34.925205    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7671-d16c76" event={"ID":"4d545046-40ad-46a8-ae37-aaf244f091c9","Type":"ContainerStarted","Data":"aeb5569299dfcc76686bb95647f85642300bd660dda61cd1a24f45018e91d610"}
Sep 14 03:11:34.925648 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:34.925256    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7671-d16c76" event={"ID":"4d545046-40ad-46a8-ae37-aaf244f091c9","Type":"ContainerStarted","Data":"7c44457613460b0c42e46aeeb77fc4e23ff39c2d679e1126828734a20769620d"}
Sep 14 03:11:34.925648 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:34.925345    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7671-9eebf3" podUID="8e1c1eeb-792d-462e-90ce-a81133803da5" containerName="test-curl-7671-9eebf3" containerID="cri-o://72ac32937c05f06b0253820d11fae222f711840effc6e7ca9af65e541c3bec3f" gracePeriod=2
Sep 14 03:11:34.942594 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:11:34.942561    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7671-9eebf3\" is forbidden: User \"system:node:ip-10-0-141-4.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-4.ec2.internal' and this object" podUID="8e1c1eeb-792d-462e-90ce-a81133803da5" pod="opendatahub/test-curl-7671-9eebf3"
Sep 14 03:11:34.942986 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:34.942948    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7671-d16c76" podStartSLOduration=1.9429330839999999 podStartE2EDuration="1.942933084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:11:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:11:34.940242968 +0000 UTC m=+1140.321327192" watchObservedRunningTime="2026-09-14 03:11:34.942933084 +0000 UTC m=+1140.324017283"
Sep 14 03:11:35.099267 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:11:35.099238    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7671-9eebf3\" is forbidden: User \"system:node:ip-10-0-141-4.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-4.ec2.internal' and this object" podUID="8e1c1eeb-792d-462e-90ce-a81133803da5" pod="opendatahub/test-curl-7671-9eebf3"
Sep 14 03:11:35.105150 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:35.105129    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7671-e60d47"
Sep 14 03:11:35.107605 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:11:35.107580    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7671-e60d47\" is forbidden: User \"system:node:ip-10-0-141-4.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-4.ec2.internal' and this object" podUID="45b8e89b-b746-4ff9-84b4-f11c8a7c5aeb" pod="opendatahub/test-curl-7671-e60d47"
Sep 14 03:11:35.244308 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:35.244274    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-czswp\" (UniqueName: \"kubernetes.io/projected/45b8e89b-b746-4ff9-84b4-f11c8a7c5aeb-kube-api-access-czswp\") pod \"45b8e89b-b746-4ff9-84b4-f11c8a7c5aeb\" (UID: \"45b8e89b-b746-4ff9-84b4-f11c8a7c5aeb\") "
Sep 14 03:11:35.246321 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:35.246291    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45b8e89b-b746-4ff9-84b4-f11c8a7c5aeb-kube-api-access-czswp" (OuterVolumeSpecName: "kube-api-access-czswp") pod "45b8e89b-b746-4ff9-84b4-f11c8a7c5aeb" (UID: "45b8e89b-b746-4ff9-84b4-f11c8a7c5aeb"). InnerVolumeSpecName "kube-api-access-czswp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:11:35.346154 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:35.346123    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czswp\" (UniqueName: \"kubernetes.io/projected/45b8e89b-b746-4ff9-84b4-f11c8a7c5aeb-kube-api-access-czswp\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:11:35.514057 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:35.513974    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7671-d16c76"]
Sep 14 03:11:35.519740 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:35.519712    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7671-d16c76"]
Sep 14 03:11:35.633830 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:35.633796    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7671-818013"]
Sep 14 03:11:35.634274 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:35.634259    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d545046-40ad-46a8-ae37-aaf244f091c9" containerName="test-curl-7671-d16c76"
Sep 14 03:11:35.634326 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:35.634276    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d545046-40ad-46a8-ae37-aaf244f091c9" containerName="test-curl-7671-d16c76"
Sep 14 03:11:35.634363 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:35.634354    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d545046-40ad-46a8-ae37-aaf244f091c9" containerName="test-curl-7671-d16c76"
Sep 14 03:11:35.639122 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:35.639097    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7671-818013"
Sep 14 03:11:35.641664 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:11:35.641633    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7671-e60d47\" is forbidden: User \"system:node:ip-10-0-141-4.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-4.ec2.internal' and this object" podUID="45b8e89b-b746-4ff9-84b4-f11c8a7c5aeb" pod="opendatahub/test-curl-7671-e60d47"
Sep 14 03:11:35.642194 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:35.642171    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7671-818013"]
Sep 14 03:11:35.751293 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:35.751251    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zszdw\" (UniqueName: \"kubernetes.io/projected/3965a26e-c54b-4eb1-9ea3-812b1522c291-kube-api-access-zszdw\") pod \"test-curl-7671-818013\" (UID: \"3965a26e-c54b-4eb1-9ea3-812b1522c291\") " pod="opendatahub/test-curl-7671-818013"
Sep 14 03:11:35.852315 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:35.852282    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zszdw\" (UniqueName: \"kubernetes.io/projected/3965a26e-c54b-4eb1-9ea3-812b1522c291-kube-api-access-zszdw\") pod \"test-curl-7671-818013\" (UID: \"3965a26e-c54b-4eb1-9ea3-812b1522c291\") " pod="opendatahub/test-curl-7671-818013"
Sep 14 03:11:35.861738 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:35.861703    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zszdw\" (UniqueName: \"kubernetes.io/projected/3965a26e-c54b-4eb1-9ea3-812b1522c291-kube-api-access-zszdw\") pod \"test-curl-7671-818013\" (UID: \"3965a26e-c54b-4eb1-9ea3-812b1522c291\") " pod="opendatahub/test-curl-7671-818013"
Sep 14 03:11:35.931418 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:35.931383    2564 generic.go:356] "Generic (PLEG): container finished" podID="45b8e89b-b746-4ff9-84b4-f11c8a7c5aeb" containerID="240346ac0205f5c45ef1c1403a14b6cb0b77e63a793e53b4fed94b8a91f115e1" exitCode=137
Sep 14 03:11:35.931798 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:35.931439    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7671-e60d47"
Sep 14 03:11:35.931798 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:35.931490    2564 scope.go:117] "RemoveContainer" containerID="240346ac0205f5c45ef1c1403a14b6cb0b77e63a793e53b4fed94b8a91f115e1"
Sep 14 03:11:35.931911 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:11:35.931823    2564 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7671-d16c76" container="test-curl-7671-d16c76"
Sep 14 03:11:35.934111 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:11:35.934087    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7671-e60d47\" is forbidden: User \"system:node:ip-10-0-141-4.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-4.ec2.internal' and this object" podUID="45b8e89b-b746-4ff9-84b4-f11c8a7c5aeb" pod="opendatahub/test-curl-7671-e60d47"
Sep 14 03:11:35.941117 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:35.941100    2564 scope.go:117] "RemoveContainer" containerID="240346ac0205f5c45ef1c1403a14b6cb0b77e63a793e53b4fed94b8a91f115e1"
Sep 14 03:11:35.941382 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:11:35.941364    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"240346ac0205f5c45ef1c1403a14b6cb0b77e63a793e53b4fed94b8a91f115e1\": container with ID starting with 240346ac0205f5c45ef1c1403a14b6cb0b77e63a793e53b4fed94b8a91f115e1 not found: ID does not exist" containerID="240346ac0205f5c45ef1c1403a14b6cb0b77e63a793e53b4fed94b8a91f115e1"
Sep 14 03:11:35.941449 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:35.941391    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"240346ac0205f5c45ef1c1403a14b6cb0b77e63a793e53b4fed94b8a91f115e1"} err="failed to get container status \"240346ac0205f5c45ef1c1403a14b6cb0b77e63a793e53b4fed94b8a91f115e1\": rpc error: code = NotFound desc = could not find container \"240346ac0205f5c45ef1c1403a14b6cb0b77e63a793e53b4fed94b8a91f115e1\": container with ID starting with 240346ac0205f5c45ef1c1403a14b6cb0b77e63a793e53b4fed94b8a91f115e1 not found: ID does not exist"
Sep 14 03:11:35.942869 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:11:35.942847    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7671-e60d47\" is forbidden: User \"system:node:ip-10-0-141-4.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-4.ec2.internal' and this object" podUID="45b8e89b-b746-4ff9-84b4-f11c8a7c5aeb" pod="opendatahub/test-curl-7671-e60d47"
Sep 14 03:11:35.949992 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:35.949979    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7671-818013"
Sep 14 03:11:36.224554 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:36.220952    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7671-818013"]
Sep 14 03:11:36.227813 ip-10-0-141-4 kubenswrapper[2564]: W0914 03:11:36.227776    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3965a26e_c54b_4eb1_9ea3_812b1522c291.slice/crio-618c6260455bce74b0147c45783309a5351561d381829598e1dbf40d2c6e9c31 WatchSource:0}: Error finding container 618c6260455bce74b0147c45783309a5351561d381829598e1dbf40d2c6e9c31: Status 404 returned error can't find the container with id 618c6260455bce74b0147c45783309a5351561d381829598e1dbf40d2c6e9c31
Sep 14 03:11:36.938455 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:36.938399    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7671-818013" event={"ID":"3965a26e-c54b-4eb1-9ea3-812b1522c291","Type":"ContainerStarted","Data":"1eae766c3003247612af23b7666f7f91698d4ec2472047967240d4cf746e5a62"}
Sep 14 03:11:36.938847 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:36.938470    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7671-818013" event={"ID":"3965a26e-c54b-4eb1-9ea3-812b1522c291","Type":"ContainerStarted","Data":"618c6260455bce74b0147c45783309a5351561d381829598e1dbf40d2c6e9c31"}
Sep 14 03:11:36.939540 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:36.939516    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7671-d16c76" podUID="4d545046-40ad-46a8-ae37-aaf244f091c9" containerName="test-curl-7671-d16c76" containerID="cri-o://aeb5569299dfcc76686bb95647f85642300bd660dda61cd1a24f45018e91d610" gracePeriod=2
Sep 14 03:11:36.941279 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:11:36.941256    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7671-e60d47\" is forbidden: User \"system:node:ip-10-0-141-4.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-4.ec2.internal' and this object" podUID="45b8e89b-b746-4ff9-84b4-f11c8a7c5aeb" pod="opendatahub/test-curl-7671-e60d47"
Sep 14 03:11:36.960450 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:36.960395    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7671-818013" podStartSLOduration=1.960381235 podStartE2EDuration="1.960381235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:11:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:11:36.958841281 +0000 UTC m=+1142.339925480" watchObservedRunningTime="2026-09-14 03:11:36.960381235 +0000 UTC m=+1142.341465432"
Sep 14 03:11:36.961302 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:11:36.961277    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7671-e60d47\" is forbidden: User \"system:node:ip-10-0-141-4.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-4.ec2.internal' and this object" podUID="45b8e89b-b746-4ff9-84b4-f11c8a7c5aeb" pod="opendatahub/test-curl-7671-e60d47"
Sep 14 03:11:36.963492 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:11:36.963456    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7671-d16c76\" is forbidden: User \"system:node:ip-10-0-141-4.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-4.ec2.internal' and this object" podUID="4d545046-40ad-46a8-ae37-aaf244f091c9" pod="opendatahub/test-curl-7671-d16c76"
Sep 14 03:11:37.094168 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:37.094143    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7671-9eebf3"
Sep 14 03:11:37.096834 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:11:37.096807    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7671-e60d47\" is forbidden: User \"system:node:ip-10-0-141-4.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-4.ec2.internal' and this object" podUID="45b8e89b-b746-4ff9-84b4-f11c8a7c5aeb" pod="opendatahub/test-curl-7671-e60d47"
Sep 14 03:11:37.099008 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:37.098985    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45b8e89b-b746-4ff9-84b4-f11c8a7c5aeb" path="/var/lib/kubelet/pods/45b8e89b-b746-4ff9-84b4-f11c8a7c5aeb/volumes"
Sep 14 03:11:37.099103 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:11:37.099059    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7671-d16c76\" is forbidden: User \"system:node:ip-10-0-141-4.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-4.ec2.internal' and this object" podUID="4d545046-40ad-46a8-ae37-aaf244f091c9" pod="opendatahub/test-curl-7671-d16c76"
Sep 14 03:11:37.164686 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:37.164657    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8wggm\" (UniqueName: \"kubernetes.io/projected/8e1c1eeb-792d-462e-90ce-a81133803da5-kube-api-access-8wggm\") pod \"8e1c1eeb-792d-462e-90ce-a81133803da5\" (UID: \"8e1c1eeb-792d-462e-90ce-a81133803da5\") "
Sep 14 03:11:37.166819 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:37.166785    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e1c1eeb-792d-462e-90ce-a81133803da5-kube-api-access-8wggm" (OuterVolumeSpecName: "kube-api-access-8wggm") pod "8e1c1eeb-792d-462e-90ce-a81133803da5" (UID: "8e1c1eeb-792d-462e-90ce-a81133803da5"). InnerVolumeSpecName "kube-api-access-8wggm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:11:37.265563 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:37.265474    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wggm\" (UniqueName: \"kubernetes.io/projected/8e1c1eeb-792d-462e-90ce-a81133803da5-kube-api-access-8wggm\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:11:37.539114 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:37.539078    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7671-818013"]
Sep 14 03:11:37.544527 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:37.544502    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7671-818013"]
Sep 14 03:11:37.946485 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:37.946387    2564 generic.go:356] "Generic (PLEG): container finished" podID="8e1c1eeb-792d-462e-90ce-a81133803da5" containerID="72ac32937c05f06b0253820d11fae222f711840effc6e7ca9af65e541c3bec3f" exitCode=137
Sep 14 03:11:37.946485 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:37.946449    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7671-9eebf3"
Sep 14 03:11:37.946922 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:37.946450    2564 scope.go:117] "RemoveContainer" containerID="72ac32937c05f06b0253820d11fae222f711840effc6e7ca9af65e541c3bec3f"
Sep 14 03:11:37.946922 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:11:37.946663    2564 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7671-818013" container="test-curl-7671-818013"
Sep 14 03:11:37.956083 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:37.956067    2564 scope.go:117] "RemoveContainer" containerID="72ac32937c05f06b0253820d11fae222f711840effc6e7ca9af65e541c3bec3f"
Sep 14 03:11:37.956368 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:11:37.956349    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72ac32937c05f06b0253820d11fae222f711840effc6e7ca9af65e541c3bec3f\": container with ID starting with 72ac32937c05f06b0253820d11fae222f711840effc6e7ca9af65e541c3bec3f not found: ID does not exist" containerID="72ac32937c05f06b0253820d11fae222f711840effc6e7ca9af65e541c3bec3f"
Sep 14 03:11:37.956422 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:37.956376    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72ac32937c05f06b0253820d11fae222f711840effc6e7ca9af65e541c3bec3f"} err="failed to get container status \"72ac32937c05f06b0253820d11fae222f711840effc6e7ca9af65e541c3bec3f\": rpc error: code = NotFound desc = could not find container \"72ac32937c05f06b0253820d11fae222f711840effc6e7ca9af65e541c3bec3f\": container with ID starting with 72ac32937c05f06b0253820d11fae222f711840effc6e7ca9af65e541c3bec3f not found: ID does not exist"
Sep 14 03:11:38.134876 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:38.134845    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:11:38.185598 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:11:38.185568    2564 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod45b8e89b_b746_4ff9_84b4_f11c8a7c5aeb.slice/crio-74976527d5ad5a4609da722e382b3e41a9f6b44b71032b97965a91026e9c9d5c\": RecentStats: unable to find data in memory cache]"
Sep 14 03:11:38.447842 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:38.447807    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:11:38.722532 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:38.722446    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:11:38.954489 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:38.954462    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7671-818013" podUID="3965a26e-c54b-4eb1-9ea3-812b1522c291" containerName="test-curl-7671-818013" containerID="cri-o://1eae766c3003247612af23b7666f7f91698d4ec2472047967240d4cf746e5a62" gracePeriod=2
Sep 14 03:11:38.957345 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:11:38.957321    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7671-818013\" is forbidden: User \"system:node:ip-10-0-141-4.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-4.ec2.internal' and this object" podUID="3965a26e-c54b-4eb1-9ea3-812b1522c291" pod="opendatahub/test-curl-7671-818013"
Sep 14 03:11:39.099320 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:39.099291    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e1c1eeb-792d-462e-90ce-a81133803da5" path="/var/lib/kubelet/pods/8e1c1eeb-792d-462e-90ce-a81133803da5/volumes"
Sep 14 03:11:39.121045 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:39.121007    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7671-d16c76"
Sep 14 03:11:39.184624 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:39.184591    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qwxpr\" (UniqueName: \"kubernetes.io/projected/4d545046-40ad-46a8-ae37-aaf244f091c9-kube-api-access-qwxpr\") pod \"4d545046-40ad-46a8-ae37-aaf244f091c9\" (UID: \"4d545046-40ad-46a8-ae37-aaf244f091c9\") "
Sep 14 03:11:39.186666 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:39.186643    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d545046-40ad-46a8-ae37-aaf244f091c9-kube-api-access-qwxpr" (OuterVolumeSpecName: "kube-api-access-qwxpr") pod "4d545046-40ad-46a8-ae37-aaf244f091c9" (UID: "4d545046-40ad-46a8-ae37-aaf244f091c9"). InnerVolumeSpecName "kube-api-access-qwxpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:11:39.285495 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:39.285462    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwxpr\" (UniqueName: \"kubernetes.io/projected/4d545046-40ad-46a8-ae37-aaf244f091c9-kube-api-access-qwxpr\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:11:39.961520 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:39.961490    2564 generic.go:356] "Generic (PLEG): container finished" podID="4d545046-40ad-46a8-ae37-aaf244f091c9" containerID="aeb5569299dfcc76686bb95647f85642300bd660dda61cd1a24f45018e91d610" exitCode=137
Sep 14 03:11:39.961946 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:39.961531    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7671-d16c76"
Sep 14 03:11:39.961946 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:39.961594    2564 scope.go:117] "RemoveContainer" containerID="aeb5569299dfcc76686bb95647f85642300bd660dda61cd1a24f45018e91d610"
Sep 14 03:11:39.970463 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:39.970448    2564 scope.go:117] "RemoveContainer" containerID="aeb5569299dfcc76686bb95647f85642300bd660dda61cd1a24f45018e91d610"
Sep 14 03:11:39.970759 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:11:39.970738    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aeb5569299dfcc76686bb95647f85642300bd660dda61cd1a24f45018e91d610\": container with ID starting with aeb5569299dfcc76686bb95647f85642300bd660dda61cd1a24f45018e91d610 not found: ID does not exist" containerID="aeb5569299dfcc76686bb95647f85642300bd660dda61cd1a24f45018e91d610"
Sep 14 03:11:39.970819 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:39.970769    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aeb5569299dfcc76686bb95647f85642300bd660dda61cd1a24f45018e91d610"} err="failed to get container status \"aeb5569299dfcc76686bb95647f85642300bd660dda61cd1a24f45018e91d610\": rpc error: code = NotFound desc = could not find container \"aeb5569299dfcc76686bb95647f85642300bd660dda61cd1a24f45018e91d610\": container with ID starting with aeb5569299dfcc76686bb95647f85642300bd660dda61cd1a24f45018e91d610 not found: ID does not exist"
Sep 14 03:11:39.971993 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:11:39.971974    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7671-d16c76\" is forbidden: User \"system:node:ip-10-0-141-4.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-4.ec2.internal' and this object" podUID="4d545046-40ad-46a8-ae37-aaf244f091c9" pod="opendatahub/test-curl-7671-d16c76"
Sep 14 03:11:40.630719 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:40.630684    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:11:41.098821 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:41.098778    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d545046-40ad-46a8-ae37-aaf244f091c9" path="/var/lib/kubelet/pods/4d545046-40ad-46a8-ae37-aaf244f091c9/volumes"
Sep 14 03:11:41.134356 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:41.134332    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7671-818013"
Sep 14 03:11:41.206671 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:41.206636    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zszdw\" (UniqueName: \"kubernetes.io/projected/3965a26e-c54b-4eb1-9ea3-812b1522c291-kube-api-access-zszdw\") pod \"3965a26e-c54b-4eb1-9ea3-812b1522c291\" (UID: \"3965a26e-c54b-4eb1-9ea3-812b1522c291\") "
Sep 14 03:11:41.208712 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:41.208636    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3965a26e-c54b-4eb1-9ea3-812b1522c291-kube-api-access-zszdw" (OuterVolumeSpecName: "kube-api-access-zszdw") pod "3965a26e-c54b-4eb1-9ea3-812b1522c291" (UID: "3965a26e-c54b-4eb1-9ea3-812b1522c291"). InnerVolumeSpecName "kube-api-access-zszdw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:11:41.307905 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:41.307870    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zszdw\" (UniqueName: \"kubernetes.io/projected/3965a26e-c54b-4eb1-9ea3-812b1522c291-kube-api-access-zszdw\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:11:41.314148 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:41.314115    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:11:41.407155 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:41.407114    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:11:41.551970 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:41.551939    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:11:41.974247 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:41.974160    2564 generic.go:356] "Generic (PLEG): container finished" podID="3965a26e-c54b-4eb1-9ea3-812b1522c291" containerID="1eae766c3003247612af23b7666f7f91698d4ec2472047967240d4cf746e5a62" exitCode=137
Sep 14 03:11:41.974247 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:41.974203    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7671-818013"
Sep 14 03:11:41.974457 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:41.974271    2564 scope.go:117] "RemoveContainer" containerID="1eae766c3003247612af23b7666f7f91698d4ec2472047967240d4cf746e5a62"
Sep 14 03:11:41.983359 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:41.983336    2564 scope.go:117] "RemoveContainer" containerID="1eae766c3003247612af23b7666f7f91698d4ec2472047967240d4cf746e5a62"
Sep 14 03:11:41.983658 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:11:41.983637    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1eae766c3003247612af23b7666f7f91698d4ec2472047967240d4cf746e5a62\": container with ID starting with 1eae766c3003247612af23b7666f7f91698d4ec2472047967240d4cf746e5a62 not found: ID does not exist" containerID="1eae766c3003247612af23b7666f7f91698d4ec2472047967240d4cf746e5a62"
Sep 14 03:11:41.983708 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:41.983666    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1eae766c3003247612af23b7666f7f91698d4ec2472047967240d4cf746e5a62"} err="failed to get container status \"1eae766c3003247612af23b7666f7f91698d4ec2472047967240d4cf746e5a62\": rpc error: code = NotFound desc = could not find container \"1eae766c3003247612af23b7666f7f91698d4ec2472047967240d4cf746e5a62\": container with ID starting with 1eae766c3003247612af23b7666f7f91698d4ec2472047967240d4cf746e5a62 not found: ID does not exist"
Sep 14 03:11:41.984627 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:11:41.984605    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7671-818013\" is forbidden: User \"system:node:ip-10-0-141-4.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-4.ec2.internal' and this object" podUID="3965a26e-c54b-4eb1-9ea3-812b1522c291" pod="opendatahub/test-curl-7671-818013"
Sep 14 03:11:42.257320 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:42.257178    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:11:43.099251 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:43.099218    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3965a26e-c54b-4eb1-9ea3-812b1522c291" path="/var/lib/kubelet/pods/3965a26e-c54b-4eb1-9ea3-812b1522c291/volumes"
Sep 14 03:11:43.256902 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:43.256864    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:11:44.088930 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:44.088896    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:11:44.893085 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:44.893053    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:11:45.531355 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:45.531322    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-12ecd131-78cd7db6f7-6rt5l"]
Sep 14 03:11:45.532101 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:45.532077    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3965a26e-c54b-4eb1-9ea3-812b1522c291" containerName="test-curl-7671-818013"
Sep 14 03:11:45.532263 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:45.532239    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3965a26e-c54b-4eb1-9ea3-812b1522c291" containerName="test-curl-7671-818013"
Sep 14 03:11:45.532462 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:45.532445    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="3965a26e-c54b-4eb1-9ea3-812b1522c291" containerName="test-curl-7671-818013"
Sep 14 03:11:45.537381 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:45.537361    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-12ecd131-78cd7db6f7-6rt5l"
Sep 14 03:11:45.539856 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:45.539834    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-12ecd131-serving-cert\""
Sep 14 03:11:45.549974 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:45.549946    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-12ecd131-78cd7db6f7-6rt5l"]
Sep 14 03:11:45.567095 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:45.567059    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:11:45.633990 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:45.633959    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:11:45.649059 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:45.649030    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dffb2be6-e854-460c-9af8-005afdad9108-maas-api-tls\") pod \"maas-api-e2e-mt-12ecd131-78cd7db6f7-6rt5l\" (UID: \"dffb2be6-e854-460c-9af8-005afdad9108\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-12ecd131-78cd7db6f7-6rt5l"
Sep 14 03:11:45.649224 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:45.649065    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dffb2be6-e854-460c-9af8-005afdad9108-metrics-certs\") pod \"maas-api-e2e-mt-12ecd131-78cd7db6f7-6rt5l\" (UID: \"dffb2be6-e854-460c-9af8-005afdad9108\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-12ecd131-78cd7db6f7-6rt5l"
Sep 14 03:11:45.649224 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:45.649146    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dffb2be6-e854-460c-9af8-005afdad9108-openshift-service-ca\") pod \"maas-api-e2e-mt-12ecd131-78cd7db6f7-6rt5l\" (UID: \"dffb2be6-e854-460c-9af8-005afdad9108\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-12ecd131-78cd7db6f7-6rt5l"
Sep 14 03:11:45.649292 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:45.649220    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx6b2\" (UniqueName: \"kubernetes.io/projected/dffb2be6-e854-460c-9af8-005afdad9108-kube-api-access-zx6b2\") pod \"maas-api-e2e-mt-12ecd131-78cd7db6f7-6rt5l\" (UID: \"dffb2be6-e854-460c-9af8-005afdad9108\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-12ecd131-78cd7db6f7-6rt5l"
Sep 14 03:11:45.749963 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:45.749918    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dffb2be6-e854-460c-9af8-005afdad9108-maas-api-tls\") pod \"maas-api-e2e-mt-12ecd131-78cd7db6f7-6rt5l\" (UID: \"dffb2be6-e854-460c-9af8-005afdad9108\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-12ecd131-78cd7db6f7-6rt5l"
Sep 14 03:11:45.749963 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:45.749970    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dffb2be6-e854-460c-9af8-005afdad9108-metrics-certs\") pod \"maas-api-e2e-mt-12ecd131-78cd7db6f7-6rt5l\" (UID: \"dffb2be6-e854-460c-9af8-005afdad9108\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-12ecd131-78cd7db6f7-6rt5l"
Sep 14 03:11:45.750197 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:45.750038    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dffb2be6-e854-460c-9af8-005afdad9108-openshift-service-ca\") pod \"maas-api-e2e-mt-12ecd131-78cd7db6f7-6rt5l\" (UID: \"dffb2be6-e854-460c-9af8-005afdad9108\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-12ecd131-78cd7db6f7-6rt5l"
Sep 14 03:11:45.750197 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:11:45.750098    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-12ecd131-serving-cert: secret "maas-api-e2e-mt-12ecd131-serving-cert" not found
Sep 14 03:11:45.750197 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:45.750113    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx6b2\" (UniqueName: \"kubernetes.io/projected/dffb2be6-e854-460c-9af8-005afdad9108-kube-api-access-zx6b2\") pod \"maas-api-e2e-mt-12ecd131-78cd7db6f7-6rt5l\" (UID: \"dffb2be6-e854-460c-9af8-005afdad9108\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-12ecd131-78cd7db6f7-6rt5l"
Sep 14 03:11:45.750197 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:11:45.750171    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dffb2be6-e854-460c-9af8-005afdad9108-maas-api-tls podName:dffb2be6-e854-460c-9af8-005afdad9108 nodeName:}" failed. No retries permitted until 2026-09-14 03:11:46.250149776 +0000 UTC m=+1151.631233955 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dffb2be6-e854-460c-9af8-005afdad9108-maas-api-tls") pod "maas-api-e2e-mt-12ecd131-78cd7db6f7-6rt5l" (UID: "dffb2be6-e854-460c-9af8-005afdad9108") : secret "maas-api-e2e-mt-12ecd131-serving-cert" not found
Sep 14 03:11:45.750849 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:45.750824    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dffb2be6-e854-460c-9af8-005afdad9108-openshift-service-ca\") pod \"maas-api-e2e-mt-12ecd131-78cd7db6f7-6rt5l\" (UID: \"dffb2be6-e854-460c-9af8-005afdad9108\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-12ecd131-78cd7db6f7-6rt5l"
Sep 14 03:11:45.752526 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:45.752503    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dffb2be6-e854-460c-9af8-005afdad9108-metrics-certs\") pod \"maas-api-e2e-mt-12ecd131-78cd7db6f7-6rt5l\" (UID: \"dffb2be6-e854-460c-9af8-005afdad9108\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-12ecd131-78cd7db6f7-6rt5l"
Sep 14 03:11:45.758919 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:45.758889    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx6b2\" (UniqueName: \"kubernetes.io/projected/dffb2be6-e854-460c-9af8-005afdad9108-kube-api-access-zx6b2\") pod \"maas-api-e2e-mt-12ecd131-78cd7db6f7-6rt5l\" (UID: \"dffb2be6-e854-460c-9af8-005afdad9108\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-12ecd131-78cd7db6f7-6rt5l"
Sep 14 03:11:45.959128 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:45.959031    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:11:46.055094 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:46.055060    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:11:46.146259 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:46.146223    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:11:46.150679 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:46.150644    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-12ecd131-7f655f44fd-cm47f"]
Sep 14 03:11:46.154558 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:46.154542    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-12ecd131-7f655f44fd-cm47f"
Sep 14 03:11:46.159179 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:46.159158    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-12ecd131-dockercfg-5l6lp\""
Sep 14 03:11:46.159314 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:46.159184    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-12ecd131\""
Sep 14 03:11:46.171820 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:46.171792    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-12ecd131-7f655f44fd-cm47f"]
Sep 14 03:11:46.256503 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:46.256422    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dffb2be6-e854-460c-9af8-005afdad9108-maas-api-tls\") pod \"maas-api-e2e-mt-12ecd131-78cd7db6f7-6rt5l\" (UID: \"dffb2be6-e854-460c-9af8-005afdad9108\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-12ecd131-78cd7db6f7-6rt5l"
Sep 14 03:11:46.256503 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:46.256485    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64j45\" (UniqueName: \"kubernetes.io/projected/238e6109-9a10-45a4-9014-cfba173627c7-kube-api-access-64j45\") pod \"payload-pre-processing-e2e-mt-12ecd131-7f655f44fd-cm47f\" (UID: \"238e6109-9a10-45a4-9014-cfba173627c7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-12ecd131-7f655f44fd-cm47f"
Sep 14 03:11:46.256701 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:46.256627    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/238e6109-9a10-45a4-9014-cfba173627c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-12ecd131-7f655f44fd-cm47f\" (UID: \"238e6109-9a10-45a4-9014-cfba173627c7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-12ecd131-7f655f44fd-cm47f"
Sep 14 03:11:46.258746 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:46.258726    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dffb2be6-e854-460c-9af8-005afdad9108-maas-api-tls\") pod \"maas-api-e2e-mt-12ecd131-78cd7db6f7-6rt5l\" (UID: \"dffb2be6-e854-460c-9af8-005afdad9108\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-12ecd131-78cd7db6f7-6rt5l"
Sep 14 03:11:46.357932 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:46.357885    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/238e6109-9a10-45a4-9014-cfba173627c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-12ecd131-7f655f44fd-cm47f\" (UID: \"238e6109-9a10-45a4-9014-cfba173627c7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-12ecd131-7f655f44fd-cm47f"
Sep 14 03:11:46.358103 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:46.357984    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64j45\" (UniqueName: \"kubernetes.io/projected/238e6109-9a10-45a4-9014-cfba173627c7-kube-api-access-64j45\") pod \"payload-pre-processing-e2e-mt-12ecd131-7f655f44fd-cm47f\" (UID: \"238e6109-9a10-45a4-9014-cfba173627c7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-12ecd131-7f655f44fd-cm47f"
Sep 14 03:11:46.358573 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:46.358554    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/238e6109-9a10-45a4-9014-cfba173627c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-12ecd131-7f655f44fd-cm47f\" (UID: \"238e6109-9a10-45a4-9014-cfba173627c7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-12ecd131-7f655f44fd-cm47f"
Sep 14 03:11:46.366888 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:46.366867    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64j45\" (UniqueName: \"kubernetes.io/projected/238e6109-9a10-45a4-9014-cfba173627c7-kube-api-access-64j45\") pod \"payload-pre-processing-e2e-mt-12ecd131-7f655f44fd-cm47f\" (UID: \"238e6109-9a10-45a4-9014-cfba173627c7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-12ecd131-7f655f44fd-cm47f"
Sep 14 03:11:46.450957 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:46.450923    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-12ecd131-78cd7db6f7-6rt5l"
Sep 14 03:11:46.465274 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:46.465248    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-12ecd131-7f655f44fd-cm47f"
Sep 14 03:11:46.726630 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:46.726596    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-12ecd131-78cd7db6f7-6rt5l"]
Sep 14 03:11:46.727097 ip-10-0-141-4 kubenswrapper[2564]: W0914 03:11:46.727069    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddffb2be6_e854_460c_9af8_005afdad9108.slice/crio-15c69408bdcc9aa2d364e4c159cb720633a9912091859bd36f8d0f3ca72d4a75 WatchSource:0}: Error finding container 15c69408bdcc9aa2d364e4c159cb720633a9912091859bd36f8d0f3ca72d4a75: Status 404 returned error can't find the container with id 15c69408bdcc9aa2d364e4c159cb720633a9912091859bd36f8d0f3ca72d4a75
Sep 14 03:11:46.728735 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:46.728705    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 14 03:11:46.745446 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:46.745423    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-12ecd131-7f655f44fd-cm47f"]
Sep 14 03:11:46.746390 ip-10-0-141-4 kubenswrapper[2564]: W0914 03:11:46.746369    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod238e6109_9a10_45a4_9014_cfba173627c7.slice/crio-04e687f185208b0f34a82a74dfc2b52a439c573ab0846a2152aaf65fba805c2a WatchSource:0}: Error finding container 04e687f185208b0f34a82a74dfc2b52a439c573ab0846a2152aaf65fba805c2a: Status 404 returned error can't find the container with id 04e687f185208b0f34a82a74dfc2b52a439c573ab0846a2152aaf65fba805c2a
Sep 14 03:11:47.003594 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:47.003557    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-12ecd131-78cd7db6f7-6rt5l" event={"ID":"dffb2be6-e854-460c-9af8-005afdad9108","Type":"ContainerStarted","Data":"15c69408bdcc9aa2d364e4c159cb720633a9912091859bd36f8d0f3ca72d4a75"}
Sep 14 03:11:47.004969 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:47.004944    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-12ecd131-7f655f44fd-cm47f" event={"ID":"238e6109-9a10-45a4-9014-cfba173627c7","Type":"ContainerStarted","Data":"d3b1c947d4f556f7dad261924a2768f8d9ad8bad987af382fcc319906aacd079"}
Sep 14 03:11:47.005093 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:47.004984    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-12ecd131-7f655f44fd-cm47f" event={"ID":"238e6109-9a10-45a4-9014-cfba173627c7","Type":"ContainerStarted","Data":"04e687f185208b0f34a82a74dfc2b52a439c573ab0846a2152aaf65fba805c2a"}
Sep 14 03:11:47.005093 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:47.005077    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-12ecd131-7f655f44fd-cm47f"
Sep 14 03:11:47.040559 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:47.040482    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-12ecd131-7f655f44fd-cm47f" podStartSLOduration=1.04046267 podStartE2EDuration="1.04046267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:11:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:11:47.039705793 +0000 UTC m=+1152.420789982" watchObservedRunningTime="2026-09-14 03:11:47.04046267 +0000 UTC m=+1152.421546868"
Sep 14 03:11:48.011848 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:48.011800    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-12ecd131-78cd7db6f7-6rt5l" event={"ID":"dffb2be6-e854-460c-9af8-005afdad9108","Type":"ContainerStarted","Data":"8cc98ac423fae5fc6857c32e2461087a27fd7d6ca25f49e93c110582179bb8f9"}
Sep 14 03:11:48.012288 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:48.012014    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-12ecd131-78cd7db6f7-6rt5l"
Sep 14 03:11:48.031929 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:48.031873    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-12ecd131-78cd7db6f7-6rt5l" podStartSLOduration=2.338823202 podStartE2EDuration="3.031855562s" totalImagesPullingTime="693.03236ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:11:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:11:48.030744127 +0000 UTC m=+1153.411828326" watchObservedRunningTime="2026-09-14 03:11:48.031855562 +0000 UTC m=+1153.412939759"
Sep 14 03:11:48.201555 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:11:48.197835    2564 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod45b8e89b_b746_4ff9_84b4_f11c8a7c5aeb.slice/crio-74976527d5ad5a4609da722e382b3e41a9f6b44b71032b97965a91026e9c9d5c\": RecentStats: unable to find data in memory cache]"
Sep 14 03:11:50.715563 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:11:50.715526    2564 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod45b8e89b_b746_4ff9_84b4_f11c8a7c5aeb.slice/crio-74976527d5ad5a4609da722e382b3e41a9f6b44b71032b97965a91026e9c9d5c\": RecentStats: unable to find data in memory cache]"
Sep 14 03:11:52.856902 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:52.856869    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:11:53.594017 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:53.593983    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:11:53.822732 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:53.822693    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:11:53.922293 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:53.922212    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:11:54.023123 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:54.023098    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-12ecd131-78cd7db6f7-6rt5l"
Sep 14 03:11:58.013841 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:58.013769    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-12ecd131-7f655f44fd-cm47f"
Sep 14 03:11:58.257317 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:11:58.257287    2564 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod45b8e89b_b746_4ff9_84b4_f11c8a7c5aeb.slice/crio-74976527d5ad5a4609da722e382b3e41a9f6b44b71032b97965a91026e9c9d5c\": RecentStats: unable to find data in memory cache]"
Sep 14 03:11:58.899395 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:58.899363    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-f16030/noten-model-f16030-kserve-86d665dbc-snqc4"]
Sep 14 03:11:58.994469 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:11:58.994436    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:12:00.266369 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:00.266332    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:12:05.599314 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:05.599277    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:12:05.718936 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:12:05.718900    2564 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod45b8e89b_b746_4ff9_84b4_f11c8a7c5aeb.slice/crio-74976527d5ad5a4609da722e382b3e41a9f6b44b71032b97965a91026e9c9d5c\": RecentStats: unable to find data in memory cache]"
Sep 14 03:12:05.851553 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:05.851464    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:12:06.301230 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:06.301193    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:12:07.186857 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:07.186823    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:12:08.103755 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:08.103717    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:12:08.268171 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:12:08.268135    2564 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod45b8e89b_b746_4ff9_84b4_f11c8a7c5aeb.slice/crio-74976527d5ad5a4609da722e382b3e41a9f6b44b71032b97965a91026e9c9d5c\": RecentStats: unable to find data in memory cache]"
Sep 14 03:12:08.395136 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:08.395044    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:12:08.851792 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:08.851753    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:12:09.648969 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:09.648934    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:12:10.283314 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:10.283280    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:12:10.342844 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:10.342810    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-b-7a5ebdfc-kserve-77998f997d-dl52h"]
Sep 14 03:12:10.373510 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:10.373477    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:12:11.123015 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:11.122971    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:12:11.393349 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:11.393248    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:12:11.848137 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:11.848092    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:12:12.780727 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:12.780692    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:12:13.731254 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:13.731217    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:12:13.918686 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:13.918622    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-f16030/noten-model-f16030-kserve-86d665dbc-snqc4" podUID="f345be18-091e-48e6-bbf4-d4581301b800" containerName="main" containerID="cri-o://78f2cf32c8d1bf3d3b3d3535cd52fd8781ee19bae589c1760772a65e0295df71" gracePeriod=45
Sep 14 03:12:14.156937 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:14.156741    2564 generic.go:356] "Generic (PLEG): container finished" podID="f345be18-091e-48e6-bbf4-d4581301b800" containerID="78f2cf32c8d1bf3d3b3d3535cd52fd8781ee19bae589c1760772a65e0295df71" exitCode=0
Sep 14 03:12:14.156937 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:14.156815    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-f16030/noten-model-f16030-kserve-86d665dbc-snqc4" event={"ID":"f345be18-091e-48e6-bbf4-d4581301b800","Type":"ContainerDied","Data":"78f2cf32c8d1bf3d3b3d3535cd52fd8781ee19bae589c1760772a65e0295df71"}
Sep 14 03:12:14.156937 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:14.156857    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-f16030/noten-model-f16030-kserve-86d665dbc-snqc4" event={"ID":"f345be18-091e-48e6-bbf4-d4581301b800","Type":"ContainerDied","Data":"54340071ca6415e0d6768b09287a90ac1e82d83dd2880471e0cd2dcd7152f00d"}
Sep 14 03:12:14.156937 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:14.156870    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54340071ca6415e0d6768b09287a90ac1e82d83dd2880471e0cd2dcd7152f00d"
Sep 14 03:12:14.174154 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:14.174130    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-f16030/noten-model-f16030-kserve-86d665dbc-snqc4"
Sep 14 03:12:14.250515 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:14.250487    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f345be18-091e-48e6-bbf4-d4581301b800-dshm\") pod \"f345be18-091e-48e6-bbf4-d4581301b800\" (UID: \"f345be18-091e-48e6-bbf4-d4581301b800\") "
Sep 14 03:12:14.250682 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:14.250557    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f345be18-091e-48e6-bbf4-d4581301b800-model-cache\") pod \"f345be18-091e-48e6-bbf4-d4581301b800\" (UID: \"f345be18-091e-48e6-bbf4-d4581301b800\") "
Sep 14 03:12:14.250682 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:14.250588    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f345be18-091e-48e6-bbf4-d4581301b800-home\") pod \"f345be18-091e-48e6-bbf4-d4581301b800\" (UID: \"f345be18-091e-48e6-bbf4-d4581301b800\") "
Sep 14 03:12:14.250682 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:14.250662    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f345be18-091e-48e6-bbf4-d4581301b800-tls-certs\") pod \"f345be18-091e-48e6-bbf4-d4581301b800\" (UID: \"f345be18-091e-48e6-bbf4-d4581301b800\") "
Sep 14 03:12:14.250854 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:14.250716    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q8xl2\" (UniqueName: \"kubernetes.io/projected/f345be18-091e-48e6-bbf4-d4581301b800-kube-api-access-q8xl2\") pod \"f345be18-091e-48e6-bbf4-d4581301b800\" (UID: \"f345be18-091e-48e6-bbf4-d4581301b800\") "
Sep 14 03:12:14.250854 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:14.250768    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f345be18-091e-48e6-bbf4-d4581301b800-tmp-dir\") pod \"f345be18-091e-48e6-bbf4-d4581301b800\" (UID: \"f345be18-091e-48e6-bbf4-d4581301b800\") "
Sep 14 03:12:14.250969 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:14.250843    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f345be18-091e-48e6-bbf4-d4581301b800-model-cache" (OuterVolumeSpecName: "model-cache") pod "f345be18-091e-48e6-bbf4-d4581301b800" (UID: "f345be18-091e-48e6-bbf4-d4581301b800"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:12:14.250969 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:14.250927    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f345be18-091e-48e6-bbf4-d4581301b800-home" (OuterVolumeSpecName: "home") pod "f345be18-091e-48e6-bbf4-d4581301b800" (UID: "f345be18-091e-48e6-bbf4-d4581301b800"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:12:14.251124 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:14.251103    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f345be18-091e-48e6-bbf4-d4581301b800-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f345be18-091e-48e6-bbf4-d4581301b800" (UID: "f345be18-091e-48e6-bbf4-d4581301b800"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:12:14.251182 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:14.251163    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f345be18-091e-48e6-bbf4-d4581301b800-model-cache\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:12:14.251222 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:14.251190    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f345be18-091e-48e6-bbf4-d4581301b800-home\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:12:14.252803 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:14.252782    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f345be18-091e-48e6-bbf4-d4581301b800-dshm" (OuterVolumeSpecName: "dshm") pod "f345be18-091e-48e6-bbf4-d4581301b800" (UID: "f345be18-091e-48e6-bbf4-d4581301b800"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:12:14.252904 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:14.252820    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f345be18-091e-48e6-bbf4-d4581301b800-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f345be18-091e-48e6-bbf4-d4581301b800" (UID: "f345be18-091e-48e6-bbf4-d4581301b800"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 03:12:14.252904 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:14.252838    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f345be18-091e-48e6-bbf4-d4581301b800-kube-api-access-q8xl2" (OuterVolumeSpecName: "kube-api-access-q8xl2") pod "f345be18-091e-48e6-bbf4-d4581301b800" (UID: "f345be18-091e-48e6-bbf4-d4581301b800"). InnerVolumeSpecName "kube-api-access-q8xl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:12:14.352215 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:14.352139    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8xl2\" (UniqueName: \"kubernetes.io/projected/f345be18-091e-48e6-bbf4-d4581301b800-kube-api-access-q8xl2\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:12:14.352215 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:14.352167    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f345be18-091e-48e6-bbf4-d4581301b800-tmp-dir\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:12:14.352215 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:14.352177    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f345be18-091e-48e6-bbf4-d4581301b800-dshm\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:12:14.352215 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:14.352186    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f345be18-091e-48e6-bbf4-d4581301b800-tls-certs\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:12:15.162588 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:15.162558    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-f16030/noten-model-f16030-kserve-86d665dbc-snqc4"
Sep 14 03:12:15.182652 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:15.182620    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-f16030/noten-model-f16030-kserve-86d665dbc-snqc4"]
Sep 14 03:12:15.186317 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:15.186294    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-f16030/noten-model-f16030-kserve-86d665dbc-snqc4"]
Sep 14 03:12:16.313689 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:16.313654    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:12:16.855286 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:16.855248    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4eb26b-6bcfc6b59-fwtl2"]
Sep 14 03:12:16.855555 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:16.855530    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4eb26b-6bcfc6b59-fwtl2" podUID="2169752d-69ca-4313-aa66-b4c853c1b255" containerName="maas-api" containerID="cri-o://f954504db1f5aeacc09d0a0c95d2b25db58a7de9dc7187134f9fef95307d535f" gracePeriod=30
Sep 14 03:12:16.896821 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:16.896789    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:12:16.987958 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:16.987916    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:12:17.099862 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:17.099831    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f345be18-091e-48e6-bbf4-d4581301b800" path="/var/lib/kubelet/pods/f345be18-091e-48e6-bbf4-d4581301b800/volumes"
Sep 14 03:12:17.118481 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:17.118423    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4eb26b-6bcfc6b59-fwtl2"
Sep 14 03:12:17.145331 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:17.145299    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:12:17.174822 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:17.174789    2564 generic.go:356] "Generic (PLEG): container finished" podID="2169752d-69ca-4313-aa66-b4c853c1b255" containerID="f954504db1f5aeacc09d0a0c95d2b25db58a7de9dc7187134f9fef95307d535f" exitCode=0
Sep 14 03:12:17.174962 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:17.174834    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4eb26b-6bcfc6b59-fwtl2"
Sep 14 03:12:17.174962 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:17.174832    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4eb26b-6bcfc6b59-fwtl2" event={"ID":"2169752d-69ca-4313-aa66-b4c853c1b255","Type":"ContainerDied","Data":"f954504db1f5aeacc09d0a0c95d2b25db58a7de9dc7187134f9fef95307d535f"}
Sep 14 03:12:17.174962 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:17.174893    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4eb26b-6bcfc6b59-fwtl2" event={"ID":"2169752d-69ca-4313-aa66-b4c853c1b255","Type":"ContainerDied","Data":"a0291458d1bbc9e9dec7a96015b4170639d49caf3cd8eb7dda88116258488041"}
Sep 14 03:12:17.174962 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:17.174923    2564 scope.go:117] "RemoveContainer" containerID="f954504db1f5aeacc09d0a0c95d2b25db58a7de9dc7187134f9fef95307d535f"
Sep 14 03:12:17.183843 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:17.183823    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-krp7b\" (UniqueName: \"kubernetes.io/projected/2169752d-69ca-4313-aa66-b4c853c1b255-kube-api-access-krp7b\") pod \"2169752d-69ca-4313-aa66-b4c853c1b255\" (UID: \"2169752d-69ca-4313-aa66-b4c853c1b255\") "
Sep 14 03:12:17.183949 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:17.183857    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2169752d-69ca-4313-aa66-b4c853c1b255-maas-api-tls\") pod \"2169752d-69ca-4313-aa66-b4c853c1b255\" (UID: \"2169752d-69ca-4313-aa66-b4c853c1b255\") "
Sep 14 03:12:17.183949 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:17.183939    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2169752d-69ca-4313-aa66-b4c853c1b255-openshift-service-ca\") pod \"2169752d-69ca-4313-aa66-b4c853c1b255\" (UID: \"2169752d-69ca-4313-aa66-b4c853c1b255\") "
Sep 14 03:12:17.184076 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:17.184012    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2169752d-69ca-4313-aa66-b4c853c1b255-metrics-certs\") pod \"2169752d-69ca-4313-aa66-b4c853c1b255\" (UID: \"2169752d-69ca-4313-aa66-b4c853c1b255\") "
Sep 14 03:12:17.184220 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:17.184200    2564 scope.go:117] "RemoveContainer" containerID="f954504db1f5aeacc09d0a0c95d2b25db58a7de9dc7187134f9fef95307d535f"
Sep 14 03:12:17.184421 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:17.184380    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2169752d-69ca-4313-aa66-b4c853c1b255-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2169752d-69ca-4313-aa66-b4c853c1b255" (UID: "2169752d-69ca-4313-aa66-b4c853c1b255"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 03:12:17.184626 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:12:17.184590    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f954504db1f5aeacc09d0a0c95d2b25db58a7de9dc7187134f9fef95307d535f\": container with ID starting with f954504db1f5aeacc09d0a0c95d2b25db58a7de9dc7187134f9fef95307d535f not found: ID does not exist" containerID="f954504db1f5aeacc09d0a0c95d2b25db58a7de9dc7187134f9fef95307d535f"
Sep 14 03:12:17.184737 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:17.184635    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f954504db1f5aeacc09d0a0c95d2b25db58a7de9dc7187134f9fef95307d535f"} err="failed to get container status \"f954504db1f5aeacc09d0a0c95d2b25db58a7de9dc7187134f9fef95307d535f\": rpc error: code = NotFound desc = could not find container \"f954504db1f5aeacc09d0a0c95d2b25db58a7de9dc7187134f9fef95307d535f\": container with ID starting with f954504db1f5aeacc09d0a0c95d2b25db58a7de9dc7187134f9fef95307d535f not found: ID does not exist"
Sep 14 03:12:17.186177 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:17.186150    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2169752d-69ca-4313-aa66-b4c853c1b255-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2169752d-69ca-4313-aa66-b4c853c1b255" (UID: "2169752d-69ca-4313-aa66-b4c853c1b255"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 03:12:17.186177 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:17.186166    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2169752d-69ca-4313-aa66-b4c853c1b255-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2169752d-69ca-4313-aa66-b4c853c1b255" (UID: "2169752d-69ca-4313-aa66-b4c853c1b255"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 03:12:17.186483 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:17.186466    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2169752d-69ca-4313-aa66-b4c853c1b255-kube-api-access-krp7b" (OuterVolumeSpecName: "kube-api-access-krp7b") pod "2169752d-69ca-4313-aa66-b4c853c1b255" (UID: "2169752d-69ca-4313-aa66-b4c853c1b255"). InnerVolumeSpecName "kube-api-access-krp7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:12:17.285323 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:17.285292    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2169752d-69ca-4313-aa66-b4c853c1b255-metrics-certs\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:12:17.285323 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:17.285324    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krp7b\" (UniqueName: \"kubernetes.io/projected/2169752d-69ca-4313-aa66-b4c853c1b255-kube-api-access-krp7b\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:12:17.285551 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:17.285339    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2169752d-69ca-4313-aa66-b4c853c1b255-maas-api-tls\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:12:17.285551 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:17.285353    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2169752d-69ca-4313-aa66-b4c853c1b255-openshift-service-ca\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:12:17.296519 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:17.296485    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:12:17.504030 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:17.503998    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4eb26b-6bcfc6b59-fwtl2"]
Sep 14 03:12:17.508597 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:17.508568    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4eb26b-6bcfc6b59-fwtl2"]
Sep 14 03:12:17.547600 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:17.547557    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:12:17.663258 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:17.663218    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:12:18.324077 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:12:18.324043    2564 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod45b8e89b_b746_4ff9_84b4_f11c8a7c5aeb.slice/crio-74976527d5ad5a4609da722e382b3e41a9f6b44b71032b97965a91026e9c9d5c\": RecentStats: unable to find data in memory cache]"
Sep 14 03:12:18.438615 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:18.438578    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:12:19.098560 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:19.098528    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2169752d-69ca-4313-aa66-b4c853c1b255" path="/var/lib/kubelet/pods/2169752d-69ca-4313-aa66-b4c853c1b255/volumes"
Sep 14 03:12:19.214787 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:19.214752    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:12:20.125149 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:20.125109    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:12:20.677679 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:12:20.677643    2564 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod45b8e89b_b746_4ff9_84b4_f11c8a7c5aeb.slice/crio-74976527d5ad5a4609da722e382b3e41a9f6b44b71032b97965a91026e9c9d5c\": RecentStats: unable to find data in memory cache]"
Sep 14 03:12:20.771676 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:20.771641    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:12:22.026888 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:22.026841    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:12:23.780955 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:23.780919    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:12:23.859901 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:23.859865    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:12:24.037427 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:24.037303    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:12:24.591037 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:24.591001    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:12:24.966996 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:24.966903    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:12:25.071700 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:25.071669    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:12:25.361675 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:25.361633    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-b-7a5ebdfc-kserve-77998f997d-dl52h" podUID="0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f" containerName="main" containerID="cri-o://3591d13aaffff55dad566c7833116fe88c0267af3c9c4bab778ce8eb8f2d4c44" gracePeriod=45
Sep 14 03:12:25.614456 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:25.614382    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-b-7a5ebdfc-kserve-77998f997d-dl52h"
Sep 14 03:12:25.667666 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:25.667635    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f-dshm\") pod \"0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f\" (UID: \"0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f\") "
Sep 14 03:12:25.667858 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:25.667673    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f-model-cache\") pod \"0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f\" (UID: \"0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f\") "
Sep 14 03:12:25.667858 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:25.667714    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f-tls-certs\") pod \"0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f\" (UID: \"0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f\") "
Sep 14 03:12:25.667858 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:25.667763    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m7ftm\" (UniqueName: \"kubernetes.io/projected/0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f-kube-api-access-m7ftm\") pod \"0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f\" (UID: \"0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f\") "
Sep 14 03:12:25.667858 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:25.667805    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f-tmp-dir\") pod \"0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f\" (UID: \"0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f\") "
Sep 14 03:12:25.667858 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:25.667834    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f-home\") pod \"0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f\" (UID: \"0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f\") "
Sep 14 03:12:25.668135 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:25.667959    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f-model-cache" (OuterVolumeSpecName: "model-cache") pod "0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f" (UID: "0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:12:25.668135 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:25.668106    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f" (UID: "0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:12:25.668254 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:25.668142    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f-home" (OuterVolumeSpecName: "home") pod "0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f" (UID: "0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:12:25.668310 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:25.668277    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f-tmp-dir\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:12:25.668310 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:25.668291    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f-home\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:12:25.668310 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:25.668301    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f-model-cache\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:12:25.669904 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:25.669876    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f-dshm" (OuterVolumeSpecName: "dshm") pod "0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f" (UID: "0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:12:25.670048 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:25.669938    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f-kube-api-access-m7ftm" (OuterVolumeSpecName: "kube-api-access-m7ftm") pod "0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f" (UID: "0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f"). InnerVolumeSpecName "kube-api-access-m7ftm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:12:25.670048 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:25.669958    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f" (UID: "0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 03:12:25.769480 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:25.769440    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f-tls-certs\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:12:25.769480 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:25.769474    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7ftm\" (UniqueName: \"kubernetes.io/projected/0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f-kube-api-access-m7ftm\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:12:25.769480 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:25.769486    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f-dshm\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:12:26.227886 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:26.227851    2564 generic.go:356] "Generic (PLEG): container finished" podID="0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f" containerID="3591d13aaffff55dad566c7833116fe88c0267af3c9c4bab778ce8eb8f2d4c44" exitCode=0
Sep 14 03:12:26.228307 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:26.227894    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-b-7a5ebdfc-kserve-77998f997d-dl52h"
Sep 14 03:12:26.228307 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:26.227936    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-b-7a5ebdfc-kserve-77998f997d-dl52h" event={"ID":"0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f","Type":"ContainerDied","Data":"3591d13aaffff55dad566c7833116fe88c0267af3c9c4bab778ce8eb8f2d4c44"}
Sep 14 03:12:26.228307 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:26.227986    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-b-7a5ebdfc-kserve-77998f997d-dl52h" event={"ID":"0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f","Type":"ContainerDied","Data":"f9253aa5b9af7280e99d058ba095248a13325cf0ec9e2ee05d05c61e292f2ab8"}
Sep 14 03:12:26.228307 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:26.228005    2564 scope.go:117] "RemoveContainer" containerID="3591d13aaffff55dad566c7833116fe88c0267af3c9c4bab778ce8eb8f2d4c44"
Sep 14 03:12:26.237471 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:26.237452    2564 scope.go:117] "RemoveContainer" containerID="3591d13aaffff55dad566c7833116fe88c0267af3c9c4bab778ce8eb8f2d4c44"
Sep 14 03:12:26.237771 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:12:26.237754    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3591d13aaffff55dad566c7833116fe88c0267af3c9c4bab778ce8eb8f2d4c44\": container with ID starting with 3591d13aaffff55dad566c7833116fe88c0267af3c9c4bab778ce8eb8f2d4c44 not found: ID does not exist" containerID="3591d13aaffff55dad566c7833116fe88c0267af3c9c4bab778ce8eb8f2d4c44"
Sep 14 03:12:26.237822 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:26.237782    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3591d13aaffff55dad566c7833116fe88c0267af3c9c4bab778ce8eb8f2d4c44"} err="failed to get container status \"3591d13aaffff55dad566c7833116fe88c0267af3c9c4bab778ce8eb8f2d4c44\": rpc error: code = NotFound desc = could not find container \"3591d13aaffff55dad566c7833116fe88c0267af3c9c4bab778ce8eb8f2d4c44\": container with ID starting with 3591d13aaffff55dad566c7833116fe88c0267af3c9c4bab778ce8eb8f2d4c44 not found: ID does not exist"
Sep 14 03:12:26.251011 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:26.250988    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-b-7a5ebdfc-kserve-77998f997d-dl52h"]
Sep 14 03:12:26.257168 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:26.257147    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-b-7a5ebdfc-kserve-77998f997d-dl52h"]
Sep 14 03:12:26.925204 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:26.925164    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:12:27.098592 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:27.098550    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f" path="/var/lib/kubelet/pods/0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f/volumes"
Sep 14 03:12:27.621024 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:27.620991    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:12:27.765751 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:27.765712    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:12:27.919320 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:27.919231    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:12:28.381325 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:12:28.381295    2564 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod45b8e89b_b746_4ff9_84b4_f11c8a7c5aeb.slice/crio-74976527d5ad5a4609da722e382b3e41a9f6b44b71032b97965a91026e9c9d5c\": RecentStats: unable to find data in memory cache]"
Sep 14 03:12:28.865247 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:28.865213    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:12:29.570638 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:29.570594    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:12:30.332969 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:30.332935    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:12:31.003196 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:31.003164    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:12:31.695948 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:31.695916    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:12:31.796558 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:31.796516    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:12:31.819135 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:12:31.819096    2564 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod45b8e89b_b746_4ff9_84b4_f11c8a7c5aeb.slice/crio-74976527d5ad5a4609da722e382b3e41a9f6b44b71032b97965a91026e9c9d5c\": RecentStats: unable to find data in memory cache]"
Sep 14 03:12:31.819135 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:12:31.819112    2564 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod45b8e89b_b746_4ff9_84b4_f11c8a7c5aeb.slice/crio-74976527d5ad5a4609da722e382b3e41a9f6b44b71032b97965a91026e9c9d5c\": RecentStats: unable to find data in memory cache]"
Sep 14 03:12:31.896717 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:31.896683    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:12:32.031931 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:32.031901    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:12:32.249148 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:32.249092    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:12:32.336902 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:32.336814    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:12:32.370712 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:32.370683    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2b32aa-7d7c8cddcd-crjgz"]
Sep 14 03:12:32.371157 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:32.371144    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f345be18-091e-48e6-bbf4-d4581301b800" containerName="main"
Sep 14 03:12:32.371215 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:32.371158    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f345be18-091e-48e6-bbf4-d4581301b800" containerName="main"
Sep 14 03:12:32.371215 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:32.371173    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f" containerName="main"
Sep 14 03:12:32.371215 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:32.371178    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f" containerName="main"
Sep 14 03:12:32.371215 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:32.371198    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2169752d-69ca-4313-aa66-b4c853c1b255" containerName="maas-api"
Sep 14 03:12:32.371215 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:32.371204    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2169752d-69ca-4313-aa66-b4c853c1b255" containerName="maas-api"
Sep 14 03:12:32.371376 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:32.371268    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c6142b7-9d3e-4b1c-97a8-ed957fea4c1f" containerName="main"
Sep 14 03:12:32.371376 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:32.371277    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f345be18-091e-48e6-bbf4-d4581301b800" containerName="main"
Sep 14 03:12:32.371376 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:32.371284    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2169752d-69ca-4313-aa66-b4c853c1b255" containerName="maas-api"
Sep 14 03:12:32.376065 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:32.376047    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2b32aa-7d7c8cddcd-crjgz"
Sep 14 03:12:32.379023 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:32.378987    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-2b32aa\""
Sep 14 03:12:32.379145 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:32.379111    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-2b32aa-dockercfg-4cwkg\""
Sep 14 03:12:32.385131 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:32.385108    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2b32aa-7d7c8cddcd-crjgz"]
Sep 14 03:12:32.539964 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:32.539931    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/681e44c9-583c-4b8e-909e-59f8efbc837a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-2b32aa-7d7c8cddcd-crjgz\" (UID: \"681e44c9-583c-4b8e-909e-59f8efbc837a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2b32aa-7d7c8cddcd-crjgz"
Sep 14 03:12:32.540126 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:32.539977    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p9ws\" (UniqueName: \"kubernetes.io/projected/681e44c9-583c-4b8e-909e-59f8efbc837a-kube-api-access-6p9ws\") pod \"payload-pre-processing-e2e-shared-a-w3-2b32aa-7d7c8cddcd-crjgz\" (UID: \"681e44c9-583c-4b8e-909e-59f8efbc837a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2b32aa-7d7c8cddcd-crjgz"
Sep 14 03:12:32.641666 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:32.641569    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p9ws\" (UniqueName: \"kubernetes.io/projected/681e44c9-583c-4b8e-909e-59f8efbc837a-kube-api-access-6p9ws\") pod \"payload-pre-processing-e2e-shared-a-w3-2b32aa-7d7c8cddcd-crjgz\" (UID: \"681e44c9-583c-4b8e-909e-59f8efbc837a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2b32aa-7d7c8cddcd-crjgz"
Sep 14 03:12:32.641828 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:32.641722    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/681e44c9-583c-4b8e-909e-59f8efbc837a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-2b32aa-7d7c8cddcd-crjgz\" (UID: \"681e44c9-583c-4b8e-909e-59f8efbc837a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2b32aa-7d7c8cddcd-crjgz"
Sep 14 03:12:32.642391 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:32.642372    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/681e44c9-583c-4b8e-909e-59f8efbc837a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-2b32aa-7d7c8cddcd-crjgz\" (UID: \"681e44c9-583c-4b8e-909e-59f8efbc837a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2b32aa-7d7c8cddcd-crjgz"
Sep 14 03:12:32.651205 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:32.651179    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p9ws\" (UniqueName: \"kubernetes.io/projected/681e44c9-583c-4b8e-909e-59f8efbc837a-kube-api-access-6p9ws\") pod \"payload-pre-processing-e2e-shared-a-w3-2b32aa-7d7c8cddcd-crjgz\" (UID: \"681e44c9-583c-4b8e-909e-59f8efbc837a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2b32aa-7d7c8cddcd-crjgz"
Sep 14 03:12:32.686865 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:32.686827    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2b32aa-7d7c8cddcd-crjgz"
Sep 14 03:12:32.965423 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:32.965380    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2b32aa-7d7c8cddcd-crjgz"]
Sep 14 03:12:32.967464 ip-10-0-141-4 kubenswrapper[2564]: W0914 03:12:32.967435    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod681e44c9_583c_4b8e_909e_59f8efbc837a.slice/crio-cfe937c5d4f6ace4888fd6c5d62f28771b2b9948b37106f7a5222725e2251534 WatchSource:0}: Error finding container cfe937c5d4f6ace4888fd6c5d62f28771b2b9948b37106f7a5222725e2251534: Status 404 returned error can't find the container with id cfe937c5d4f6ace4888fd6c5d62f28771b2b9948b37106f7a5222725e2251534
Sep 14 03:12:33.276043 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:33.276009    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2b32aa-7d7c8cddcd-crjgz" event={"ID":"681e44c9-583c-4b8e-909e-59f8efbc837a","Type":"ContainerStarted","Data":"48843065543e2bedc40c08daaa066935fcd81b437d765e6c7d250b6890f65370"}
Sep 14 03:12:33.276242 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:33.276070    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2b32aa-7d7c8cddcd-crjgz" event={"ID":"681e44c9-583c-4b8e-909e-59f8efbc837a","Type":"ContainerStarted","Data":"cfe937c5d4f6ace4888fd6c5d62f28771b2b9948b37106f7a5222725e2251534"}
Sep 14 03:12:33.276242 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:33.276109    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2b32aa-7d7c8cddcd-crjgz"
Sep 14 03:12:35.840036 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:35.840001    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:12:39.430719 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:39.430571    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:12:42.853614 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:42.853561    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2b32aa-7d7c8cddcd-crjgz" podStartSLOduration=10.85354283 podStartE2EDuration="10.85354283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:12:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:12:33.315757648 +0000 UTC m=+1198.696841860" watchObservedRunningTime="2026-09-14 03:12:42.85354283 +0000 UTC m=+1208.234627028"
Sep 14 03:12:42.854435 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:42.854398    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-a-7a5ebdfc-kserve-f7f5cb5c6-8clfl"]
Sep 14 03:12:42.881143 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:42.881072    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:12:44.283815 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:44.283784    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2b32aa-7d7c8cddcd-crjgz"
Sep 14 03:12:47.194152 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:47.194119    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-12ecd131/e2e-lifecycle-model-12ecd131-kserve-78949b58dc-x8c4c"]
Sep 14 03:12:47.197773 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:47.197756    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-12ecd131/e2e-lifecycle-model-12ecd131-kserve-78949b58dc-x8c4c"
Sep 14 03:12:47.205296 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:47.205274    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-12ecd131\"/\"openshift-service-ca.crt\""
Sep 14 03:12:47.205426 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:47.205280    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-12ecd131\"/\"default-dockercfg-9sxp8\""
Sep 14 03:12:47.205426 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:47.205359    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-12ecd131\"/\"e2e-lifecycle-model-12ecd131-kserve-self-signed-certs\""
Sep 14 03:12:47.206775 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:47.206757    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-12ecd131\"/\"kube-root-ca.crt\""
Sep 14 03:12:47.213835 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:47.213813    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-12ecd131/e2e-lifecycle-model-12ecd131-kserve-78949b58dc-x8c4c"]
Sep 14 03:12:47.267104 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:47.267063    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:12:47.293479 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:47.293449    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4d4050b-2492-4a29-89d3-53016a614381-home\") pod \"e2e-lifecycle-model-12ecd131-kserve-78949b58dc-x8c4c\" (UID: \"d4d4050b-2492-4a29-89d3-53016a614381\") " pod="ai-tenant-e2e-mt-12ecd131/e2e-lifecycle-model-12ecd131-kserve-78949b58dc-x8c4c"
Sep 14 03:12:47.293655 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:47.293498    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4d4050b-2492-4a29-89d3-53016a614381-dshm\") pod \"e2e-lifecycle-model-12ecd131-kserve-78949b58dc-x8c4c\" (UID: \"d4d4050b-2492-4a29-89d3-53016a614381\") " pod="ai-tenant-e2e-mt-12ecd131/e2e-lifecycle-model-12ecd131-kserve-78949b58dc-x8c4c"
Sep 14 03:12:47.293655 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:47.293527    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4d4050b-2492-4a29-89d3-53016a614381-model-cache\") pod \"e2e-lifecycle-model-12ecd131-kserve-78949b58dc-x8c4c\" (UID: \"d4d4050b-2492-4a29-89d3-53016a614381\") " pod="ai-tenant-e2e-mt-12ecd131/e2e-lifecycle-model-12ecd131-kserve-78949b58dc-x8c4c"
Sep 14 03:12:47.293655 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:47.293580    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4d4050b-2492-4a29-89d3-53016a614381-tmp-dir\") pod \"e2e-lifecycle-model-12ecd131-kserve-78949b58dc-x8c4c\" (UID: \"d4d4050b-2492-4a29-89d3-53016a614381\") " pod="ai-tenant-e2e-mt-12ecd131/e2e-lifecycle-model-12ecd131-kserve-78949b58dc-x8c4c"
Sep 14 03:12:47.293655 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:47.293597    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4d4050b-2492-4a29-89d3-53016a614381-tls-certs\") pod \"e2e-lifecycle-model-12ecd131-kserve-78949b58dc-x8c4c\" (UID: \"d4d4050b-2492-4a29-89d3-53016a614381\") " pod="ai-tenant-e2e-mt-12ecd131/e2e-lifecycle-model-12ecd131-kserve-78949b58dc-x8c4c"
Sep 14 03:12:47.293846 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:47.293676    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwkrq\" (UniqueName: \"kubernetes.io/projected/d4d4050b-2492-4a29-89d3-53016a614381-kube-api-access-lwkrq\") pod \"e2e-lifecycle-model-12ecd131-kserve-78949b58dc-x8c4c\" (UID: \"d4d4050b-2492-4a29-89d3-53016a614381\") " pod="ai-tenant-e2e-mt-12ecd131/e2e-lifecycle-model-12ecd131-kserve-78949b58dc-x8c4c"
Sep 14 03:12:47.395267 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:47.395237    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4d4050b-2492-4a29-89d3-53016a614381-model-cache\") pod \"e2e-lifecycle-model-12ecd131-kserve-78949b58dc-x8c4c\" (UID: \"d4d4050b-2492-4a29-89d3-53016a614381\") " pod="ai-tenant-e2e-mt-12ecd131/e2e-lifecycle-model-12ecd131-kserve-78949b58dc-x8c4c"
Sep 14 03:12:47.395482 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:47.395307    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4d4050b-2492-4a29-89d3-53016a614381-tmp-dir\") pod \"e2e-lifecycle-model-12ecd131-kserve-78949b58dc-x8c4c\" (UID: \"d4d4050b-2492-4a29-89d3-53016a614381\") " pod="ai-tenant-e2e-mt-12ecd131/e2e-lifecycle-model-12ecd131-kserve-78949b58dc-x8c4c"
Sep 14 03:12:47.395482 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:47.395328    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4d4050b-2492-4a29-89d3-53016a614381-tls-certs\") pod \"e2e-lifecycle-model-12ecd131-kserve-78949b58dc-x8c4c\" (UID: \"d4d4050b-2492-4a29-89d3-53016a614381\") " pod="ai-tenant-e2e-mt-12ecd131/e2e-lifecycle-model-12ecd131-kserve-78949b58dc-x8c4c"
Sep 14 03:12:47.395482 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:47.395365    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwkrq\" (UniqueName: \"kubernetes.io/projected/d4d4050b-2492-4a29-89d3-53016a614381-kube-api-access-lwkrq\") pod \"e2e-lifecycle-model-12ecd131-kserve-78949b58dc-x8c4c\" (UID: \"d4d4050b-2492-4a29-89d3-53016a614381\") " pod="ai-tenant-e2e-mt-12ecd131/e2e-lifecycle-model-12ecd131-kserve-78949b58dc-x8c4c"
Sep 14 03:12:47.395482 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:47.395431    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4d4050b-2492-4a29-89d3-53016a614381-home\") pod \"e2e-lifecycle-model-12ecd131-kserve-78949b58dc-x8c4c\" (UID: \"d4d4050b-2492-4a29-89d3-53016a614381\") " pod="ai-tenant-e2e-mt-12ecd131/e2e-lifecycle-model-12ecd131-kserve-78949b58dc-x8c4c"
Sep 14 03:12:47.395482 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:47.395478    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4d4050b-2492-4a29-89d3-53016a614381-dshm\") pod \"e2e-lifecycle-model-12ecd131-kserve-78949b58dc-x8c4c\" (UID: \"d4d4050b-2492-4a29-89d3-53016a614381\") " pod="ai-tenant-e2e-mt-12ecd131/e2e-lifecycle-model-12ecd131-kserve-78949b58dc-x8c4c"
Sep 14 03:12:47.395758 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:47.395670    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4d4050b-2492-4a29-89d3-53016a614381-tmp-dir\") pod \"e2e-lifecycle-model-12ecd131-kserve-78949b58dc-x8c4c\" (UID: \"d4d4050b-2492-4a29-89d3-53016a614381\") " pod="ai-tenant-e2e-mt-12ecd131/e2e-lifecycle-model-12ecd131-kserve-78949b58dc-x8c4c"
Sep 14 03:12:47.395758 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:47.395731    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4d4050b-2492-4a29-89d3-53016a614381-model-cache\") pod \"e2e-lifecycle-model-12ecd131-kserve-78949b58dc-x8c4c\" (UID: \"d4d4050b-2492-4a29-89d3-53016a614381\") " pod="ai-tenant-e2e-mt-12ecd131/e2e-lifecycle-model-12ecd131-kserve-78949b58dc-x8c4c"
Sep 14 03:12:47.396006 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:47.395978    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4d4050b-2492-4a29-89d3-53016a614381-home\") pod \"e2e-lifecycle-model-12ecd131-kserve-78949b58dc-x8c4c\" (UID: \"d4d4050b-2492-4a29-89d3-53016a614381\") " pod="ai-tenant-e2e-mt-12ecd131/e2e-lifecycle-model-12ecd131-kserve-78949b58dc-x8c4c"
Sep 14 03:12:47.397713 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:47.397691    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4d4050b-2492-4a29-89d3-53016a614381-dshm\") pod \"e2e-lifecycle-model-12ecd131-kserve-78949b58dc-x8c4c\" (UID: \"d4d4050b-2492-4a29-89d3-53016a614381\") " pod="ai-tenant-e2e-mt-12ecd131/e2e-lifecycle-model-12ecd131-kserve-78949b58dc-x8c4c"
Sep 14 03:12:47.398061 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:47.398041    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4d4050b-2492-4a29-89d3-53016a614381-tls-certs\") pod \"e2e-lifecycle-model-12ecd131-kserve-78949b58dc-x8c4c\" (UID: \"d4d4050b-2492-4a29-89d3-53016a614381\") " pod="ai-tenant-e2e-mt-12ecd131/e2e-lifecycle-model-12ecd131-kserve-78949b58dc-x8c4c"
Sep 14 03:12:47.404186 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:47.404165    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwkrq\" (UniqueName: \"kubernetes.io/projected/d4d4050b-2492-4a29-89d3-53016a614381-kube-api-access-lwkrq\") pod \"e2e-lifecycle-model-12ecd131-kserve-78949b58dc-x8c4c\" (UID: \"d4d4050b-2492-4a29-89d3-53016a614381\") " pod="ai-tenant-e2e-mt-12ecd131/e2e-lifecycle-model-12ecd131-kserve-78949b58dc-x8c4c"
Sep 14 03:12:47.507917 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:47.507836    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-12ecd131/e2e-lifecycle-model-12ecd131-kserve-78949b58dc-x8c4c"
Sep 14 03:12:47.786806 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:47.786776    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-12ecd131/e2e-lifecycle-model-12ecd131-kserve-78949b58dc-x8c4c"]
Sep 14 03:12:47.788189 ip-10-0-141-4 kubenswrapper[2564]: W0914 03:12:47.788160    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd4d4050b_2492_4a29_89d3_53016a614381.slice/crio-301d3c14d058e075b76472f4507705ac6e902fed5786fa885af364d81e528a90 WatchSource:0}: Error finding container 301d3c14d058e075b76472f4507705ac6e902fed5786fa885af364d81e528a90: Status 404 returned error can't find the container with id 301d3c14d058e075b76472f4507705ac6e902fed5786fa885af364d81e528a90
Sep 14 03:12:48.366669 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:48.366629    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-12ecd131/e2e-lifecycle-model-12ecd131-kserve-78949b58dc-x8c4c" event={"ID":"d4d4050b-2492-4a29-89d3-53016a614381","Type":"ContainerStarted","Data":"6d70db1562e293eb63d3743415741b948b402b151ddd79648cb97e15c123b5be"}
Sep 14 03:12:48.367059 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:48.366677    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-12ecd131/e2e-lifecycle-model-12ecd131-kserve-78949b58dc-x8c4c" event={"ID":"d4d4050b-2492-4a29-89d3-53016a614381","Type":"ContainerStarted","Data":"301d3c14d058e075b76472f4507705ac6e902fed5786fa885af364d81e528a90"}
Sep 14 03:12:48.387213 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:48.387162    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-12ecd131/e2e-lifecycle-model-12ecd131-kserve-78949b58dc-x8c4c" podStartSLOduration=1.387145381 podStartE2EDuration="1.387145381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:12:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:12:48.385670143 +0000 UTC m=+1213.766754354" watchObservedRunningTime="2026-09-14 03:12:48.387145381 +0000 UTC m=+1213.768229579"
Sep 14 03:12:52.778866 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:52.778833    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:12:54.133107 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:54.133065    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:12:54.820905 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:54.820872    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:12:55.515849 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:55.515817    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:12:56.360153 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:56.360118    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:12:57.146752 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:57.146713    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:12:57.508998 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:57.508906    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-12ecd131/e2e-lifecycle-model-12ecd131-kserve-78949b58dc-x8c4c"
Sep 14 03:12:57.508998 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:57.508963    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-12ecd131/e2e-lifecycle-model-12ecd131-kserve-78949b58dc-x8c4c"
Sep 14 03:12:57.521498 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:57.521466    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-12ecd131/e2e-lifecycle-model-12ecd131-kserve-78949b58dc-x8c4c"
Sep 14 03:12:57.677015 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:57.676979    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:12:57.873705 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:57.873662    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-a-7a5ebdfc-kserve-f7f5cb5c6-8clfl" podUID="5a524982-7e25-4967-9efd-121d9903d4e3" containerName="main" containerID="cri-o://c92fd205d10c49f6b883330493e3ef7251fce6e6feea592e4da9bf306ced030b" gracePeriod=45
Sep 14 03:12:58.008869 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:58.008812    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:12:58.109015 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:58.108968    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:12:58.133345 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:58.133277    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-a-7a5ebdfc-kserve-f7f5cb5c6-8clfl"
Sep 14 03:12:58.214194 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:58.214167    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nwlbt\" (UniqueName: \"kubernetes.io/projected/5a524982-7e25-4967-9efd-121d9903d4e3-kube-api-access-nwlbt\") pod \"5a524982-7e25-4967-9efd-121d9903d4e3\" (UID: \"5a524982-7e25-4967-9efd-121d9903d4e3\") "
Sep 14 03:12:58.214626 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:58.214212    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a524982-7e25-4967-9efd-121d9903d4e3-model-cache\") pod \"5a524982-7e25-4967-9efd-121d9903d4e3\" (UID: \"5a524982-7e25-4967-9efd-121d9903d4e3\") "
Sep 14 03:12:58.214626 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:58.214239    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a524982-7e25-4967-9efd-121d9903d4e3-tmp-dir\") pod \"5a524982-7e25-4967-9efd-121d9903d4e3\" (UID: \"5a524982-7e25-4967-9efd-121d9903d4e3\") "
Sep 14 03:12:58.214626 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:58.214271    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a524982-7e25-4967-9efd-121d9903d4e3-home\") pod \"5a524982-7e25-4967-9efd-121d9903d4e3\" (UID: \"5a524982-7e25-4967-9efd-121d9903d4e3\") "
Sep 14 03:12:58.214626 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:58.214380    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a524982-7e25-4967-9efd-121d9903d4e3-dshm\") pod \"5a524982-7e25-4967-9efd-121d9903d4e3\" (UID: \"5a524982-7e25-4967-9efd-121d9903d4e3\") "
Sep 14 03:12:58.214626 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:58.214424    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a524982-7e25-4967-9efd-121d9903d4e3-tls-certs\") pod \"5a524982-7e25-4967-9efd-121d9903d4e3\" (UID: \"5a524982-7e25-4967-9efd-121d9903d4e3\") "
Sep 14 03:12:58.214626 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:58.214494    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a524982-7e25-4967-9efd-121d9903d4e3-model-cache" (OuterVolumeSpecName: "model-cache") pod "5a524982-7e25-4967-9efd-121d9903d4e3" (UID: "5a524982-7e25-4967-9efd-121d9903d4e3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:12:58.214626 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:58.214584    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a524982-7e25-4967-9efd-121d9903d4e3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5a524982-7e25-4967-9efd-121d9903d4e3" (UID: "5a524982-7e25-4967-9efd-121d9903d4e3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:12:58.214626 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:58.214605    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a524982-7e25-4967-9efd-121d9903d4e3-home" (OuterVolumeSpecName: "home") pod "5a524982-7e25-4967-9efd-121d9903d4e3" (UID: "5a524982-7e25-4967-9efd-121d9903d4e3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:12:58.215045 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:58.214843    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a524982-7e25-4967-9efd-121d9903d4e3-model-cache\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:12:58.215045 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:58.214862    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a524982-7e25-4967-9efd-121d9903d4e3-tmp-dir\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:12:58.215045 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:58.214874    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a524982-7e25-4967-9efd-121d9903d4e3-home\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:12:58.216525 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:58.216495    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a524982-7e25-4967-9efd-121d9903d4e3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5a524982-7e25-4967-9efd-121d9903d4e3" (UID: "5a524982-7e25-4967-9efd-121d9903d4e3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 03:12:58.216840 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:58.216815    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a524982-7e25-4967-9efd-121d9903d4e3-dshm" (OuterVolumeSpecName: "dshm") pod "5a524982-7e25-4967-9efd-121d9903d4e3" (UID: "5a524982-7e25-4967-9efd-121d9903d4e3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:12:58.216840 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:58.216831    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a524982-7e25-4967-9efd-121d9903d4e3-kube-api-access-nwlbt" (OuterVolumeSpecName: "kube-api-access-nwlbt") pod "5a524982-7e25-4967-9efd-121d9903d4e3" (UID: "5a524982-7e25-4967-9efd-121d9903d4e3"). InnerVolumeSpecName "kube-api-access-nwlbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:12:58.295893 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:58.295848    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:12:58.316078 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:58.316044    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a524982-7e25-4967-9efd-121d9903d4e3-dshm\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:12:58.316078 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:58.316072    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a524982-7e25-4967-9efd-121d9903d4e3-tls-certs\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:12:58.316078 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:58.316081    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwlbt\" (UniqueName: \"kubernetes.io/projected/5a524982-7e25-4967-9efd-121d9903d4e3-kube-api-access-nwlbt\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:12:58.426424 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:58.426313    2564 generic.go:356] "Generic (PLEG): container finished" podID="5a524982-7e25-4967-9efd-121d9903d4e3" containerID="c92fd205d10c49f6b883330493e3ef7251fce6e6feea592e4da9bf306ced030b" exitCode=0
Sep 14 03:12:58.426424 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:58.426359    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-a-7a5ebdfc-kserve-f7f5cb5c6-8clfl"
Sep 14 03:12:58.426424 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:58.426389    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-a-7a5ebdfc-kserve-f7f5cb5c6-8clfl" event={"ID":"5a524982-7e25-4967-9efd-121d9903d4e3","Type":"ContainerDied","Data":"c92fd205d10c49f6b883330493e3ef7251fce6e6feea592e4da9bf306ced030b"}
Sep 14 03:12:58.426715 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:58.426465    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-a-7a5ebdfc-kserve-f7f5cb5c6-8clfl" event={"ID":"5a524982-7e25-4967-9efd-121d9903d4e3","Type":"ContainerDied","Data":"3538f76112c63ab7eb3ed8c9cda549a5bc32ec1c2f36d3f8c8b170298a5f3c3a"}
Sep 14 03:12:58.426715 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:58.426488    2564 scope.go:117] "RemoveContainer" containerID="c92fd205d10c49f6b883330493e3ef7251fce6e6feea592e4da9bf306ced030b"
Sep 14 03:12:58.439976 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:58.439951    2564 scope.go:117] "RemoveContainer" containerID="c92fd205d10c49f6b883330493e3ef7251fce6e6feea592e4da9bf306ced030b"
Sep 14 03:12:58.440131 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:58.440115    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-12ecd131/e2e-lifecycle-model-12ecd131-kserve-78949b58dc-x8c4c"
Sep 14 03:12:58.440304 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:12:58.440284    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c92fd205d10c49f6b883330493e3ef7251fce6e6feea592e4da9bf306ced030b\": container with ID starting with c92fd205d10c49f6b883330493e3ef7251fce6e6feea592e4da9bf306ced030b not found: ID does not exist" containerID="c92fd205d10c49f6b883330493e3ef7251fce6e6feea592e4da9bf306ced030b"
Sep 14 03:12:58.440380 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:58.440315    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c92fd205d10c49f6b883330493e3ef7251fce6e6feea592e4da9bf306ced030b"} err="failed to get container status \"c92fd205d10c49f6b883330493e3ef7251fce6e6feea592e4da9bf306ced030b\": rpc error: code = NotFound desc = could not find container \"c92fd205d10c49f6b883330493e3ef7251fce6e6feea592e4da9bf306ced030b\": container with ID starting with c92fd205d10c49f6b883330493e3ef7251fce6e6feea592e4da9bf306ced030b not found: ID does not exist"
Sep 14 03:12:58.450658 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:58.450628    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-a-7a5ebdfc-kserve-f7f5cb5c6-8clfl"]
Sep 14 03:12:58.455338 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:58.455313    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-9c9f1e/e2e-conflict-a-7a5ebdfc-kserve-f7f5cb5c6-8clfl"]
Sep 14 03:12:58.465395 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:12:58.465364    2564 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a524982_7e25_4967_9efd_121d9903d4e3.slice/crio-3538f76112c63ab7eb3ed8c9cda549a5bc32ec1c2f36d3f8c8b170298a5f3c3a\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a524982_7e25_4967_9efd_121d9903d4e3.slice\": RecentStats: unable to find data in memory cache]"
Sep 14 03:12:58.653104 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:58.653072    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:12:58.834782 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:58.834745    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:12:59.099190 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:12:59.099114    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a524982-7e25-4967-9efd-121d9903d4e3" path="/var/lib/kubelet/pods/5a524982-7e25-4967-9efd-121d9903d4e3/volumes"
Sep 14 03:13:01.147712 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:01.147682    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:13:01.822344 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:01.822306    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:13:01.918548 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:01.918511    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:13:02.826217 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:02.826181    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:13:03.799983 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:03.799949    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:13:04.859319 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:04.859278    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:13:05.536099 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:05.536062    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:13:06.330537 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:06.330501    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:13:07.047947 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:07.047917    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-177593-bc8cdccc7-9psb4"]
Sep 14 03:13:07.048416 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:07.048393    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a524982-7e25-4967-9efd-121d9903d4e3" containerName="main"
Sep 14 03:13:07.048469 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:07.048423    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a524982-7e25-4967-9efd-121d9903d4e3" containerName="main"
Sep 14 03:13:07.048506 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:07.048491    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a524982-7e25-4967-9efd-121d9903d4e3" containerName="main"
Sep 14 03:13:07.051655 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:07.051635    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-177593-bc8cdccc7-9psb4"
Sep 14 03:13:07.058717 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:13:07.058687    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"maas-api-e2e-shared-b-w3-177593-serving-cert\" is forbidden: User \"system:node:ip-10-0-141-4.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-141-4.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-177593-serving-cert\"" type="*v1.Secret"
Sep 14 03:13:07.059693 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:13:07.059666    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"maas-api-e2e-shared-b-w3-177593-bc8cdccc7-9psb4\" is forbidden: User \"system:node:ip-10-0-141-4.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-141-4.ec2.internal' and this object" podUID="91302c2a-0745-436a-a6ec-4eda74bd18d7" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-177593-bc8cdccc7-9psb4"
Sep 14 03:13:07.086252 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:07.086226    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-177593-bc8cdccc7-9psb4"]
Sep 14 03:13:07.101113 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:07.101081    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:13:07.202418 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:07.202374    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:13:07.206571 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:07.206549    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91302c2a-0745-436a-a6ec-4eda74bd18d7-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-177593-bc8cdccc7-9psb4\" (UID: \"91302c2a-0745-436a-a6ec-4eda74bd18d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-177593-bc8cdccc7-9psb4"
Sep 14 03:13:07.206639 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:07.206590    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t4bs\" (UniqueName: \"kubernetes.io/projected/91302c2a-0745-436a-a6ec-4eda74bd18d7-kube-api-access-9t4bs\") pod \"maas-api-e2e-shared-b-w3-177593-bc8cdccc7-9psb4\" (UID: \"91302c2a-0745-436a-a6ec-4eda74bd18d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-177593-bc8cdccc7-9psb4"
Sep 14 03:13:07.206681 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:07.206673    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91302c2a-0745-436a-a6ec-4eda74bd18d7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-177593-bc8cdccc7-9psb4\" (UID: \"91302c2a-0745-436a-a6ec-4eda74bd18d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-177593-bc8cdccc7-9psb4"
Sep 14 03:13:07.206717 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:07.206701    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91302c2a-0745-436a-a6ec-4eda74bd18d7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-177593-bc8cdccc7-9psb4\" (UID: \"91302c2a-0745-436a-a6ec-4eda74bd18d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-177593-bc8cdccc7-9psb4"
Sep 14 03:13:07.303512 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:07.303415    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:13:07.307537 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:07.307518    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91302c2a-0745-436a-a6ec-4eda74bd18d7-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-177593-bc8cdccc7-9psb4\" (UID: \"91302c2a-0745-436a-a6ec-4eda74bd18d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-177593-bc8cdccc7-9psb4"
Sep 14 03:13:07.307606 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:07.307558    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t4bs\" (UniqueName: \"kubernetes.io/projected/91302c2a-0745-436a-a6ec-4eda74bd18d7-kube-api-access-9t4bs\") pod \"maas-api-e2e-shared-b-w3-177593-bc8cdccc7-9psb4\" (UID: \"91302c2a-0745-436a-a6ec-4eda74bd18d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-177593-bc8cdccc7-9psb4"
Sep 14 03:13:07.307661 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:07.307605    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91302c2a-0745-436a-a6ec-4eda74bd18d7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-177593-bc8cdccc7-9psb4\" (UID: \"91302c2a-0745-436a-a6ec-4eda74bd18d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-177593-bc8cdccc7-9psb4"
Sep 14 03:13:07.307661 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:07.307631    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91302c2a-0745-436a-a6ec-4eda74bd18d7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-177593-bc8cdccc7-9psb4\" (UID: \"91302c2a-0745-436a-a6ec-4eda74bd18d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-177593-bc8cdccc7-9psb4"
Sep 14 03:13:07.308299 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:07.308278    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91302c2a-0745-436a-a6ec-4eda74bd18d7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-177593-bc8cdccc7-9psb4\" (UID: \"91302c2a-0745-436a-a6ec-4eda74bd18d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-177593-bc8cdccc7-9psb4"
Sep 14 03:13:07.309980 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:07.309957    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91302c2a-0745-436a-a6ec-4eda74bd18d7-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-177593-bc8cdccc7-9psb4\" (UID: \"91302c2a-0745-436a-a6ec-4eda74bd18d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-177593-bc8cdccc7-9psb4"
Sep 14 03:13:07.317264 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:07.317242    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t4bs\" (UniqueName: \"kubernetes.io/projected/91302c2a-0745-436a-a6ec-4eda74bd18d7-kube-api-access-9t4bs\") pod \"maas-api-e2e-shared-b-w3-177593-bc8cdccc7-9psb4\" (UID: \"91302c2a-0745-436a-a6ec-4eda74bd18d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-177593-bc8cdccc7-9psb4"
Sep 14 03:13:07.585331 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:07.585248    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:13:07.693072 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:07.693037    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:13:07.757336 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:07.757306    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-177593-b77c9949d-8gtv4"]
Sep 14 03:13:07.761364 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:07.761344    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-177593-b77c9949d-8gtv4"
Sep 14 03:13:07.764955 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:07.764928    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-177593-dockercfg-4vp8m\""
Sep 14 03:13:07.765461 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:07.765445    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-177593\""
Sep 14 03:13:07.777859 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:07.777838    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-177593-b77c9949d-8gtv4"]
Sep 14 03:13:07.913333 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:07.913241    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8j5g\" (UniqueName: \"kubernetes.io/projected/6ea449a0-fb86-44c4-a89f-e8314595862d-kube-api-access-l8j5g\") pod \"payload-pre-processing-e2e-shared-b-w3-177593-b77c9949d-8gtv4\" (UID: \"6ea449a0-fb86-44c4-a89f-e8314595862d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-177593-b77c9949d-8gtv4"
Sep 14 03:13:07.913528 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:07.913388    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ea449a0-fb86-44c4-a89f-e8314595862d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-177593-b77c9949d-8gtv4\" (UID: \"6ea449a0-fb86-44c4-a89f-e8314595862d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-177593-b77c9949d-8gtv4"
Sep 14 03:13:08.013964 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:08.013919    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ea449a0-fb86-44c4-a89f-e8314595862d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-177593-b77c9949d-8gtv4\" (UID: \"6ea449a0-fb86-44c4-a89f-e8314595862d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-177593-b77c9949d-8gtv4"
Sep 14 03:13:08.014141 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:08.014003    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8j5g\" (UniqueName: \"kubernetes.io/projected/6ea449a0-fb86-44c4-a89f-e8314595862d-kube-api-access-l8j5g\") pod \"payload-pre-processing-e2e-shared-b-w3-177593-b77c9949d-8gtv4\" (UID: \"6ea449a0-fb86-44c4-a89f-e8314595862d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-177593-b77c9949d-8gtv4"
Sep 14 03:13:08.014644 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:08.014624    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ea449a0-fb86-44c4-a89f-e8314595862d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-177593-b77c9949d-8gtv4\" (UID: \"6ea449a0-fb86-44c4-a89f-e8314595862d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-177593-b77c9949d-8gtv4"
Sep 14 03:13:08.022280 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:08.022257    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8j5g\" (UniqueName: \"kubernetes.io/projected/6ea449a0-fb86-44c4-a89f-e8314595862d-kube-api-access-l8j5g\") pod \"payload-pre-processing-e2e-shared-b-w3-177593-b77c9949d-8gtv4\" (UID: \"6ea449a0-fb86-44c4-a89f-e8314595862d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-177593-b77c9949d-8gtv4"
Sep 14 03:13:08.072983 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:08.072954    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-177593-b77c9949d-8gtv4"
Sep 14 03:13:08.308492 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:13:08.308457    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-177593-serving-cert: failed to sync secret cache: timed out waiting for the condition
Sep 14 03:13:08.308663 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:13:08.308553    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91302c2a-0745-436a-a6ec-4eda74bd18d7-maas-api-tls podName:91302c2a-0745-436a-a6ec-4eda74bd18d7 nodeName:}" failed. No retries permitted until 2026-09-14 03:13:08.808536248 +0000 UTC m=+1234.189620425 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/91302c2a-0745-436a-a6ec-4eda74bd18d7-maas-api-tls") pod "maas-api-e2e-shared-b-w3-177593-bc8cdccc7-9psb4" (UID: "91302c2a-0745-436a-a6ec-4eda74bd18d7") : failed to sync secret cache: timed out waiting for the condition
Sep 14 03:13:08.352666 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:08.352637    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-177593-b77c9949d-8gtv4"]
Sep 14 03:13:08.354973 ip-10-0-141-4 kubenswrapper[2564]: W0914 03:13:08.354928    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ea449a0_fb86_44c4_a89f_e8314595862d.slice/crio-3834edebcf987751c7d75c67d8bb9bd888adb4bc90be4b5ba343f45b4a8e3949 WatchSource:0}: Error finding container 3834edebcf987751c7d75c67d8bb9bd888adb4bc90be4b5ba343f45b4a8e3949: Status 404 returned error can't find the container with id 3834edebcf987751c7d75c67d8bb9bd888adb4bc90be4b5ba343f45b4a8e3949
Sep 14 03:13:08.456791 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:08.456758    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-177593-serving-cert\""
Sep 14 03:13:08.486783 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:08.486744    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-177593-b77c9949d-8gtv4" event={"ID":"6ea449a0-fb86-44c4-a89f-e8314595862d","Type":"ContainerStarted","Data":"e3c33d0c54f6fd8a07e1405224d784d92ac1ae6be692b344def4e18764fdb5fc"}
Sep 14 03:13:08.486966 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:08.486801    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-177593-b77c9949d-8gtv4" event={"ID":"6ea449a0-fb86-44c4-a89f-e8314595862d","Type":"ContainerStarted","Data":"3834edebcf987751c7d75c67d8bb9bd888adb4bc90be4b5ba343f45b4a8e3949"}
Sep 14 03:13:08.486966 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:08.486863    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-177593-b77c9949d-8gtv4"
Sep 14 03:13:08.504557 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:08.504509    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-177593-b77c9949d-8gtv4" podStartSLOduration=1.5044948740000001 podStartE2EDuration="1.504494874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:13:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:13:08.504047381 +0000 UTC m=+1233.885131580" watchObservedRunningTime="2026-09-14 03:13:08.504494874 +0000 UTC m=+1233.885579073"
Sep 14 03:13:08.823118 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:08.823078    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91302c2a-0745-436a-a6ec-4eda74bd18d7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-177593-bc8cdccc7-9psb4\" (UID: \"91302c2a-0745-436a-a6ec-4eda74bd18d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-177593-bc8cdccc7-9psb4"
Sep 14 03:13:08.825623 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:08.825602    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91302c2a-0745-436a-a6ec-4eda74bd18d7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-177593-bc8cdccc7-9psb4\" (UID: \"91302c2a-0745-436a-a6ec-4eda74bd18d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-177593-bc8cdccc7-9psb4"
Sep 14 03:13:08.862305 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:08.862273    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-177593-bc8cdccc7-9psb4"
Sep 14 03:13:09.139208 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:09.139181    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-177593-bc8cdccc7-9psb4"]
Sep 14 03:13:09.141069 ip-10-0-141-4 kubenswrapper[2564]: W0914 03:13:09.141041    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91302c2a_0745_436a_a6ec_4eda74bd18d7.slice/crio-706ece2dbfb65c2a4cccb776b9a20e78f4c4576cf8ddbd3333bbcb9f193f3f44 WatchSource:0}: Error finding container 706ece2dbfb65c2a4cccb776b9a20e78f4c4576cf8ddbd3333bbcb9f193f3f44: Status 404 returned error can't find the container with id 706ece2dbfb65c2a4cccb776b9a20e78f4c4576cf8ddbd3333bbcb9f193f3f44
Sep 14 03:13:09.493321 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:09.493250    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-177593-bc8cdccc7-9psb4" event={"ID":"91302c2a-0745-436a-a6ec-4eda74bd18d7","Type":"ContainerStarted","Data":"706ece2dbfb65c2a4cccb776b9a20e78f4c4576cf8ddbd3333bbcb9f193f3f44"}
Sep 14 03:13:09.938904 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:09.938870    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-9c9f1e/distinct-w4-9c9f1e-kserve-579d94566b-tgjjk"]
Sep 14 03:13:09.946620 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:09.944801    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9c9f1e/distinct-w4-9c9f1e-kserve-579d94566b-tgjjk"
Sep 14 03:13:09.948973 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:09.948944    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-9c9f1e\"/\"default-dockercfg-5qjfv\""
Sep 14 03:13:09.949204 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:09.949004    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-9c9f1e\"/\"openshift-service-ca.crt\""
Sep 14 03:13:09.949204 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:09.949058    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-9c9f1e\"/\"distinct-w4-9c9f1e-kserve-self-signed-certs\""
Sep 14 03:13:09.949204 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:09.949184    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-9c9f1e\"/\"kube-root-ca.crt\""
Sep 14 03:13:09.952016 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:09.951997    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-9c9f1e/distinct-w4-9c9f1e-kserve-579d94566b-tgjjk"]
Sep 14 03:13:10.035833 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:10.035800    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbxvp\" (UniqueName: \"kubernetes.io/projected/09fd993a-f4ed-44c2-983d-73f7501e6b9b-kube-api-access-pbxvp\") pod \"distinct-w4-9c9f1e-kserve-579d94566b-tgjjk\" (UID: \"09fd993a-f4ed-44c2-983d-73f7501e6b9b\") " pod="e2e-models-e2e-worker-w4-9c9f1e/distinct-w4-9c9f1e-kserve-579d94566b-tgjjk"
Sep 14 03:13:10.035996 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:10.035847    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09fd993a-f4ed-44c2-983d-73f7501e6b9b-home\") pod \"distinct-w4-9c9f1e-kserve-579d94566b-tgjjk\" (UID: \"09fd993a-f4ed-44c2-983d-73f7501e6b9b\") " pod="e2e-models-e2e-worker-w4-9c9f1e/distinct-w4-9c9f1e-kserve-579d94566b-tgjjk"
Sep 14 03:13:10.035996 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:10.035955    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09fd993a-f4ed-44c2-983d-73f7501e6b9b-model-cache\") pod \"distinct-w4-9c9f1e-kserve-579d94566b-tgjjk\" (UID: \"09fd993a-f4ed-44c2-983d-73f7501e6b9b\") " pod="e2e-models-e2e-worker-w4-9c9f1e/distinct-w4-9c9f1e-kserve-579d94566b-tgjjk"
Sep 14 03:13:10.035996 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:10.035995    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09fd993a-f4ed-44c2-983d-73f7501e6b9b-tls-certs\") pod \"distinct-w4-9c9f1e-kserve-579d94566b-tgjjk\" (UID: \"09fd993a-f4ed-44c2-983d-73f7501e6b9b\") " pod="e2e-models-e2e-worker-w4-9c9f1e/distinct-w4-9c9f1e-kserve-579d94566b-tgjjk"
Sep 14 03:13:10.036128 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:10.036051    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09fd993a-f4ed-44c2-983d-73f7501e6b9b-dshm\") pod \"distinct-w4-9c9f1e-kserve-579d94566b-tgjjk\" (UID: \"09fd993a-f4ed-44c2-983d-73f7501e6b9b\") " pod="e2e-models-e2e-worker-w4-9c9f1e/distinct-w4-9c9f1e-kserve-579d94566b-tgjjk"
Sep 14 03:13:10.036128 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:10.036104    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09fd993a-f4ed-44c2-983d-73f7501e6b9b-tmp-dir\") pod \"distinct-w4-9c9f1e-kserve-579d94566b-tgjjk\" (UID: \"09fd993a-f4ed-44c2-983d-73f7501e6b9b\") " pod="e2e-models-e2e-worker-w4-9c9f1e/distinct-w4-9c9f1e-kserve-579d94566b-tgjjk"
Sep 14 03:13:10.137160 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:10.137122    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09fd993a-f4ed-44c2-983d-73f7501e6b9b-tmp-dir\") pod \"distinct-w4-9c9f1e-kserve-579d94566b-tgjjk\" (UID: \"09fd993a-f4ed-44c2-983d-73f7501e6b9b\") " pod="e2e-models-e2e-worker-w4-9c9f1e/distinct-w4-9c9f1e-kserve-579d94566b-tgjjk"
Sep 14 03:13:10.137324 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:10.137185    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbxvp\" (UniqueName: \"kubernetes.io/projected/09fd993a-f4ed-44c2-983d-73f7501e6b9b-kube-api-access-pbxvp\") pod \"distinct-w4-9c9f1e-kserve-579d94566b-tgjjk\" (UID: \"09fd993a-f4ed-44c2-983d-73f7501e6b9b\") " pod="e2e-models-e2e-worker-w4-9c9f1e/distinct-w4-9c9f1e-kserve-579d94566b-tgjjk"
Sep 14 03:13:10.137324 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:10.137214    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09fd993a-f4ed-44c2-983d-73f7501e6b9b-home\") pod \"distinct-w4-9c9f1e-kserve-579d94566b-tgjjk\" (UID: \"09fd993a-f4ed-44c2-983d-73f7501e6b9b\") " pod="e2e-models-e2e-worker-w4-9c9f1e/distinct-w4-9c9f1e-kserve-579d94566b-tgjjk"
Sep 14 03:13:10.137324 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:10.137280    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09fd993a-f4ed-44c2-983d-73f7501e6b9b-model-cache\") pod \"distinct-w4-9c9f1e-kserve-579d94566b-tgjjk\" (UID: \"09fd993a-f4ed-44c2-983d-73f7501e6b9b\") " pod="e2e-models-e2e-worker-w4-9c9f1e/distinct-w4-9c9f1e-kserve-579d94566b-tgjjk"
Sep 14 03:13:10.137324 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:10.137314    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09fd993a-f4ed-44c2-983d-73f7501e6b9b-tls-certs\") pod \"distinct-w4-9c9f1e-kserve-579d94566b-tgjjk\" (UID: \"09fd993a-f4ed-44c2-983d-73f7501e6b9b\") " pod="e2e-models-e2e-worker-w4-9c9f1e/distinct-w4-9c9f1e-kserve-579d94566b-tgjjk"
Sep 14 03:13:10.137593 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:10.137366    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09fd993a-f4ed-44c2-983d-73f7501e6b9b-dshm\") pod \"distinct-w4-9c9f1e-kserve-579d94566b-tgjjk\" (UID: \"09fd993a-f4ed-44c2-983d-73f7501e6b9b\") " pod="e2e-models-e2e-worker-w4-9c9f1e/distinct-w4-9c9f1e-kserve-579d94566b-tgjjk"
Sep 14 03:13:10.137593 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:10.137573    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09fd993a-f4ed-44c2-983d-73f7501e6b9b-tmp-dir\") pod \"distinct-w4-9c9f1e-kserve-579d94566b-tgjjk\" (UID: \"09fd993a-f4ed-44c2-983d-73f7501e6b9b\") " pod="e2e-models-e2e-worker-w4-9c9f1e/distinct-w4-9c9f1e-kserve-579d94566b-tgjjk"
Sep 14 03:13:10.137702 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:10.137653    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09fd993a-f4ed-44c2-983d-73f7501e6b9b-home\") pod \"distinct-w4-9c9f1e-kserve-579d94566b-tgjjk\" (UID: \"09fd993a-f4ed-44c2-983d-73f7501e6b9b\") " pod="e2e-models-e2e-worker-w4-9c9f1e/distinct-w4-9c9f1e-kserve-579d94566b-tgjjk"
Sep 14 03:13:10.137702 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:10.137677    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09fd993a-f4ed-44c2-983d-73f7501e6b9b-model-cache\") pod \"distinct-w4-9c9f1e-kserve-579d94566b-tgjjk\" (UID: \"09fd993a-f4ed-44c2-983d-73f7501e6b9b\") " pod="e2e-models-e2e-worker-w4-9c9f1e/distinct-w4-9c9f1e-kserve-579d94566b-tgjjk"
Sep 14 03:13:10.139664 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:10.139640    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09fd993a-f4ed-44c2-983d-73f7501e6b9b-dshm\") pod \"distinct-w4-9c9f1e-kserve-579d94566b-tgjjk\" (UID: \"09fd993a-f4ed-44c2-983d-73f7501e6b9b\") " pod="e2e-models-e2e-worker-w4-9c9f1e/distinct-w4-9c9f1e-kserve-579d94566b-tgjjk"
Sep 14 03:13:10.139947 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:10.139928    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09fd993a-f4ed-44c2-983d-73f7501e6b9b-tls-certs\") pod \"distinct-w4-9c9f1e-kserve-579d94566b-tgjjk\" (UID: \"09fd993a-f4ed-44c2-983d-73f7501e6b9b\") " pod="e2e-models-e2e-worker-w4-9c9f1e/distinct-w4-9c9f1e-kserve-579d94566b-tgjjk"
Sep 14 03:13:10.145845 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:10.145827    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbxvp\" (UniqueName: \"kubernetes.io/projected/09fd993a-f4ed-44c2-983d-73f7501e6b9b-kube-api-access-pbxvp\") pod \"distinct-w4-9c9f1e-kserve-579d94566b-tgjjk\" (UID: \"09fd993a-f4ed-44c2-983d-73f7501e6b9b\") " pod="e2e-models-e2e-worker-w4-9c9f1e/distinct-w4-9c9f1e-kserve-579d94566b-tgjjk"
Sep 14 03:13:10.256773 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:10.256688    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9c9f1e/distinct-w4-9c9f1e-kserve-579d94566b-tgjjk"
Sep 14 03:13:10.449572 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:10.449544    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:13:10.500025 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:10.499995    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-177593-bc8cdccc7-9psb4" event={"ID":"91302c2a-0745-436a-a6ec-4eda74bd18d7","Type":"ContainerStarted","Data":"f08b9a1f8602de5e829e8013800e0db0dc22d29c19e70be0a467cd79a6afa32f"}
Sep 14 03:13:10.500147 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:10.500124    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-177593-bc8cdccc7-9psb4"
Sep 14 03:13:10.520206 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:10.520111    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-177593-bc8cdccc7-9psb4" podStartSLOduration=4.161762498 podStartE2EDuration="4.520094572s" totalImagesPullingTime="358.332074ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:13:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:13:10.517593942 +0000 UTC m=+1235.898678140" watchObservedRunningTime="2026-09-14 03:13:10.520094572 +0000 UTC m=+1235.901178773"
Sep 14 03:13:10.532729 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:10.532700    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-9c9f1e/distinct-w4-9c9f1e-kserve-579d94566b-tgjjk"]
Sep 14 03:13:10.534235 ip-10-0-141-4 kubenswrapper[2564]: W0914 03:13:10.534207    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod09fd993a_f4ed_44c2_983d_73f7501e6b9b.slice/crio-cd1c084692167f0d8e5673872b74af5d982f90226836b0a6a02a72b9e7599786 WatchSource:0}: Error finding container cd1c084692167f0d8e5673872b74af5d982f90226836b0a6a02a72b9e7599786: Status 404 returned error can't find the container with id cd1c084692167f0d8e5673872b74af5d982f90226836b0a6a02a72b9e7599786
Sep 14 03:13:11.506958 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:11.506920    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9c9f1e/distinct-w4-9c9f1e-kserve-579d94566b-tgjjk" event={"ID":"09fd993a-f4ed-44c2-983d-73f7501e6b9b","Type":"ContainerStarted","Data":"3bafbacfe580478b16a366fa151010c9508709cd6d16a8611ba4b826e7e89a5b"}
Sep 14 03:13:11.507328 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:11.506967    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9c9f1e/distinct-w4-9c9f1e-kserve-579d94566b-tgjjk" event={"ID":"09fd993a-f4ed-44c2-983d-73f7501e6b9b","Type":"ContainerStarted","Data":"cd1c084692167f0d8e5673872b74af5d982f90226836b0a6a02a72b9e7599786"}
Sep 14 03:13:11.526321 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:11.526277    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-9c9f1e/distinct-w4-9c9f1e-kserve-579d94566b-tgjjk" podStartSLOduration=2.526261248 podStartE2EDuration="2.526261248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:13:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:13:11.524595178 +0000 UTC m=+1236.905679378" watchObservedRunningTime="2026-09-14 03:13:11.526261248 +0000 UTC m=+1236.907345447"
Sep 14 03:13:13.907233 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:13.907194    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:13:13.971248 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:13.971217    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:13:16.512744 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:16.512715    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-177593-bc8cdccc7-9psb4"
Sep 14 03:13:19.495699 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:19.495657    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-177593-b77c9949d-8gtv4"
Sep 14 03:13:20.257127 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:20.257082    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9c9f1e/distinct-w4-9c9f1e-kserve-579d94566b-tgjjk"
Sep 14 03:13:20.257307 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:20.257152    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-9c9f1e/distinct-w4-9c9f1e-kserve-579d94566b-tgjjk"
Sep 14 03:13:20.270150 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:20.270120    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-9c9f1e/distinct-w4-9c9f1e-kserve-579d94566b-tgjjk"
Sep 14 03:13:20.575663 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:20.575627    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9c9f1e/distinct-w4-9c9f1e-kserve-579d94566b-tgjjk"
Sep 14 03:13:29.032744 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:29.032649    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:13:29.739197 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:29.739160    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:13:30.493974 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:30.493937    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:13:31.139074 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:31.139032    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:13:31.894640 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:31.894602    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:13:31.994763 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:31.994719    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:13:32.873067 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:32.873019    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:13:33.731795 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:33.731751    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:13:34.538073 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:34.538036    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:13:47.538271 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:47.538232    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:13:48.225603 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:48.225567    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:13:49.063692 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:49.063655    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:13:49.845899 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:49.845862    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:13:50.596862 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:50.596827    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:13:51.412048 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:51.412005    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:13:51.995525 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:51.995484    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:13:52.132194 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:52.132158    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:13:52.850352 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:52.850288    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:13:53.643986 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:13:53.643953    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:14:02.792872 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:02.792836    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-12ecd131-78cd7db6f7-6rt5l"]
Sep 14 03:14:02.793523 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:02.793059    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-12ecd131-78cd7db6f7-6rt5l" podUID="dffb2be6-e854-460c-9af8-005afdad9108" containerName="maas-api" containerID="cri-o://8cc98ac423fae5fc6857c32e2461087a27fd7d6ca25f49e93c110582179bb8f9" gracePeriod=30
Sep 14 03:14:02.829385 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:02.829350    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:14:02.911758 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:02.911731    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-12ecd131-7f655f44fd-cm47f"]
Sep 14 03:14:02.911973 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:02.911933    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-12ecd131-7f655f44fd-cm47f" podUID="238e6109-9a10-45a4-9014-cfba173627c7" containerName="payload-pre-processing" containerID="cri-o://d3b1c947d4f556f7dad261924a2768f8d9ad8bad987af382fcc319906aacd079" gracePeriod=30
Sep 14 03:14:02.972446 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:02.972386    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:14:03.071611 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:03.071583    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-12ecd131-78cd7db6f7-6rt5l"
Sep 14 03:14:03.179900 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:03.179866    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:14:03.184054 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:03.184032    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-12ecd131-7f655f44fd-cm47f"
Sep 14 03:14:03.197600 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:03.196745    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dffb2be6-e854-460c-9af8-005afdad9108-metrics-certs\") pod \"dffb2be6-e854-460c-9af8-005afdad9108\" (UID: \"dffb2be6-e854-460c-9af8-005afdad9108\") "
Sep 14 03:14:03.197600 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:03.196864    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dffb2be6-e854-460c-9af8-005afdad9108-openshift-service-ca\") pod \"dffb2be6-e854-460c-9af8-005afdad9108\" (UID: \"dffb2be6-e854-460c-9af8-005afdad9108\") "
Sep 14 03:14:03.197600 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:03.196930    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dffb2be6-e854-460c-9af8-005afdad9108-maas-api-tls\") pod \"dffb2be6-e854-460c-9af8-005afdad9108\" (UID: \"dffb2be6-e854-460c-9af8-005afdad9108\") "
Sep 14 03:14:03.197600 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:03.196961    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zx6b2\" (UniqueName: \"kubernetes.io/projected/dffb2be6-e854-460c-9af8-005afdad9108-kube-api-access-zx6b2\") pod \"dffb2be6-e854-460c-9af8-005afdad9108\" (UID: \"dffb2be6-e854-460c-9af8-005afdad9108\") "
Sep 14 03:14:03.197600 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:03.197244    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dffb2be6-e854-460c-9af8-005afdad9108-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "dffb2be6-e854-460c-9af8-005afdad9108" (UID: "dffb2be6-e854-460c-9af8-005afdad9108"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 03:14:03.197600 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:03.197442    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dffb2be6-e854-460c-9af8-005afdad9108-openshift-service-ca\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:14:03.199422 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:03.199374    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dffb2be6-e854-460c-9af8-005afdad9108-kube-api-access-zx6b2" (OuterVolumeSpecName: "kube-api-access-zx6b2") pod "dffb2be6-e854-460c-9af8-005afdad9108" (UID: "dffb2be6-e854-460c-9af8-005afdad9108"). InnerVolumeSpecName "kube-api-access-zx6b2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:14:03.199552 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:03.199452    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dffb2be6-e854-460c-9af8-005afdad9108-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "dffb2be6-e854-460c-9af8-005afdad9108" (UID: "dffb2be6-e854-460c-9af8-005afdad9108"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 03:14:03.199552 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:03.199469    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dffb2be6-e854-460c-9af8-005afdad9108-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "dffb2be6-e854-460c-9af8-005afdad9108" (UID: "dffb2be6-e854-460c-9af8-005afdad9108"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 03:14:03.297972 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:03.297938    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/238e6109-9a10-45a4-9014-cfba173627c7-plugins-config-volume\") pod \"238e6109-9a10-45a4-9014-cfba173627c7\" (UID: \"238e6109-9a10-45a4-9014-cfba173627c7\") "
Sep 14 03:14:03.298154 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:03.298005    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-64j45\" (UniqueName: \"kubernetes.io/projected/238e6109-9a10-45a4-9014-cfba173627c7-kube-api-access-64j45\") pod \"238e6109-9a10-45a4-9014-cfba173627c7\" (UID: \"238e6109-9a10-45a4-9014-cfba173627c7\") "
Sep 14 03:14:03.298331 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:03.298306    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/238e6109-9a10-45a4-9014-cfba173627c7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "238e6109-9a10-45a4-9014-cfba173627c7" (UID: "238e6109-9a10-45a4-9014-cfba173627c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 03:14:03.298382 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:03.298338    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dffb2be6-e854-460c-9af8-005afdad9108-maas-api-tls\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:14:03.298382 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:03.298360    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zx6b2\" (UniqueName: \"kubernetes.io/projected/dffb2be6-e854-460c-9af8-005afdad9108-kube-api-access-zx6b2\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:14:03.298382 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:03.298376    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dffb2be6-e854-460c-9af8-005afdad9108-metrics-certs\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:14:03.300195 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:03.300174    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/238e6109-9a10-45a4-9014-cfba173627c7-kube-api-access-64j45" (OuterVolumeSpecName: "kube-api-access-64j45") pod "238e6109-9a10-45a4-9014-cfba173627c7" (UID: "238e6109-9a10-45a4-9014-cfba173627c7"). InnerVolumeSpecName "kube-api-access-64j45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:14:03.347740 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:03.347617    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:14:03.399065 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:03.399032    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/238e6109-9a10-45a4-9014-cfba173627c7-plugins-config-volume\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:14:03.399065 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:03.399062    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64j45\" (UniqueName: \"kubernetes.io/projected/238e6109-9a10-45a4-9014-cfba173627c7-kube-api-access-64j45\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:14:03.482893 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:03.482859    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:14:03.812278 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:03.812247    2564 generic.go:356] "Generic (PLEG): container finished" podID="238e6109-9a10-45a4-9014-cfba173627c7" containerID="d3b1c947d4f556f7dad261924a2768f8d9ad8bad987af382fcc319906aacd079" exitCode=0
Sep 14 03:14:03.812625 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:03.812294    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-12ecd131-7f655f44fd-cm47f"
Sep 14 03:14:03.812625 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:03.812329    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-12ecd131-7f655f44fd-cm47f" event={"ID":"238e6109-9a10-45a4-9014-cfba173627c7","Type":"ContainerDied","Data":"d3b1c947d4f556f7dad261924a2768f8d9ad8bad987af382fcc319906aacd079"}
Sep 14 03:14:03.812625 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:03.812423    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-12ecd131-7f655f44fd-cm47f" event={"ID":"238e6109-9a10-45a4-9014-cfba173627c7","Type":"ContainerDied","Data":"04e687f185208b0f34a82a74dfc2b52a439c573ab0846a2152aaf65fba805c2a"}
Sep 14 03:14:03.812625 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:03.812455    2564 scope.go:117] "RemoveContainer" containerID="d3b1c947d4f556f7dad261924a2768f8d9ad8bad987af382fcc319906aacd079"
Sep 14 03:14:03.814164 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:03.814144    2564 generic.go:356] "Generic (PLEG): container finished" podID="dffb2be6-e854-460c-9af8-005afdad9108" containerID="8cc98ac423fae5fc6857c32e2461087a27fd7d6ca25f49e93c110582179bb8f9" exitCode=0
Sep 14 03:14:03.814290 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:03.814199    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-12ecd131-78cd7db6f7-6rt5l"
Sep 14 03:14:03.814367 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:03.814217    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-12ecd131-78cd7db6f7-6rt5l" event={"ID":"dffb2be6-e854-460c-9af8-005afdad9108","Type":"ContainerDied","Data":"8cc98ac423fae5fc6857c32e2461087a27fd7d6ca25f49e93c110582179bb8f9"}
Sep 14 03:14:03.814454 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:03.814424    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-12ecd131-78cd7db6f7-6rt5l" event={"ID":"dffb2be6-e854-460c-9af8-005afdad9108","Type":"ContainerDied","Data":"15c69408bdcc9aa2d364e4c159cb720633a9912091859bd36f8d0f3ca72d4a75"}
Sep 14 03:14:03.822867 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:03.822849    2564 scope.go:117] "RemoveContainer" containerID="d3b1c947d4f556f7dad261924a2768f8d9ad8bad987af382fcc319906aacd079"
Sep 14 03:14:03.823197 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:14:03.823178    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3b1c947d4f556f7dad261924a2768f8d9ad8bad987af382fcc319906aacd079\": container with ID starting with d3b1c947d4f556f7dad261924a2768f8d9ad8bad987af382fcc319906aacd079 not found: ID does not exist" containerID="d3b1c947d4f556f7dad261924a2768f8d9ad8bad987af382fcc319906aacd079"
Sep 14 03:14:03.823246 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:03.823206    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3b1c947d4f556f7dad261924a2768f8d9ad8bad987af382fcc319906aacd079"} err="failed to get container status \"d3b1c947d4f556f7dad261924a2768f8d9ad8bad987af382fcc319906aacd079\": rpc error: code = NotFound desc = could not find container \"d3b1c947d4f556f7dad261924a2768f8d9ad8bad987af382fcc319906aacd079\": container with ID starting with d3b1c947d4f556f7dad261924a2768f8d9ad8bad987af382fcc319906aacd079 not found: ID does not exist"
Sep 14 03:14:03.823246 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:03.823224    2564 scope.go:117] "RemoveContainer" containerID="8cc98ac423fae5fc6857c32e2461087a27fd7d6ca25f49e93c110582179bb8f9"
Sep 14 03:14:03.832076 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:03.832060    2564 scope.go:117] "RemoveContainer" containerID="8cc98ac423fae5fc6857c32e2461087a27fd7d6ca25f49e93c110582179bb8f9"
Sep 14 03:14:03.832362 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:14:03.832344    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cc98ac423fae5fc6857c32e2461087a27fd7d6ca25f49e93c110582179bb8f9\": container with ID starting with 8cc98ac423fae5fc6857c32e2461087a27fd7d6ca25f49e93c110582179bb8f9 not found: ID does not exist" containerID="8cc98ac423fae5fc6857c32e2461087a27fd7d6ca25f49e93c110582179bb8f9"
Sep 14 03:14:03.832433 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:03.832369    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cc98ac423fae5fc6857c32e2461087a27fd7d6ca25f49e93c110582179bb8f9"} err="failed to get container status \"8cc98ac423fae5fc6857c32e2461087a27fd7d6ca25f49e93c110582179bb8f9\": rpc error: code = NotFound desc = could not find container \"8cc98ac423fae5fc6857c32e2461087a27fd7d6ca25f49e93c110582179bb8f9\": container with ID starting with 8cc98ac423fae5fc6857c32e2461087a27fd7d6ca25f49e93c110582179bb8f9 not found: ID does not exist"
Sep 14 03:14:03.840913 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:03.840872    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-12ecd131-7f655f44fd-cm47f"]
Sep 14 03:14:03.843307 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:03.843283    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-12ecd131-7f655f44fd-cm47f"]
Sep 14 03:14:03.848413 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:03.848380    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:14:03.856948 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:03.856925    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-12ecd131-78cd7db6f7-6rt5l"]
Sep 14 03:14:03.859786 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:03.859766    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-12ecd131-78cd7db6f7-6rt5l"]
Sep 14 03:14:04.737419 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:04.737364    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:14:05.099193 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:05.099155    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="238e6109-9a10-45a4-9014-cfba173627c7" path="/var/lib/kubelet/pods/238e6109-9a10-45a4-9014-cfba173627c7/volumes"
Sep 14 03:14:05.099680 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:05.099596    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dffb2be6-e854-460c-9af8-005afdad9108" path="/var/lib/kubelet/pods/dffb2be6-e854-460c-9af8-005afdad9108/volumes"
Sep 14 03:14:05.604277 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:05.604245    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:14:06.356036 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:06.356004    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:14:06.531918 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:06.531888    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:14:07.029507 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:07.029467    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:14:07.873028 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:07.872998    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:14:08.601774 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:08.601715    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:14:11.056434 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:11.056383    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-9c9f1e/distinct-2-w4-9c9f1e-kserve-58779c476f-qmt8b"]
Sep 14 03:14:11.057088 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:11.057060    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dffb2be6-e854-460c-9af8-005afdad9108" containerName="maas-api"
Sep 14 03:14:11.057088 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:11.057089    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="dffb2be6-e854-460c-9af8-005afdad9108" containerName="maas-api"
Sep 14 03:14:11.057279 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:11.057120    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="238e6109-9a10-45a4-9014-cfba173627c7" containerName="payload-pre-processing"
Sep 14 03:14:11.057279 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:11.057129    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="238e6109-9a10-45a4-9014-cfba173627c7" containerName="payload-pre-processing"
Sep 14 03:14:11.057279 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:11.057251    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="dffb2be6-e854-460c-9af8-005afdad9108" containerName="maas-api"
Sep 14 03:14:11.057279 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:11.057268    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="238e6109-9a10-45a4-9014-cfba173627c7" containerName="payload-pre-processing"
Sep 14 03:14:11.062960 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:11.062934    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9c9f1e/distinct-2-w4-9c9f1e-kserve-58779c476f-qmt8b"
Sep 14 03:14:11.065693 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:11.065670    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-9c9f1e\"/\"distinct-2-w4-9c9f1e-kserve-self-signed-certs\""
Sep 14 03:14:11.075018 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:11.074774    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-9c9f1e/distinct-2-w4-9c9f1e-kserve-58779c476f-qmt8b"]
Sep 14 03:14:11.088164 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:11.088131    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:14:11.184206 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:11.184160    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsvsz\" (UniqueName: \"kubernetes.io/projected/4ced0495-da0f-47b8-ab59-d369f274eb29-kube-api-access-vsvsz\") pod \"distinct-2-w4-9c9f1e-kserve-58779c476f-qmt8b\" (UID: \"4ced0495-da0f-47b8-ab59-d369f274eb29\") " pod="e2e-models-e2e-worker-w4-9c9f1e/distinct-2-w4-9c9f1e-kserve-58779c476f-qmt8b"
Sep 14 03:14:11.184390 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:11.184315    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ced0495-da0f-47b8-ab59-d369f274eb29-tmp-dir\") pod \"distinct-2-w4-9c9f1e-kserve-58779c476f-qmt8b\" (UID: \"4ced0495-da0f-47b8-ab59-d369f274eb29\") " pod="e2e-models-e2e-worker-w4-9c9f1e/distinct-2-w4-9c9f1e-kserve-58779c476f-qmt8b"
Sep 14 03:14:11.184390 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:11.184368    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ced0495-da0f-47b8-ab59-d369f274eb29-home\") pod \"distinct-2-w4-9c9f1e-kserve-58779c476f-qmt8b\" (UID: \"4ced0495-da0f-47b8-ab59-d369f274eb29\") " pod="e2e-models-e2e-worker-w4-9c9f1e/distinct-2-w4-9c9f1e-kserve-58779c476f-qmt8b"
Sep 14 03:14:11.184534 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:11.184484    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ced0495-da0f-47b8-ab59-d369f274eb29-model-cache\") pod \"distinct-2-w4-9c9f1e-kserve-58779c476f-qmt8b\" (UID: \"4ced0495-da0f-47b8-ab59-d369f274eb29\") " pod="e2e-models-e2e-worker-w4-9c9f1e/distinct-2-w4-9c9f1e-kserve-58779c476f-qmt8b"
Sep 14 03:14:11.184619 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:11.184602    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ced0495-da0f-47b8-ab59-d369f274eb29-tls-certs\") pod \"distinct-2-w4-9c9f1e-kserve-58779c476f-qmt8b\" (UID: \"4ced0495-da0f-47b8-ab59-d369f274eb29\") " pod="e2e-models-e2e-worker-w4-9c9f1e/distinct-2-w4-9c9f1e-kserve-58779c476f-qmt8b"
Sep 14 03:14:11.184680 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:11.184669    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ced0495-da0f-47b8-ab59-d369f274eb29-dshm\") pod \"distinct-2-w4-9c9f1e-kserve-58779c476f-qmt8b\" (UID: \"4ced0495-da0f-47b8-ab59-d369f274eb29\") " pod="e2e-models-e2e-worker-w4-9c9f1e/distinct-2-w4-9c9f1e-kserve-58779c476f-qmt8b"
Sep 14 03:14:11.286009 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:11.285973    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ced0495-da0f-47b8-ab59-d369f274eb29-dshm\") pod \"distinct-2-w4-9c9f1e-kserve-58779c476f-qmt8b\" (UID: \"4ced0495-da0f-47b8-ab59-d369f274eb29\") " pod="e2e-models-e2e-worker-w4-9c9f1e/distinct-2-w4-9c9f1e-kserve-58779c476f-qmt8b"
Sep 14 03:14:11.286200 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:11.286031    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsvsz\" (UniqueName: \"kubernetes.io/projected/4ced0495-da0f-47b8-ab59-d369f274eb29-kube-api-access-vsvsz\") pod \"distinct-2-w4-9c9f1e-kserve-58779c476f-qmt8b\" (UID: \"4ced0495-da0f-47b8-ab59-d369f274eb29\") " pod="e2e-models-e2e-worker-w4-9c9f1e/distinct-2-w4-9c9f1e-kserve-58779c476f-qmt8b"
Sep 14 03:14:11.286200 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:11.286084    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ced0495-da0f-47b8-ab59-d369f274eb29-tmp-dir\") pod \"distinct-2-w4-9c9f1e-kserve-58779c476f-qmt8b\" (UID: \"4ced0495-da0f-47b8-ab59-d369f274eb29\") " pod="e2e-models-e2e-worker-w4-9c9f1e/distinct-2-w4-9c9f1e-kserve-58779c476f-qmt8b"
Sep 14 03:14:11.286200 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:11.286113    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ced0495-da0f-47b8-ab59-d369f274eb29-home\") pod \"distinct-2-w4-9c9f1e-kserve-58779c476f-qmt8b\" (UID: \"4ced0495-da0f-47b8-ab59-d369f274eb29\") " pod="e2e-models-e2e-worker-w4-9c9f1e/distinct-2-w4-9c9f1e-kserve-58779c476f-qmt8b"
Sep 14 03:14:11.286200 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:11.286149    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ced0495-da0f-47b8-ab59-d369f274eb29-model-cache\") pod \"distinct-2-w4-9c9f1e-kserve-58779c476f-qmt8b\" (UID: \"4ced0495-da0f-47b8-ab59-d369f274eb29\") " pod="e2e-models-e2e-worker-w4-9c9f1e/distinct-2-w4-9c9f1e-kserve-58779c476f-qmt8b"
Sep 14 03:14:11.286449 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:11.286212    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ced0495-da0f-47b8-ab59-d369f274eb29-tls-certs\") pod \"distinct-2-w4-9c9f1e-kserve-58779c476f-qmt8b\" (UID: \"4ced0495-da0f-47b8-ab59-d369f274eb29\") " pod="e2e-models-e2e-worker-w4-9c9f1e/distinct-2-w4-9c9f1e-kserve-58779c476f-qmt8b"
Sep 14 03:14:11.286556 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:11.286530    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ced0495-da0f-47b8-ab59-d369f274eb29-tmp-dir\") pod \"distinct-2-w4-9c9f1e-kserve-58779c476f-qmt8b\" (UID: \"4ced0495-da0f-47b8-ab59-d369f274eb29\") " pod="e2e-models-e2e-worker-w4-9c9f1e/distinct-2-w4-9c9f1e-kserve-58779c476f-qmt8b"
Sep 14 03:14:11.286627 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:11.286552    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ced0495-da0f-47b8-ab59-d369f274eb29-model-cache\") pod \"distinct-2-w4-9c9f1e-kserve-58779c476f-qmt8b\" (UID: \"4ced0495-da0f-47b8-ab59-d369f274eb29\") " pod="e2e-models-e2e-worker-w4-9c9f1e/distinct-2-w4-9c9f1e-kserve-58779c476f-qmt8b"
Sep 14 03:14:11.286627 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:11.286618    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ced0495-da0f-47b8-ab59-d369f274eb29-home\") pod \"distinct-2-w4-9c9f1e-kserve-58779c476f-qmt8b\" (UID: \"4ced0495-da0f-47b8-ab59-d369f274eb29\") " pod="e2e-models-e2e-worker-w4-9c9f1e/distinct-2-w4-9c9f1e-kserve-58779c476f-qmt8b"
Sep 14 03:14:11.288308 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:11.288277    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ced0495-da0f-47b8-ab59-d369f274eb29-dshm\") pod \"distinct-2-w4-9c9f1e-kserve-58779c476f-qmt8b\" (UID: \"4ced0495-da0f-47b8-ab59-d369f274eb29\") " pod="e2e-models-e2e-worker-w4-9c9f1e/distinct-2-w4-9c9f1e-kserve-58779c476f-qmt8b"
Sep 14 03:14:11.288649 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:11.288632    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ced0495-da0f-47b8-ab59-d369f274eb29-tls-certs\") pod \"distinct-2-w4-9c9f1e-kserve-58779c476f-qmt8b\" (UID: \"4ced0495-da0f-47b8-ab59-d369f274eb29\") " pod="e2e-models-e2e-worker-w4-9c9f1e/distinct-2-w4-9c9f1e-kserve-58779c476f-qmt8b"
Sep 14 03:14:11.294778 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:11.294757    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsvsz\" (UniqueName: \"kubernetes.io/projected/4ced0495-da0f-47b8-ab59-d369f274eb29-kube-api-access-vsvsz\") pod \"distinct-2-w4-9c9f1e-kserve-58779c476f-qmt8b\" (UID: \"4ced0495-da0f-47b8-ab59-d369f274eb29\") " pod="e2e-models-e2e-worker-w4-9c9f1e/distinct-2-w4-9c9f1e-kserve-58779c476f-qmt8b"
Sep 14 03:14:11.373446 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:11.373350    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9c9f1e/distinct-2-w4-9c9f1e-kserve-58779c476f-qmt8b"
Sep 14 03:14:11.510077 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:11.510043    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-12ecd131/e2e-lifecycle-model-12ecd131-kserve-78949b58dc-x8c4c"]
Sep 14 03:14:11.604729 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:11.604692    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:14:11.652338 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:11.652311    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-9c9f1e/distinct-2-w4-9c9f1e-kserve-58779c476f-qmt8b"]
Sep 14 03:14:11.654272 ip-10-0-141-4 kubenswrapper[2564]: W0914 03:14:11.654239    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4ced0495_da0f_47b8_ab59_d369f274eb29.slice/crio-06e1af2b9d342a8e65e429698ccd2b57427073eea5ce92e0bc15f50ca675894d WatchSource:0}: Error finding container 06e1af2b9d342a8e65e429698ccd2b57427073eea5ce92e0bc15f50ca675894d: Status 404 returned error can't find the container with id 06e1af2b9d342a8e65e429698ccd2b57427073eea5ce92e0bc15f50ca675894d
Sep 14 03:14:11.864956 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:11.864920    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9c9f1e/distinct-2-w4-9c9f1e-kserve-58779c476f-qmt8b" event={"ID":"4ced0495-da0f-47b8-ab59-d369f274eb29","Type":"ContainerStarted","Data":"e006c7368c1c2260aa2ae0c60c877847dc3920aa684e3a6fd97df6569a6a6709"}
Sep 14 03:14:11.865125 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:11.864965    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9c9f1e/distinct-2-w4-9c9f1e-kserve-58779c476f-qmt8b" event={"ID":"4ced0495-da0f-47b8-ab59-d369f274eb29","Type":"ContainerStarted","Data":"06e1af2b9d342a8e65e429698ccd2b57427073eea5ce92e0bc15f50ca675894d"}
Sep 14 03:14:11.886397 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:11.886268    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-9c9f1e/distinct-2-w4-9c9f1e-kserve-58779c476f-qmt8b" podStartSLOduration=0.886248423 podStartE2EDuration="886.248423ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:14:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:14:11.885790339 +0000 UTC m=+1297.266874538" watchObservedRunningTime="2026-09-14 03:14:11.886248423 +0000 UTC m=+1297.267332623"
Sep 14 03:14:21.373551 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:21.373510    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9c9f1e/distinct-2-w4-9c9f1e-kserve-58779c476f-qmt8b"
Sep 14 03:14:21.374036 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:21.373566    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-9c9f1e/distinct-2-w4-9c9f1e-kserve-58779c476f-qmt8b"
Sep 14 03:14:21.385996 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:21.385969    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-9c9f1e/distinct-2-w4-9c9f1e-kserve-58779c476f-qmt8b"
Sep 14 03:14:21.935273 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:21.935241    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9c9f1e/distinct-2-w4-9c9f1e-kserve-58779c476f-qmt8b"
Sep 14 03:14:26.533377 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:26.533335    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-12ecd131/e2e-lifecycle-model-12ecd131-kserve-78949b58dc-x8c4c" podUID="d4d4050b-2492-4a29-89d3-53016a614381" containerName="main" containerID="cri-o://6d70db1562e293eb63d3743415741b948b402b151ddd79648cb97e15c123b5be" gracePeriod=45
Sep 14 03:14:26.791116 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:26.791050    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-12ecd131/e2e-lifecycle-model-12ecd131-kserve-78949b58dc-x8c4c"
Sep 14 03:14:26.845929 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:26.845884    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4d4050b-2492-4a29-89d3-53016a614381-dshm\") pod \"d4d4050b-2492-4a29-89d3-53016a614381\" (UID: \"d4d4050b-2492-4a29-89d3-53016a614381\") "
Sep 14 03:14:26.846083 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:26.845990    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4d4050b-2492-4a29-89d3-53016a614381-tls-certs\") pod \"d4d4050b-2492-4a29-89d3-53016a614381\" (UID: \"d4d4050b-2492-4a29-89d3-53016a614381\") "
Sep 14 03:14:26.846083 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:26.846019    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4d4050b-2492-4a29-89d3-53016a614381-tmp-dir\") pod \"d4d4050b-2492-4a29-89d3-53016a614381\" (UID: \"d4d4050b-2492-4a29-89d3-53016a614381\") "
Sep 14 03:14:26.846164 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:26.846084    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lwkrq\" (UniqueName: \"kubernetes.io/projected/d4d4050b-2492-4a29-89d3-53016a614381-kube-api-access-lwkrq\") pod \"d4d4050b-2492-4a29-89d3-53016a614381\" (UID: \"d4d4050b-2492-4a29-89d3-53016a614381\") "
Sep 14 03:14:26.846164 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:26.846143    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4d4050b-2492-4a29-89d3-53016a614381-model-cache\") pod \"d4d4050b-2492-4a29-89d3-53016a614381\" (UID: \"d4d4050b-2492-4a29-89d3-53016a614381\") "
Sep 14 03:14:26.846268 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:26.846209    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4d4050b-2492-4a29-89d3-53016a614381-home\") pod \"d4d4050b-2492-4a29-89d3-53016a614381\" (UID: \"d4d4050b-2492-4a29-89d3-53016a614381\") "
Sep 14 03:14:26.846421 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:26.846380    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4d4050b-2492-4a29-89d3-53016a614381-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d4d4050b-2492-4a29-89d3-53016a614381" (UID: "d4d4050b-2492-4a29-89d3-53016a614381"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:14:26.846663 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:26.846537    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4d4050b-2492-4a29-89d3-53016a614381-home" (OuterVolumeSpecName: "home") pod "d4d4050b-2492-4a29-89d3-53016a614381" (UID: "d4d4050b-2492-4a29-89d3-53016a614381"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:14:26.846663 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:26.846498    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4d4050b-2492-4a29-89d3-53016a614381-model-cache" (OuterVolumeSpecName: "model-cache") pod "d4d4050b-2492-4a29-89d3-53016a614381" (UID: "d4d4050b-2492-4a29-89d3-53016a614381"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:14:26.846794 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:26.846672    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4d4050b-2492-4a29-89d3-53016a614381-tmp-dir\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:14:26.846794 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:26.846691    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4d4050b-2492-4a29-89d3-53016a614381-model-cache\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:14:26.846794 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:26.846704    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4d4050b-2492-4a29-89d3-53016a614381-home\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:14:26.848670 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:26.848642    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4d4050b-2492-4a29-89d3-53016a614381-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d4d4050b-2492-4a29-89d3-53016a614381" (UID: "d4d4050b-2492-4a29-89d3-53016a614381"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 03:14:26.848782 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:26.848736    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4d4050b-2492-4a29-89d3-53016a614381-dshm" (OuterVolumeSpecName: "dshm") pod "d4d4050b-2492-4a29-89d3-53016a614381" (UID: "d4d4050b-2492-4a29-89d3-53016a614381"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:14:26.848782 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:26.848747    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4d4050b-2492-4a29-89d3-53016a614381-kube-api-access-lwkrq" (OuterVolumeSpecName: "kube-api-access-lwkrq") pod "d4d4050b-2492-4a29-89d3-53016a614381" (UID: "d4d4050b-2492-4a29-89d3-53016a614381"). InnerVolumeSpecName "kube-api-access-lwkrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:14:26.947479 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:26.947436    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwkrq\" (UniqueName: \"kubernetes.io/projected/d4d4050b-2492-4a29-89d3-53016a614381-kube-api-access-lwkrq\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:14:26.947479 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:26.947472    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4d4050b-2492-4a29-89d3-53016a614381-dshm\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:14:26.947479 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:26.947485    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4d4050b-2492-4a29-89d3-53016a614381-tls-certs\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:14:26.951792 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:26.951761    2564 generic.go:356] "Generic (PLEG): container finished" podID="d4d4050b-2492-4a29-89d3-53016a614381" containerID="6d70db1562e293eb63d3743415741b948b402b151ddd79648cb97e15c123b5be" exitCode=0
Sep 14 03:14:26.951921 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:26.951808    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-12ecd131/e2e-lifecycle-model-12ecd131-kserve-78949b58dc-x8c4c"
Sep 14 03:14:26.951921 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:26.951848    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-12ecd131/e2e-lifecycle-model-12ecd131-kserve-78949b58dc-x8c4c" event={"ID":"d4d4050b-2492-4a29-89d3-53016a614381","Type":"ContainerDied","Data":"6d70db1562e293eb63d3743415741b948b402b151ddd79648cb97e15c123b5be"}
Sep 14 03:14:26.951921 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:26.951889    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-12ecd131/e2e-lifecycle-model-12ecd131-kserve-78949b58dc-x8c4c" event={"ID":"d4d4050b-2492-4a29-89d3-53016a614381","Type":"ContainerDied","Data":"301d3c14d058e075b76472f4507705ac6e902fed5786fa885af364d81e528a90"}
Sep 14 03:14:26.951921 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:26.951913    2564 scope.go:117] "RemoveContainer" containerID="6d70db1562e293eb63d3743415741b948b402b151ddd79648cb97e15c123b5be"
Sep 14 03:14:26.961624 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:26.961607    2564 scope.go:117] "RemoveContainer" containerID="6d70db1562e293eb63d3743415741b948b402b151ddd79648cb97e15c123b5be"
Sep 14 03:14:26.961901 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:14:26.961880    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d70db1562e293eb63d3743415741b948b402b151ddd79648cb97e15c123b5be\": container with ID starting with 6d70db1562e293eb63d3743415741b948b402b151ddd79648cb97e15c123b5be not found: ID does not exist" containerID="6d70db1562e293eb63d3743415741b948b402b151ddd79648cb97e15c123b5be"
Sep 14 03:14:26.961962 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:26.961909    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d70db1562e293eb63d3743415741b948b402b151ddd79648cb97e15c123b5be"} err="failed to get container status \"6d70db1562e293eb63d3743415741b948b402b151ddd79648cb97e15c123b5be\": rpc error: code = NotFound desc = could not find container \"6d70db1562e293eb63d3743415741b948b402b151ddd79648cb97e15c123b5be\": container with ID starting with 6d70db1562e293eb63d3743415741b948b402b151ddd79648cb97e15c123b5be not found: ID does not exist"
Sep 14 03:14:26.980266 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:26.980239    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-12ecd131/e2e-lifecycle-model-12ecd131-kserve-78949b58dc-x8c4c"]
Sep 14 03:14:26.984545 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:26.984523    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-12ecd131/e2e-lifecycle-model-12ecd131-kserve-78949b58dc-x8c4c"]
Sep 14 03:14:27.099326 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:27.099247    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4d4050b-2492-4a29-89d3-53016a614381" path="/var/lib/kubelet/pods/d4d4050b-2492-4a29-89d3-53016a614381/volumes"
Sep 14 03:14:48.291053 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:48.291016    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:14:48.807934 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:48.807906    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:14:54.543084 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:54.543047    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:14:54.616530 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:54.616493    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:14:54.678924 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:54.678892    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:14:54.809313 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:54.809227    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:14:56.186278 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:56.186222    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:14:56.276737 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:56.276706    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:14:56.386823 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:56.386788    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:14:57.119775 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:57.119745    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:14:57.220617 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:57.220587    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:14:57.316150 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:57.316119    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:14:58.651749 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:14:58.651703    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:15:00.233658 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:00.233622    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9c9f1e-29822595-z4rrc"]
Sep 14 03:15:00.234374 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:00.234353    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4d4050b-2492-4a29-89d3-53016a614381" containerName="main"
Sep 14 03:15:00.234456 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:00.234379    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4d4050b-2492-4a29-89d3-53016a614381" containerName="main"
Sep 14 03:15:00.234563 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:00.234549    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4d4050b-2492-4a29-89d3-53016a614381" containerName="main"
Sep 14 03:15:00.239594 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:00.239575    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9c9f1e-29822595-z4rrc"
Sep 14 03:15:00.242572 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:00.242552    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sv5sd\""
Sep 14 03:15:00.247078 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:00.247054    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9c9f1e-29822595-z4rrc"]
Sep 14 03:15:00.252818 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:00.252800    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9c9f1e-29822595-z4rrc"
Sep 14 03:15:00.527543 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:00.527519    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9c9f1e-29822595-z4rrc"]
Sep 14 03:15:00.529374 ip-10-0-141-4 kubenswrapper[2564]: W0914 03:15:00.529346    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda633a62e_b4c2_49d5_82ba_e6ae0b7d02ec.slice/crio-5e001e0eb5ec166dbb1c7e44f00740fff1899b371fc47904dfcac2321053f646 WatchSource:0}: Error finding container 5e001e0eb5ec166dbb1c7e44f00740fff1899b371fc47904dfcac2321053f646: Status 404 returned error can't find the container with id 5e001e0eb5ec166dbb1c7e44f00740fff1899b371fc47904dfcac2321053f646
Sep 14 03:15:01.138743 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:01.138710    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9c9f1e-29822595-z4rrc" event={"ID":"a633a62e-b4c2-49d5-82ba-e6ae0b7d02ec","Type":"ContainerStarted","Data":"5e001e0eb5ec166dbb1c7e44f00740fff1899b371fc47904dfcac2321053f646"}
Sep 14 03:15:03.864879 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:03.864845    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-3a752f/embedding-w0-3a752f-kserve-65c77cc784-2gd2r"]
Sep 14 03:15:03.870293 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:03.870270    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3a752f/embedding-w0-3a752f-kserve-65c77cc784-2gd2r"
Sep 14 03:15:03.874722 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:03.874521    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-3a752f\"/\"embedding-w0-3a752f-kserve-self-signed-certs\""
Sep 14 03:15:03.874869 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:03.874543    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-3a752f\"/\"kube-root-ca.crt\""
Sep 14 03:15:03.875014 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:03.874654    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-3a752f\"/\"default-dockercfg-w6bjn\""
Sep 14 03:15:03.875138 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:03.874672    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-3a752f\"/\"openshift-service-ca.crt\""
Sep 14 03:15:03.877781 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:03.877753    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-3a752f/embedding-w0-3a752f-kserve-65c77cc784-2gd2r"]
Sep 14 03:15:03.895643 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:03.895604    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:15:03.920774 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:03.920734    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw69q\" (UniqueName: \"kubernetes.io/projected/d9a686f2-3fa7-4b42-b237-53be6ba2b050-kube-api-access-lw69q\") pod \"embedding-w0-3a752f-kserve-65c77cc784-2gd2r\" (UID: \"d9a686f2-3fa7-4b42-b237-53be6ba2b050\") " pod="e2e-models-e2e-worker-w0-3a752f/embedding-w0-3a752f-kserve-65c77cc784-2gd2r"
Sep 14 03:15:03.920939 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:03.920825    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9a686f2-3fa7-4b42-b237-53be6ba2b050-tmp-dir\") pod \"embedding-w0-3a752f-kserve-65c77cc784-2gd2r\" (UID: \"d9a686f2-3fa7-4b42-b237-53be6ba2b050\") " pod="e2e-models-e2e-worker-w0-3a752f/embedding-w0-3a752f-kserve-65c77cc784-2gd2r"
Sep 14 03:15:03.920939 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:03.920870    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9a686f2-3fa7-4b42-b237-53be6ba2b050-dshm\") pod \"embedding-w0-3a752f-kserve-65c77cc784-2gd2r\" (UID: \"d9a686f2-3fa7-4b42-b237-53be6ba2b050\") " pod="e2e-models-e2e-worker-w0-3a752f/embedding-w0-3a752f-kserve-65c77cc784-2gd2r"
Sep 14 03:15:03.920939 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:03.920902    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9a686f2-3fa7-4b42-b237-53be6ba2b050-tls-certs\") pod \"embedding-w0-3a752f-kserve-65c77cc784-2gd2r\" (UID: \"d9a686f2-3fa7-4b42-b237-53be6ba2b050\") " pod="e2e-models-e2e-worker-w0-3a752f/embedding-w0-3a752f-kserve-65c77cc784-2gd2r"
Sep 14 03:15:03.920939 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:03.920921    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9a686f2-3fa7-4b42-b237-53be6ba2b050-model-cache\") pod \"embedding-w0-3a752f-kserve-65c77cc784-2gd2r\" (UID: \"d9a686f2-3fa7-4b42-b237-53be6ba2b050\") " pod="e2e-models-e2e-worker-w0-3a752f/embedding-w0-3a752f-kserve-65c77cc784-2gd2r"
Sep 14 03:15:03.921089 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:03.920980    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9a686f2-3fa7-4b42-b237-53be6ba2b050-home\") pod \"embedding-w0-3a752f-kserve-65c77cc784-2gd2r\" (UID: \"d9a686f2-3fa7-4b42-b237-53be6ba2b050\") " pod="e2e-models-e2e-worker-w0-3a752f/embedding-w0-3a752f-kserve-65c77cc784-2gd2r"
Sep 14 03:15:04.022214 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:04.022173    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9a686f2-3fa7-4b42-b237-53be6ba2b050-tls-certs\") pod \"embedding-w0-3a752f-kserve-65c77cc784-2gd2r\" (UID: \"d9a686f2-3fa7-4b42-b237-53be6ba2b050\") " pod="e2e-models-e2e-worker-w0-3a752f/embedding-w0-3a752f-kserve-65c77cc784-2gd2r"
Sep 14 03:15:04.022214 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:04.022216    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9a686f2-3fa7-4b42-b237-53be6ba2b050-model-cache\") pod \"embedding-w0-3a752f-kserve-65c77cc784-2gd2r\" (UID: \"d9a686f2-3fa7-4b42-b237-53be6ba2b050\") " pod="e2e-models-e2e-worker-w0-3a752f/embedding-w0-3a752f-kserve-65c77cc784-2gd2r"
Sep 14 03:15:04.022528 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:04.022256    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9a686f2-3fa7-4b42-b237-53be6ba2b050-home\") pod \"embedding-w0-3a752f-kserve-65c77cc784-2gd2r\" (UID: \"d9a686f2-3fa7-4b42-b237-53be6ba2b050\") " pod="e2e-models-e2e-worker-w0-3a752f/embedding-w0-3a752f-kserve-65c77cc784-2gd2r"
Sep 14 03:15:04.022528 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:04.022285    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw69q\" (UniqueName: \"kubernetes.io/projected/d9a686f2-3fa7-4b42-b237-53be6ba2b050-kube-api-access-lw69q\") pod \"embedding-w0-3a752f-kserve-65c77cc784-2gd2r\" (UID: \"d9a686f2-3fa7-4b42-b237-53be6ba2b050\") " pod="e2e-models-e2e-worker-w0-3a752f/embedding-w0-3a752f-kserve-65c77cc784-2gd2r"
Sep 14 03:15:04.022528 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:04.022342    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9a686f2-3fa7-4b42-b237-53be6ba2b050-tmp-dir\") pod \"embedding-w0-3a752f-kserve-65c77cc784-2gd2r\" (UID: \"d9a686f2-3fa7-4b42-b237-53be6ba2b050\") " pod="e2e-models-e2e-worker-w0-3a752f/embedding-w0-3a752f-kserve-65c77cc784-2gd2r"
Sep 14 03:15:04.022528 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:04.022398    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9a686f2-3fa7-4b42-b237-53be6ba2b050-dshm\") pod \"embedding-w0-3a752f-kserve-65c77cc784-2gd2r\" (UID: \"d9a686f2-3fa7-4b42-b237-53be6ba2b050\") " pod="e2e-models-e2e-worker-w0-3a752f/embedding-w0-3a752f-kserve-65c77cc784-2gd2r"
Sep 14 03:15:04.022790 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:04.022769    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9a686f2-3fa7-4b42-b237-53be6ba2b050-model-cache\") pod \"embedding-w0-3a752f-kserve-65c77cc784-2gd2r\" (UID: \"d9a686f2-3fa7-4b42-b237-53be6ba2b050\") " pod="e2e-models-e2e-worker-w0-3a752f/embedding-w0-3a752f-kserve-65c77cc784-2gd2r"
Sep 14 03:15:04.022855 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:04.022773    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9a686f2-3fa7-4b42-b237-53be6ba2b050-home\") pod \"embedding-w0-3a752f-kserve-65c77cc784-2gd2r\" (UID: \"d9a686f2-3fa7-4b42-b237-53be6ba2b050\") " pod="e2e-models-e2e-worker-w0-3a752f/embedding-w0-3a752f-kserve-65c77cc784-2gd2r"
Sep 14 03:15:04.022855 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:04.022815    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9a686f2-3fa7-4b42-b237-53be6ba2b050-tmp-dir\") pod \"embedding-w0-3a752f-kserve-65c77cc784-2gd2r\" (UID: \"d9a686f2-3fa7-4b42-b237-53be6ba2b050\") " pod="e2e-models-e2e-worker-w0-3a752f/embedding-w0-3a752f-kserve-65c77cc784-2gd2r"
Sep 14 03:15:04.024609 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:04.024590    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9a686f2-3fa7-4b42-b237-53be6ba2b050-dshm\") pod \"embedding-w0-3a752f-kserve-65c77cc784-2gd2r\" (UID: \"d9a686f2-3fa7-4b42-b237-53be6ba2b050\") " pod="e2e-models-e2e-worker-w0-3a752f/embedding-w0-3a752f-kserve-65c77cc784-2gd2r"
Sep 14 03:15:04.024704 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:04.024687    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9a686f2-3fa7-4b42-b237-53be6ba2b050-tls-certs\") pod \"embedding-w0-3a752f-kserve-65c77cc784-2gd2r\" (UID: \"d9a686f2-3fa7-4b42-b237-53be6ba2b050\") " pod="e2e-models-e2e-worker-w0-3a752f/embedding-w0-3a752f-kserve-65c77cc784-2gd2r"
Sep 14 03:15:04.031037 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:04.031009    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw69q\" (UniqueName: \"kubernetes.io/projected/d9a686f2-3fa7-4b42-b237-53be6ba2b050-kube-api-access-lw69q\") pod \"embedding-w0-3a752f-kserve-65c77cc784-2gd2r\" (UID: \"d9a686f2-3fa7-4b42-b237-53be6ba2b050\") " pod="e2e-models-e2e-worker-w0-3a752f/embedding-w0-3a752f-kserve-65c77cc784-2gd2r"
Sep 14 03:15:04.185073 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:04.184965    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3a752f/embedding-w0-3a752f-kserve-65c77cc784-2gd2r"
Sep 14 03:15:04.464258 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:04.464224    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-3a752f/embedding-w0-3a752f-kserve-65c77cc784-2gd2r"]
Sep 14 03:15:04.466134 ip-10-0-141-4 kubenswrapper[2564]: W0914 03:15:04.466099    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd9a686f2_3fa7_4b42_b237_53be6ba2b050.slice/crio-5f62e04dd82e33826398439bfeed93346b395593ad0b3975fa32f7fdd5813f01 WatchSource:0}: Error finding container 5f62e04dd82e33826398439bfeed93346b395593ad0b3975fa32f7fdd5813f01: Status 404 returned error can't find the container with id 5f62e04dd82e33826398439bfeed93346b395593ad0b3975fa32f7fdd5813f01
Sep 14 03:15:04.913905 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:04.913866    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:15:05.042569 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:05.042533    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:15:05.164701 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:05.164610    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3a752f/embedding-w0-3a752f-kserve-65c77cc784-2gd2r" event={"ID":"d9a686f2-3fa7-4b42-b237-53be6ba2b050","Type":"ContainerStarted","Data":"b5299b825348b563b36a8b2701026e1e36bde22ade897339a3ff0cead5376051"}
Sep 14 03:15:05.164701 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:05.164660    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3a752f/embedding-w0-3a752f-kserve-65c77cc784-2gd2r" event={"ID":"d9a686f2-3fa7-4b42-b237-53be6ba2b050","Type":"ContainerStarted","Data":"5f62e04dd82e33826398439bfeed93346b395593ad0b3975fa32f7fdd5813f01"}
Sep 14 03:15:05.184664 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:05.184615    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-3a752f/embedding-w0-3a752f-kserve-65c77cc784-2gd2r" podStartSLOduration=2.184597137 podStartE2EDuration="2.184597137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:15:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:15:05.18289107 +0000 UTC m=+1350.563975292" watchObservedRunningTime="2026-09-14 03:15:05.184597137 +0000 UTC m=+1350.565681362"
Sep 14 03:15:05.558806 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:05.558732    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:15:06.171588 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:06.171493    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9c9f1e-29822595-z4rrc" event={"ID":"a633a62e-b4c2-49d5-82ba-e6ae0b7d02ec","Type":"ContainerStarted","Data":"1f47a0cca98f448fd69878725a945fdbc8bf6a7dcfe7b9c0fbec06de96300b5c"}
Sep 14 03:15:06.190514 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:06.190468    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9c9f1e-29822595-z4rrc" podStartSLOduration=1.168005523 podStartE2EDuration="6.190443337s" totalImagesPullingTime="5.022437814s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:15:06.189013901 +0000 UTC m=+1351.570098100" watchObservedRunningTime="2026-09-14 03:15:06.190443337 +0000 UTC m=+1351.571527534"
Sep 14 03:15:07.179513 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:07.179479    2564 generic.go:356] "Generic (PLEG): container finished" podID="a633a62e-b4c2-49d5-82ba-e6ae0b7d02ec" containerID="1f47a0cca98f448fd69878725a945fdbc8bf6a7dcfe7b9c0fbec06de96300b5c" exitCode=0
Sep 14 03:15:07.179513 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:07.179516    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9c9f1e-29822595-z4rrc" event={"ID":"a633a62e-b4c2-49d5-82ba-e6ae0b7d02ec","Type":"ContainerDied","Data":"1f47a0cca98f448fd69878725a945fdbc8bf6a7dcfe7b9c0fbec06de96300b5c"}
Sep 14 03:15:08.321967 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:08.321940    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9c9f1e-29822595-z4rrc"
Sep 14 03:15:09.191091 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:09.191063    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9c9f1e-29822595-z4rrc"
Sep 14 03:15:09.191274 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:09.191098    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9c9f1e-29822595-z4rrc" event={"ID":"a633a62e-b4c2-49d5-82ba-e6ae0b7d02ec","Type":"ContainerDied","Data":"5e001e0eb5ec166dbb1c7e44f00740fff1899b371fc47904dfcac2321053f646"}
Sep 14 03:15:09.191274 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:09.191152    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e001e0eb5ec166dbb1c7e44f00740fff1899b371fc47904dfcac2321053f646"
Sep 14 03:15:11.408471 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:11.408437    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:15:11.515416 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:11.515374    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:15:11.664533 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:11.664441    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:15:14.185168 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:14.185123    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-3a752f/embedding-w0-3a752f-kserve-65c77cc784-2gd2r"
Sep 14 03:15:14.185706 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:14.185194    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3a752f/embedding-w0-3a752f-kserve-65c77cc784-2gd2r"
Sep 14 03:15:14.198014 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:14.197983    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-3a752f/embedding-w0-3a752f-kserve-65c77cc784-2gd2r"
Sep 14 03:15:14.231195 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:14.231170    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3a752f/embedding-w0-3a752f-kserve-65c77cc784-2gd2r"
Sep 14 03:15:18.899267 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:18.899233    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:15:22.132347 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:22.132311    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:15:22.344531 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:22.344498    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:15:33.127784 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:33.127741    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:15:39.271701 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:39.271668    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:15:43.286916 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:43.286880    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:15:47.913890 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:47.913850    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:15:48.015469 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:48.015431    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:15:49.350796 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:49.350736    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:15:49.432471 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:49.432432    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:15:49.672450 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:49.672340    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:15:49.979956 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:49.979862    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:15:50.262123 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:50.262031    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:15:52.152143 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:52.152107    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:15:52.815911 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:52.815873    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:15:53.027306 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:53.027268    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:15:53.414529 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:53.414480    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:15:54.158053 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:54.158021    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:15:54.930588 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:54.930548    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:15:55.750723 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:55.750649    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:15:56.428620 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:56.428583    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:15:57.230607 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:57.230566    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:15:58.025306 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:58.025268    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3a33c1-55db8cd5c4-h87l7"]
Sep 14 03:15:58.026098 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:58.026072    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a633a62e-b4c2-49d5-82ba-e6ae0b7d02ec" containerName="cleanup"
Sep 14 03:15:58.026098 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:58.026098    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a633a62e-b4c2-49d5-82ba-e6ae0b7d02ec" containerName="cleanup"
Sep 14 03:15:58.026260 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:58.026229    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a633a62e-b4c2-49d5-82ba-e6ae0b7d02ec" containerName="cleanup"
Sep 14 03:15:58.033710 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:58.033675    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a33c1-55db8cd5c4-h87l7"
Sep 14 03:15:58.036673 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:58.036639    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-3a33c1-serving-cert\""
Sep 14 03:15:58.042172 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:58.042143    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3a33c1-55db8cd5c4-h87l7"]
Sep 14 03:15:58.059865 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:58.059823    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:15:58.091791 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:58.091748    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5437a89f-6ac4-46f0-909d-808ecf23e86f-metrics-certs\") pod \"maas-api-e2e-api-a-3a33c1-55db8cd5c4-h87l7\" (UID: \"5437a89f-6ac4-46f0-909d-808ecf23e86f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a33c1-55db8cd5c4-h87l7"
Sep 14 03:15:58.092002 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:58.091808    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq9k5\" (UniqueName: \"kubernetes.io/projected/5437a89f-6ac4-46f0-909d-808ecf23e86f-kube-api-access-wq9k5\") pod \"maas-api-e2e-api-a-3a33c1-55db8cd5c4-h87l7\" (UID: \"5437a89f-6ac4-46f0-909d-808ecf23e86f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a33c1-55db8cd5c4-h87l7"
Sep 14 03:15:58.092002 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:58.091842    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5437a89f-6ac4-46f0-909d-808ecf23e86f-maas-api-tls\") pod \"maas-api-e2e-api-a-3a33c1-55db8cd5c4-h87l7\" (UID: \"5437a89f-6ac4-46f0-909d-808ecf23e86f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a33c1-55db8cd5c4-h87l7"
Sep 14 03:15:58.092097 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:58.092003    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5437a89f-6ac4-46f0-909d-808ecf23e86f-openshift-service-ca\") pod \"maas-api-e2e-api-a-3a33c1-55db8cd5c4-h87l7\" (UID: \"5437a89f-6ac4-46f0-909d-808ecf23e86f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a33c1-55db8cd5c4-h87l7"
Sep 14 03:15:58.119090 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:58.119043    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:15:58.193018 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:58.192961    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq9k5\" (UniqueName: \"kubernetes.io/projected/5437a89f-6ac4-46f0-909d-808ecf23e86f-kube-api-access-wq9k5\") pod \"maas-api-e2e-api-a-3a33c1-55db8cd5c4-h87l7\" (UID: \"5437a89f-6ac4-46f0-909d-808ecf23e86f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a33c1-55db8cd5c4-h87l7"
Sep 14 03:15:58.193238 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:58.193032    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5437a89f-6ac4-46f0-909d-808ecf23e86f-maas-api-tls\") pod \"maas-api-e2e-api-a-3a33c1-55db8cd5c4-h87l7\" (UID: \"5437a89f-6ac4-46f0-909d-808ecf23e86f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a33c1-55db8cd5c4-h87l7"
Sep 14 03:15:58.193238 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:15:58.193213    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-3a33c1-serving-cert: secret "maas-api-e2e-api-a-3a33c1-serving-cert" not found
Sep 14 03:15:58.193356 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:58.193235    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5437a89f-6ac4-46f0-909d-808ecf23e86f-openshift-service-ca\") pod \"maas-api-e2e-api-a-3a33c1-55db8cd5c4-h87l7\" (UID: \"5437a89f-6ac4-46f0-909d-808ecf23e86f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a33c1-55db8cd5c4-h87l7"
Sep 14 03:15:58.193356 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:15:58.193323    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5437a89f-6ac4-46f0-909d-808ecf23e86f-maas-api-tls podName:5437a89f-6ac4-46f0-909d-808ecf23e86f nodeName:}" failed. No retries permitted until 2026-09-14 03:15:58.693298617 +0000 UTC m=+1404.074382806 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5437a89f-6ac4-46f0-909d-808ecf23e86f-maas-api-tls") pod "maas-api-e2e-api-a-3a33c1-55db8cd5c4-h87l7" (UID: "5437a89f-6ac4-46f0-909d-808ecf23e86f") : secret "maas-api-e2e-api-a-3a33c1-serving-cert" not found
Sep 14 03:15:58.193595 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:58.193568    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5437a89f-6ac4-46f0-909d-808ecf23e86f-metrics-certs\") pod \"maas-api-e2e-api-a-3a33c1-55db8cd5c4-h87l7\" (UID: \"5437a89f-6ac4-46f0-909d-808ecf23e86f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a33c1-55db8cd5c4-h87l7"
Sep 14 03:15:58.194008 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:58.193973    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5437a89f-6ac4-46f0-909d-808ecf23e86f-openshift-service-ca\") pod \"maas-api-e2e-api-a-3a33c1-55db8cd5c4-h87l7\" (UID: \"5437a89f-6ac4-46f0-909d-808ecf23e86f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a33c1-55db8cd5c4-h87l7"
Sep 14 03:15:58.196177 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:58.196154    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5437a89f-6ac4-46f0-909d-808ecf23e86f-metrics-certs\") pod \"maas-api-e2e-api-a-3a33c1-55db8cd5c4-h87l7\" (UID: \"5437a89f-6ac4-46f0-909d-808ecf23e86f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a33c1-55db8cd5c4-h87l7"
Sep 14 03:15:58.205033 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:58.205002    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq9k5\" (UniqueName: \"kubernetes.io/projected/5437a89f-6ac4-46f0-909d-808ecf23e86f-kube-api-access-wq9k5\") pod \"maas-api-e2e-api-a-3a33c1-55db8cd5c4-h87l7\" (UID: \"5437a89f-6ac4-46f0-909d-808ecf23e86f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a33c1-55db8cd5c4-h87l7"
Sep 14 03:15:58.536592 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:58.536556    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:15:58.646665 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:58.646631    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3a33c1-58cdf544cd-v2xw4"]
Sep 14 03:15:58.648581 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:58.648544    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:15:58.657254 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:58.652088    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3a33c1-58cdf544cd-v2xw4"
Sep 14 03:15:58.658632 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:58.658367    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-3a33c1-dockercfg-q47wv\""
Sep 14 03:15:58.658924 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:58.658367    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-3a33c1\""
Sep 14 03:15:58.661826 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:58.661798    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3a33c1-58cdf544cd-v2xw4"]
Sep 14 03:15:58.700215 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:58.700182    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5437a89f-6ac4-46f0-909d-808ecf23e86f-maas-api-tls\") pod \"maas-api-e2e-api-a-3a33c1-55db8cd5c4-h87l7\" (UID: \"5437a89f-6ac4-46f0-909d-808ecf23e86f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a33c1-55db8cd5c4-h87l7"
Sep 14 03:15:58.702967 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:58.702938    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5437a89f-6ac4-46f0-909d-808ecf23e86f-maas-api-tls\") pod \"maas-api-e2e-api-a-3a33c1-55db8cd5c4-h87l7\" (UID: \"5437a89f-6ac4-46f0-909d-808ecf23e86f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a33c1-55db8cd5c4-h87l7"
Sep 14 03:15:58.724841 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:58.724796    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:15:58.801879 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:58.801786    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrb7r\" (UniqueName: \"kubernetes.io/projected/d13e2896-2b10-4f6c-8583-1d60c1390838-kube-api-access-qrb7r\") pod \"payload-pre-processing-e2e-api-a-3a33c1-58cdf544cd-v2xw4\" (UID: \"d13e2896-2b10-4f6c-8583-1d60c1390838\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3a33c1-58cdf544cd-v2xw4"
Sep 14 03:15:58.801879 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:58.801860    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d13e2896-2b10-4f6c-8583-1d60c1390838-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-3a33c1-58cdf544cd-v2xw4\" (UID: \"d13e2896-2b10-4f6c-8583-1d60c1390838\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3a33c1-58cdf544cd-v2xw4"
Sep 14 03:15:58.902957 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:58.902910    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrb7r\" (UniqueName: \"kubernetes.io/projected/d13e2896-2b10-4f6c-8583-1d60c1390838-kube-api-access-qrb7r\") pod \"payload-pre-processing-e2e-api-a-3a33c1-58cdf544cd-v2xw4\" (UID: \"d13e2896-2b10-4f6c-8583-1d60c1390838\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3a33c1-58cdf544cd-v2xw4"
Sep 14 03:15:58.903148 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:58.903010    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d13e2896-2b10-4f6c-8583-1d60c1390838-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-3a33c1-58cdf544cd-v2xw4\" (UID: \"d13e2896-2b10-4f6c-8583-1d60c1390838\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3a33c1-58cdf544cd-v2xw4"
Sep 14 03:15:58.903820 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:58.903799    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d13e2896-2b10-4f6c-8583-1d60c1390838-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-3a33c1-58cdf544cd-v2xw4\" (UID: \"d13e2896-2b10-4f6c-8583-1d60c1390838\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3a33c1-58cdf544cd-v2xw4"
Sep 14 03:15:58.914359 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:58.914321    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrb7r\" (UniqueName: \"kubernetes.io/projected/d13e2896-2b10-4f6c-8583-1d60c1390838-kube-api-access-qrb7r\") pod \"payload-pre-processing-e2e-api-a-3a33c1-58cdf544cd-v2xw4\" (UID: \"d13e2896-2b10-4f6c-8583-1d60c1390838\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3a33c1-58cdf544cd-v2xw4"
Sep 14 03:15:58.947393 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:58.947352    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a33c1-55db8cd5c4-h87l7"
Sep 14 03:15:58.972964 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:58.972930    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3a33c1-58cdf544cd-v2xw4"
Sep 14 03:15:59.235432 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:59.235386    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3a33c1-55db8cd5c4-h87l7"]
Sep 14 03:15:59.237222 ip-10-0-141-4 kubenswrapper[2564]: W0914 03:15:59.237188    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5437a89f_6ac4_46f0_909d_808ecf23e86f.slice/crio-6c9403029d28f694bb26ab17e09fb44f242005de7ece674eb547522223e1bcfa WatchSource:0}: Error finding container 6c9403029d28f694bb26ab17e09fb44f242005de7ece674eb547522223e1bcfa: Status 404 returned error can't find the container with id 6c9403029d28f694bb26ab17e09fb44f242005de7ece674eb547522223e1bcfa
Sep 14 03:15:59.264000 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:59.263971    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3a33c1-58cdf544cd-v2xw4"]
Sep 14 03:15:59.265815 ip-10-0-141-4 kubenswrapper[2564]: W0914 03:15:59.265778    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd13e2896_2b10_4f6c_8583_1d60c1390838.slice/crio-6593b733665bfcc328de43ea6d917431b012d400393d6325f212d8d47c74881a WatchSource:0}: Error finding container 6593b733665bfcc328de43ea6d917431b012d400393d6325f212d8d47c74881a: Status 404 returned error can't find the container with id 6593b733665bfcc328de43ea6d917431b012d400393d6325f212d8d47c74881a
Sep 14 03:15:59.474860 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:59.474822    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3a33c1-58cdf544cd-v2xw4" event={"ID":"d13e2896-2b10-4f6c-8583-1d60c1390838","Type":"ContainerStarted","Data":"ced79a87a92353f3d89d8b28d34a1d1fdd9ff14d15222ef355c518d122416851"}
Sep 14 03:15:59.475084 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:59.475028    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3a33c1-58cdf544cd-v2xw4"
Sep 14 03:15:59.475138 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:59.475086    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3a33c1-58cdf544cd-v2xw4" event={"ID":"d13e2896-2b10-4f6c-8583-1d60c1390838","Type":"ContainerStarted","Data":"6593b733665bfcc328de43ea6d917431b012d400393d6325f212d8d47c74881a"}
Sep 14 03:15:59.476274 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:59.476247    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a33c1-55db8cd5c4-h87l7" event={"ID":"5437a89f-6ac4-46f0-909d-808ecf23e86f","Type":"ContainerStarted","Data":"6c9403029d28f694bb26ab17e09fb44f242005de7ece674eb547522223e1bcfa"}
Sep 14 03:15:59.492947 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:15:59.492891    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3a33c1-58cdf544cd-v2xw4" podStartSLOduration=1.492872079 podStartE2EDuration="1.492872079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:15:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:15:59.49116426 +0000 UTC m=+1404.872248451" watchObservedRunningTime="2026-09-14 03:15:59.492872079 +0000 UTC m=+1404.873956373"
Sep 14 03:16:00.484888 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:00.484841    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a33c1-55db8cd5c4-h87l7" event={"ID":"5437a89f-6ac4-46f0-909d-808ecf23e86f","Type":"ContainerStarted","Data":"fa2a0cf3f9fd0b381cf9d6e0aac8521cc273d01512e928464f3e75e59deb2348"}
Sep 14 03:16:00.485333 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:00.485083    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a33c1-55db8cd5c4-h87l7"
Sep 14 03:16:00.504832 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:00.504772    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a33c1-55db8cd5c4-h87l7" podStartSLOduration=2.078485011 podStartE2EDuration="2.504752508s" totalImagesPullingTime="426.267497ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:15:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:16:00.502533095 +0000 UTC m=+1405.883617299" watchObservedRunningTime="2026-09-14 03:16:00.504752508 +0000 UTC m=+1405.885836707"
Sep 14 03:16:04.929530 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:04.929495    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:16:06.496451 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:06.496425    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a33c1-55db8cd5c4-h87l7"
Sep 14 03:16:10.487349 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:10.487315    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3a33c1-58cdf544cd-v2xw4"
Sep 14 03:16:19.054846 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:19.054802    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:16:20.237899 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:20.237861    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:16:20.897887 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:20.897855    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:16:21.622708 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:21.622667    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:16:22.501654 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:22.501617    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:16:22.860008 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:22.859977    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:16:23.306037 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:23.306008    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:16:23.437394 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:23.437360    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:16:23.711625 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:23.711534    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:16:24.428622 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:24.428587    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:16:24.529056 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:24.529024    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:16:25.109013 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:25.108972    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:16:25.903611 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:25.903571    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:16:26.246755 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:26.246642    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:16:26.776961 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:26.776927    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:16:26.888277 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:26.888242    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:16:27.668178 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:27.668096    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:16:28.433511 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:28.433472    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:16:29.146977 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:29.146941    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:16:29.939054 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:29.939017    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:16:30.702137 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:30.702104    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:16:31.346698 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:31.346664    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-3a752f/embedding-w0-3a752f-kserve-65c77cc784-2gd2r"]
Sep 14 03:16:31.438983 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:31.438931    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:16:31.535268 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:31.535232    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:16:32.447557 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:32.447519    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:16:33.281266 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:33.281235    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-97da78-7b967d5785-ksvwm"]
Sep 14 03:16:33.285437 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:33.285418    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-97da78-7b967d5785-ksvwm"
Sep 14 03:16:33.288651 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:33.288628    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-97da78-serving-cert\""
Sep 14 03:16:33.301912 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:33.301883    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-97da78-7b967d5785-ksvwm"]
Sep 14 03:16:33.345831 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:33.345796    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:16:33.362900 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:33.362867    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwpk2\" (UniqueName: \"kubernetes.io/projected/49ef063e-3050-42ba-aa15-4fc3504d52ef-kube-api-access-bwpk2\") pod \"maas-api-e2e-api-b-97da78-7b967d5785-ksvwm\" (UID: \"49ef063e-3050-42ba-aa15-4fc3504d52ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-97da78-7b967d5785-ksvwm"
Sep 14 03:16:33.363063 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:33.362941    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49ef063e-3050-42ba-aa15-4fc3504d52ef-maas-api-tls\") pod \"maas-api-e2e-api-b-97da78-7b967d5785-ksvwm\" (UID: \"49ef063e-3050-42ba-aa15-4fc3504d52ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-97da78-7b967d5785-ksvwm"
Sep 14 03:16:33.363063 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:33.363025    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49ef063e-3050-42ba-aa15-4fc3504d52ef-metrics-certs\") pod \"maas-api-e2e-api-b-97da78-7b967d5785-ksvwm\" (UID: \"49ef063e-3050-42ba-aa15-4fc3504d52ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-97da78-7b967d5785-ksvwm"
Sep 14 03:16:33.363063 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:33.363056    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49ef063e-3050-42ba-aa15-4fc3504d52ef-openshift-service-ca\") pod \"maas-api-e2e-api-b-97da78-7b967d5785-ksvwm\" (UID: \"49ef063e-3050-42ba-aa15-4fc3504d52ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-97da78-7b967d5785-ksvwm"
Sep 14 03:16:33.463596 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:33.463559    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwpk2\" (UniqueName: \"kubernetes.io/projected/49ef063e-3050-42ba-aa15-4fc3504d52ef-kube-api-access-bwpk2\") pod \"maas-api-e2e-api-b-97da78-7b967d5785-ksvwm\" (UID: \"49ef063e-3050-42ba-aa15-4fc3504d52ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-97da78-7b967d5785-ksvwm"
Sep 14 03:16:33.464028 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:33.463635    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49ef063e-3050-42ba-aa15-4fc3504d52ef-maas-api-tls\") pod \"maas-api-e2e-api-b-97da78-7b967d5785-ksvwm\" (UID: \"49ef063e-3050-42ba-aa15-4fc3504d52ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-97da78-7b967d5785-ksvwm"
Sep 14 03:16:33.464028 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:33.463706    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49ef063e-3050-42ba-aa15-4fc3504d52ef-metrics-certs\") pod \"maas-api-e2e-api-b-97da78-7b967d5785-ksvwm\" (UID: \"49ef063e-3050-42ba-aa15-4fc3504d52ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-97da78-7b967d5785-ksvwm"
Sep 14 03:16:33.464028 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:33.463748    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49ef063e-3050-42ba-aa15-4fc3504d52ef-openshift-service-ca\") pod \"maas-api-e2e-api-b-97da78-7b967d5785-ksvwm\" (UID: \"49ef063e-3050-42ba-aa15-4fc3504d52ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-97da78-7b967d5785-ksvwm"
Sep 14 03:16:33.464028 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:16:33.463782    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-97da78-serving-cert: secret "maas-api-e2e-api-b-97da78-serving-cert" not found
Sep 14 03:16:33.464028 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:16:33.463862    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49ef063e-3050-42ba-aa15-4fc3504d52ef-maas-api-tls podName:49ef063e-3050-42ba-aa15-4fc3504d52ef nodeName:}" failed. No retries permitted until 2026-09-14 03:16:33.963843776 +0000 UTC m=+1439.344927964 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/49ef063e-3050-42ba-aa15-4fc3504d52ef-maas-api-tls") pod "maas-api-e2e-api-b-97da78-7b967d5785-ksvwm" (UID: "49ef063e-3050-42ba-aa15-4fc3504d52ef") : secret "maas-api-e2e-api-b-97da78-serving-cert" not found
Sep 14 03:16:33.464395 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:33.464373    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49ef063e-3050-42ba-aa15-4fc3504d52ef-openshift-service-ca\") pod \"maas-api-e2e-api-b-97da78-7b967d5785-ksvwm\" (UID: \"49ef063e-3050-42ba-aa15-4fc3504d52ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-97da78-7b967d5785-ksvwm"
Sep 14 03:16:33.466152 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:33.466130    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49ef063e-3050-42ba-aa15-4fc3504d52ef-metrics-certs\") pod \"maas-api-e2e-api-b-97da78-7b967d5785-ksvwm\" (UID: \"49ef063e-3050-42ba-aa15-4fc3504d52ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-97da78-7b967d5785-ksvwm"
Sep 14 03:16:33.483394 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:33.483365    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwpk2\" (UniqueName: \"kubernetes.io/projected/49ef063e-3050-42ba-aa15-4fc3504d52ef-kube-api-access-bwpk2\") pod \"maas-api-e2e-api-b-97da78-7b967d5785-ksvwm\" (UID: \"49ef063e-3050-42ba-aa15-4fc3504d52ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-97da78-7b967d5785-ksvwm"
Sep 14 03:16:33.524270 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:33.524238    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:16:33.884681 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:33.884650    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:16:33.968171 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:33.968138    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49ef063e-3050-42ba-aa15-4fc3504d52ef-maas-api-tls\") pod \"maas-api-e2e-api-b-97da78-7b967d5785-ksvwm\" (UID: \"49ef063e-3050-42ba-aa15-4fc3504d52ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-97da78-7b967d5785-ksvwm"
Sep 14 03:16:33.970916 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:33.970884    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49ef063e-3050-42ba-aa15-4fc3504d52ef-maas-api-tls\") pod \"maas-api-e2e-api-b-97da78-7b967d5785-ksvwm\" (UID: \"49ef063e-3050-42ba-aa15-4fc3504d52ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-97da78-7b967d5785-ksvwm"
Sep 14 03:16:34.036343 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:34.036309    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:16:34.132682 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:34.132648    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:16:34.198504 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:34.198430    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-97da78-7b967d5785-ksvwm"
Sep 14 03:16:34.479977 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:34.479952    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-97da78-7b967d5785-ksvwm"]
Sep 14 03:16:34.481012 ip-10-0-141-4 kubenswrapper[2564]: W0914 03:16:34.480988    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49ef063e_3050_42ba_aa15_4fc3504d52ef.slice/crio-d27dffadbefb081e14edc19fa51db43d139d610bd157f7c7dc9db730de3f9ea2 WatchSource:0}: Error finding container d27dffadbefb081e14edc19fa51db43d139d610bd157f7c7dc9db730de3f9ea2: Status 404 returned error can't find the container with id d27dffadbefb081e14edc19fa51db43d139d610bd157f7c7dc9db730de3f9ea2
Sep 14 03:16:34.677710 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:34.677680    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-97da78-7b967d5785-ksvwm" event={"ID":"49ef063e-3050-42ba-aa15-4fc3504d52ef","Type":"ContainerStarted","Data":"d27dffadbefb081e14edc19fa51db43d139d610bd157f7c7dc9db730de3f9ea2"}
Sep 14 03:16:35.693027 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:35.692985    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-97da78-7b967d5785-ksvwm" event={"ID":"49ef063e-3050-42ba-aa15-4fc3504d52ef","Type":"ContainerStarted","Data":"d93f37c3ec35bcc355a970a069c44a6b6fd33ff9e6c6465870dba7fc699347ce"}
Sep 14 03:16:35.693512 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:35.693100    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-97da78-7b967d5785-ksvwm"
Sep 14 03:16:35.712955 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:35.712898    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-97da78-7b967d5785-ksvwm" podStartSLOduration=2.329025163 podStartE2EDuration="2.712878837s" totalImagesPullingTime="383.853674ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:16:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:16:35.712172872 +0000 UTC m=+1441.093257060" watchObservedRunningTime="2026-09-14 03:16:35.712878837 +0000 UTC m=+1441.093963036"
Sep 14 03:16:36.540104 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:36.540072    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:16:39.637890 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:39.637844    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:16:41.703609 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:41.703577    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-97da78-7b967d5785-ksvwm"
Sep 14 03:16:42.140906 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:42.140869    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-3a752f-549955bb4-tbwgt"]
Sep 14 03:16:42.141152 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:42.141130    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3a752f-549955bb4-tbwgt" podUID="ac41e03a-54d4-4b59-8e31-563773bd52ba" containerName="maas-api" containerID="cri-o://03af6e426657d52129ffd13ad8924df01509bd4c9ce41c8a1033ada23e2b0f24" gracePeriod=30
Sep 14 03:16:42.173619 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:42.173583    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:16:42.212542 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:42.212511    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-3a752f-776b9b9d9f-42gsk"]
Sep 14 03:16:42.212757 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:42.212725    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-3a752f-776b9b9d9f-42gsk" podUID="242753da-219f-47d3-ad2e-f3866269b610" containerName="payload-processing" containerID="cri-o://67a125b822416c23bf4c673a47f1b502d9ccf0ae17194e9dcc1c32016d2d5919" gracePeriod=30
Sep 14 03:16:42.229422 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:42.229374    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-3a752f-69bdf4d6b-gl7qz"]
Sep 14 03:16:42.229667 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:42.229635    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3a752f-69bdf4d6b-gl7qz" podUID="7a69a896-145c-4365-9444-3ea9f537ae6b" containerName="payload-pre-processing" containerID="cri-o://59619c12a0230288a092966749ba6752ae76ff009d27e195d20f856702d8657a" gracePeriod=30
Sep 14 03:16:42.262983 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:42.262947    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:16:42.339159 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:42.339124    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:16:42.438334 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:42.438297    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:16:42.496980 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:42.495830    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3a752f-549955bb4-tbwgt"
Sep 14 03:16:42.536759 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:42.536737    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-3a752f-776b9b9d9f-42gsk"
Sep 14 03:16:42.544313 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:42.544273    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3a752f-69bdf4d6b-gl7qz"
Sep 14 03:16:42.658360 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:42.658272    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xpds2\" (UniqueName: \"kubernetes.io/projected/7a69a896-145c-4365-9444-3ea9f537ae6b-kube-api-access-xpds2\") pod \"7a69a896-145c-4365-9444-3ea9f537ae6b\" (UID: \"7a69a896-145c-4365-9444-3ea9f537ae6b\") "
Sep 14 03:16:42.658360 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:42.658316    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a69a896-145c-4365-9444-3ea9f537ae6b-plugins-config-volume\") pod \"7a69a896-145c-4365-9444-3ea9f537ae6b\" (UID: \"7a69a896-145c-4365-9444-3ea9f537ae6b\") "
Sep 14 03:16:42.658635 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:42.658364    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac41e03a-54d4-4b59-8e31-563773bd52ba-openshift-service-ca\") pod \"ac41e03a-54d4-4b59-8e31-563773bd52ba\" (UID: \"ac41e03a-54d4-4b59-8e31-563773bd52ba\") "
Sep 14 03:16:42.658635 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:42.658414    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-srlzf\" (UniqueName: \"kubernetes.io/projected/242753da-219f-47d3-ad2e-f3866269b610-kube-api-access-srlzf\") pod \"242753da-219f-47d3-ad2e-f3866269b610\" (UID: \"242753da-219f-47d3-ad2e-f3866269b610\") "
Sep 14 03:16:42.658635 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:42.658524    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac41e03a-54d4-4b59-8e31-563773bd52ba-metrics-certs\") pod \"ac41e03a-54d4-4b59-8e31-563773bd52ba\" (UID: \"ac41e03a-54d4-4b59-8e31-563773bd52ba\") "
Sep 14 03:16:42.658635 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:42.658582    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2l2md\" (UniqueName: \"kubernetes.io/projected/ac41e03a-54d4-4b59-8e31-563773bd52ba-kube-api-access-2l2md\") pod \"ac41e03a-54d4-4b59-8e31-563773bd52ba\" (UID: \"ac41e03a-54d4-4b59-8e31-563773bd52ba\") "
Sep 14 03:16:42.658635 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:42.658622    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac41e03a-54d4-4b59-8e31-563773bd52ba-maas-api-tls\") pod \"ac41e03a-54d4-4b59-8e31-563773bd52ba\" (UID: \"ac41e03a-54d4-4b59-8e31-563773bd52ba\") "
Sep 14 03:16:42.658904 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:42.658666    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/242753da-219f-47d3-ad2e-f3866269b610-plugins-config-volume\") pod \"242753da-219f-47d3-ad2e-f3866269b610\" (UID: \"242753da-219f-47d3-ad2e-f3866269b610\") "
Sep 14 03:16:42.658904 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:42.658829    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a69a896-145c-4365-9444-3ea9f537ae6b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7a69a896-145c-4365-9444-3ea9f537ae6b" (UID: "7a69a896-145c-4365-9444-3ea9f537ae6b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 03:16:42.659161 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:42.659130    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac41e03a-54d4-4b59-8e31-563773bd52ba-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ac41e03a-54d4-4b59-8e31-563773bd52ba" (UID: "ac41e03a-54d4-4b59-8e31-563773bd52ba"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 03:16:42.659240 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:42.659145    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a69a896-145c-4365-9444-3ea9f537ae6b-plugins-config-volume\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:16:42.659300 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:42.659259    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/242753da-219f-47d3-ad2e-f3866269b610-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "242753da-219f-47d3-ad2e-f3866269b610" (UID: "242753da-219f-47d3-ad2e-f3866269b610"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 03:16:42.661036 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:42.661008    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac41e03a-54d4-4b59-8e31-563773bd52ba-kube-api-access-2l2md" (OuterVolumeSpecName: "kube-api-access-2l2md") pod "ac41e03a-54d4-4b59-8e31-563773bd52ba" (UID: "ac41e03a-54d4-4b59-8e31-563773bd52ba"). InnerVolumeSpecName "kube-api-access-2l2md". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:16:42.661226 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:42.661195    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/242753da-219f-47d3-ad2e-f3866269b610-kube-api-access-srlzf" (OuterVolumeSpecName: "kube-api-access-srlzf") pod "242753da-219f-47d3-ad2e-f3866269b610" (UID: "242753da-219f-47d3-ad2e-f3866269b610"). InnerVolumeSpecName "kube-api-access-srlzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:16:42.661325 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:42.661298    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a69a896-145c-4365-9444-3ea9f537ae6b-kube-api-access-xpds2" (OuterVolumeSpecName: "kube-api-access-xpds2") pod "7a69a896-145c-4365-9444-3ea9f537ae6b" (UID: "7a69a896-145c-4365-9444-3ea9f537ae6b"). InnerVolumeSpecName "kube-api-access-xpds2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:16:42.661457 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:42.661432    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac41e03a-54d4-4b59-8e31-563773bd52ba-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ac41e03a-54d4-4b59-8e31-563773bd52ba" (UID: "ac41e03a-54d4-4b59-8e31-563773bd52ba"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 03:16:42.661524 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:42.661449    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac41e03a-54d4-4b59-8e31-563773bd52ba-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ac41e03a-54d4-4b59-8e31-563773bd52ba" (UID: "ac41e03a-54d4-4b59-8e31-563773bd52ba"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 03:16:42.737881 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:42.737845    2564 generic.go:356] "Generic (PLEG): container finished" podID="ac41e03a-54d4-4b59-8e31-563773bd52ba" containerID="03af6e426657d52129ffd13ad8924df01509bd4c9ce41c8a1033ada23e2b0f24" exitCode=0
Sep 14 03:16:42.738346 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:42.737900    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3a752f-549955bb4-tbwgt"
Sep 14 03:16:42.738346 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:42.737938    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3a752f-549955bb4-tbwgt" event={"ID":"ac41e03a-54d4-4b59-8e31-563773bd52ba","Type":"ContainerDied","Data":"03af6e426657d52129ffd13ad8924df01509bd4c9ce41c8a1033ada23e2b0f24"}
Sep 14 03:16:42.738346 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:42.737993    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3a752f-549955bb4-tbwgt" event={"ID":"ac41e03a-54d4-4b59-8e31-563773bd52ba","Type":"ContainerDied","Data":"d5e67aab31cae3863a41a0a7ac9a24bef4717c7e6dd6c1eef7729fc693628d31"}
Sep 14 03:16:42.738346 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:42.738015    2564 scope.go:117] "RemoveContainer" containerID="03af6e426657d52129ffd13ad8924df01509bd4c9ce41c8a1033ada23e2b0f24"
Sep 14 03:16:42.739325 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:42.739305    2564 generic.go:356] "Generic (PLEG): container finished" podID="242753da-219f-47d3-ad2e-f3866269b610" containerID="67a125b822416c23bf4c673a47f1b502d9ccf0ae17194e9dcc1c32016d2d5919" exitCode=0
Sep 14 03:16:42.739441 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:42.739351    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-3a752f-776b9b9d9f-42gsk"
Sep 14 03:16:42.739441 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:42.739394    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-3a752f-776b9b9d9f-42gsk" event={"ID":"242753da-219f-47d3-ad2e-f3866269b610","Type":"ContainerDied","Data":"67a125b822416c23bf4c673a47f1b502d9ccf0ae17194e9dcc1c32016d2d5919"}
Sep 14 03:16:42.739554 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:42.739457    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-3a752f-776b9b9d9f-42gsk" event={"ID":"242753da-219f-47d3-ad2e-f3866269b610","Type":"ContainerDied","Data":"5ad1ef1afabdb4364281fad4d141be3e63d95b0193bc1dce862965c1c4313c4e"}
Sep 14 03:16:42.740844 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:42.740824    2564 generic.go:356] "Generic (PLEG): container finished" podID="7a69a896-145c-4365-9444-3ea9f537ae6b" containerID="59619c12a0230288a092966749ba6752ae76ff009d27e195d20f856702d8657a" exitCode=0
Sep 14 03:16:42.740939 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:42.740853    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3a752f-69bdf4d6b-gl7qz"
Sep 14 03:16:42.740939 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:42.740887    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3a752f-69bdf4d6b-gl7qz" event={"ID":"7a69a896-145c-4365-9444-3ea9f537ae6b","Type":"ContainerDied","Data":"59619c12a0230288a092966749ba6752ae76ff009d27e195d20f856702d8657a"}
Sep 14 03:16:42.740939 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:42.740929    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3a752f-69bdf4d6b-gl7qz" event={"ID":"7a69a896-145c-4365-9444-3ea9f537ae6b","Type":"ContainerDied","Data":"f8510908bf627df00199c1ec51187a4c144db322c755149edb65cc63bd79bea4"}
Sep 14 03:16:42.750613 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:42.750578    2564 scope.go:117] "RemoveContainer" containerID="03af6e426657d52129ffd13ad8924df01509bd4c9ce41c8a1033ada23e2b0f24"
Sep 14 03:16:42.750911 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:16:42.750891    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03af6e426657d52129ffd13ad8924df01509bd4c9ce41c8a1033ada23e2b0f24\": container with ID starting with 03af6e426657d52129ffd13ad8924df01509bd4c9ce41c8a1033ada23e2b0f24 not found: ID does not exist" containerID="03af6e426657d52129ffd13ad8924df01509bd4c9ce41c8a1033ada23e2b0f24"
Sep 14 03:16:42.750988 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:42.750917    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03af6e426657d52129ffd13ad8924df01509bd4c9ce41c8a1033ada23e2b0f24"} err="failed to get container status \"03af6e426657d52129ffd13ad8924df01509bd4c9ce41c8a1033ada23e2b0f24\": rpc error: code = NotFound desc = could not find container \"03af6e426657d52129ffd13ad8924df01509bd4c9ce41c8a1033ada23e2b0f24\": container with ID starting with 03af6e426657d52129ffd13ad8924df01509bd4c9ce41c8a1033ada23e2b0f24 not found: ID does not exist"
Sep 14 03:16:42.750988 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:42.750933    2564 scope.go:117] "RemoveContainer" containerID="67a125b822416c23bf4c673a47f1b502d9ccf0ae17194e9dcc1c32016d2d5919"
Sep 14 03:16:42.759792 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:42.759767    2564 scope.go:117] "RemoveContainer" containerID="67a125b822416c23bf4c673a47f1b502d9ccf0ae17194e9dcc1c32016d2d5919"
Sep 14 03:16:42.759962 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:42.759940    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2l2md\" (UniqueName: \"kubernetes.io/projected/ac41e03a-54d4-4b59-8e31-563773bd52ba-kube-api-access-2l2md\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:16:42.760022 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:42.759968    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac41e03a-54d4-4b59-8e31-563773bd52ba-maas-api-tls\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:16:42.760022 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:42.759980    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/242753da-219f-47d3-ad2e-f3866269b610-plugins-config-volume\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:16:42.760022 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:42.759991    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpds2\" (UniqueName: \"kubernetes.io/projected/7a69a896-145c-4365-9444-3ea9f537ae6b-kube-api-access-xpds2\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:16:42.760022 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:42.760004    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac41e03a-54d4-4b59-8e31-563773bd52ba-openshift-service-ca\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:16:42.760022 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:42.760015    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srlzf\" (UniqueName: \"kubernetes.io/projected/242753da-219f-47d3-ad2e-f3866269b610-kube-api-access-srlzf\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:16:42.760207 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:42.760025    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac41e03a-54d4-4b59-8e31-563773bd52ba-metrics-certs\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:16:42.760207 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:16:42.760066    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67a125b822416c23bf4c673a47f1b502d9ccf0ae17194e9dcc1c32016d2d5919\": container with ID starting with 67a125b822416c23bf4c673a47f1b502d9ccf0ae17194e9dcc1c32016d2d5919 not found: ID does not exist" containerID="67a125b822416c23bf4c673a47f1b502d9ccf0ae17194e9dcc1c32016d2d5919"
Sep 14 03:16:42.760207 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:42.760095    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67a125b822416c23bf4c673a47f1b502d9ccf0ae17194e9dcc1c32016d2d5919"} err="failed to get container status \"67a125b822416c23bf4c673a47f1b502d9ccf0ae17194e9dcc1c32016d2d5919\": rpc error: code = NotFound desc = could not find container \"67a125b822416c23bf4c673a47f1b502d9ccf0ae17194e9dcc1c32016d2d5919\": container with ID starting with 67a125b822416c23bf4c673a47f1b502d9ccf0ae17194e9dcc1c32016d2d5919 not found: ID does not exist"
Sep 14 03:16:42.760207 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:42.760118    2564 scope.go:117] "RemoveContainer" containerID="59619c12a0230288a092966749ba6752ae76ff009d27e195d20f856702d8657a"
Sep 14 03:16:42.768291 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:42.768273    2564 scope.go:117] "RemoveContainer" containerID="59619c12a0230288a092966749ba6752ae76ff009d27e195d20f856702d8657a"
Sep 14 03:16:42.768580 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:16:42.768562    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59619c12a0230288a092966749ba6752ae76ff009d27e195d20f856702d8657a\": container with ID starting with 59619c12a0230288a092966749ba6752ae76ff009d27e195d20f856702d8657a not found: ID does not exist" containerID="59619c12a0230288a092966749ba6752ae76ff009d27e195d20f856702d8657a"
Sep 14 03:16:42.768640 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:42.768587    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59619c12a0230288a092966749ba6752ae76ff009d27e195d20f856702d8657a"} err="failed to get container status \"59619c12a0230288a092966749ba6752ae76ff009d27e195d20f856702d8657a\": rpc error: code = NotFound desc = could not find container \"59619c12a0230288a092966749ba6752ae76ff009d27e195d20f856702d8657a\": container with ID starting with 59619c12a0230288a092966749ba6752ae76ff009d27e195d20f856702d8657a not found: ID does not exist"
Sep 14 03:16:42.775231 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:42.775207    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-3a752f-549955bb4-tbwgt"]
Sep 14 03:16:42.786805 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:42.786782    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-3a752f-549955bb4-tbwgt"]
Sep 14 03:16:42.809250 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:42.809226    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-3a752f-776b9b9d9f-42gsk"]
Sep 14 03:16:42.815548 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:42.815528    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-3a752f-776b9b9d9f-42gsk"]
Sep 14 03:16:42.820059 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:42.820029    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:16:42.829321 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:42.829303    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-3a752f-69bdf4d6b-gl7qz"]
Sep 14 03:16:42.836075 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:42.836057    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-3a752f-69bdf4d6b-gl7qz"]
Sep 14 03:16:42.907628 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:42.907593    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:16:43.099528 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:43.099494    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="242753da-219f-47d3-ad2e-f3866269b610" path="/var/lib/kubelet/pods/242753da-219f-47d3-ad2e-f3866269b610/volumes"
Sep 14 03:16:43.099961 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:43.099940    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a69a896-145c-4365-9444-3ea9f537ae6b" path="/var/lib/kubelet/pods/7a69a896-145c-4365-9444-3ea9f537ae6b/volumes"
Sep 14 03:16:43.100371 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:43.100353    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac41e03a-54d4-4b59-8e31-563773bd52ba" path="/var/lib/kubelet/pods/ac41e03a-54d4-4b59-8e31-563773bd52ba/volumes"
Sep 14 03:16:43.820518 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:43.820488    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:16:44.536862 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:44.536821    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:16:45.233142 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:45.233104    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:16:46.154046 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:46.154008    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:16:46.368544 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:46.368505    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-3a752f/embedding-w0-3a752f-kserve-65c77cc784-2gd2r" podUID="d9a686f2-3fa7-4b42-b237-53be6ba2b050" containerName="main" containerID="cri-o://b5299b825348b563b36a8b2701026e1e36bde22ade897339a3ff0cead5376051" gracePeriod=45
Sep 14 03:16:46.566737 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:46.566699    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:16:46.624844 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:46.624817    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3a752f/embedding-w0-3a752f-kserve-65c77cc784-2gd2r"
Sep 14 03:16:46.698280 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:46.698247    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9a686f2-3fa7-4b42-b237-53be6ba2b050-dshm\") pod \"d9a686f2-3fa7-4b42-b237-53be6ba2b050\" (UID: \"d9a686f2-3fa7-4b42-b237-53be6ba2b050\") "
Sep 14 03:16:46.698504 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:46.698291    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9a686f2-3fa7-4b42-b237-53be6ba2b050-home\") pod \"d9a686f2-3fa7-4b42-b237-53be6ba2b050\" (UID: \"d9a686f2-3fa7-4b42-b237-53be6ba2b050\") "
Sep 14 03:16:46.698504 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:46.698317    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lw69q\" (UniqueName: \"kubernetes.io/projected/d9a686f2-3fa7-4b42-b237-53be6ba2b050-kube-api-access-lw69q\") pod \"d9a686f2-3fa7-4b42-b237-53be6ba2b050\" (UID: \"d9a686f2-3fa7-4b42-b237-53be6ba2b050\") "
Sep 14 03:16:46.698504 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:46.698351    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9a686f2-3fa7-4b42-b237-53be6ba2b050-model-cache\") pod \"d9a686f2-3fa7-4b42-b237-53be6ba2b050\" (UID: \"d9a686f2-3fa7-4b42-b237-53be6ba2b050\") "
Sep 14 03:16:46.698504 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:46.698442    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9a686f2-3fa7-4b42-b237-53be6ba2b050-tmp-dir\") pod \"d9a686f2-3fa7-4b42-b237-53be6ba2b050\" (UID: \"d9a686f2-3fa7-4b42-b237-53be6ba2b050\") "
Sep 14 03:16:46.698504 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:46.698496    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9a686f2-3fa7-4b42-b237-53be6ba2b050-tls-certs\") pod \"d9a686f2-3fa7-4b42-b237-53be6ba2b050\" (UID: \"d9a686f2-3fa7-4b42-b237-53be6ba2b050\") "
Sep 14 03:16:46.698774 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:46.698653    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9a686f2-3fa7-4b42-b237-53be6ba2b050-home" (OuterVolumeSpecName: "home") pod "d9a686f2-3fa7-4b42-b237-53be6ba2b050" (UID: "d9a686f2-3fa7-4b42-b237-53be6ba2b050"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:16:46.698774 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:46.698761    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9a686f2-3fa7-4b42-b237-53be6ba2b050-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d9a686f2-3fa7-4b42-b237-53be6ba2b050" (UID: "d9a686f2-3fa7-4b42-b237-53be6ba2b050"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:16:46.698894 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:46.698780    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9a686f2-3fa7-4b42-b237-53be6ba2b050-model-cache" (OuterVolumeSpecName: "model-cache") pod "d9a686f2-3fa7-4b42-b237-53be6ba2b050" (UID: "d9a686f2-3fa7-4b42-b237-53be6ba2b050"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:16:46.699356 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:46.699331    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9a686f2-3fa7-4b42-b237-53be6ba2b050-home\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:16:46.699356 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:46.699357    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9a686f2-3fa7-4b42-b237-53be6ba2b050-model-cache\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:16:46.699589 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:46.699373    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9a686f2-3fa7-4b42-b237-53be6ba2b050-tmp-dir\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:16:46.700957 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:46.700918    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9a686f2-3fa7-4b42-b237-53be6ba2b050-dshm" (OuterVolumeSpecName: "dshm") pod "d9a686f2-3fa7-4b42-b237-53be6ba2b050" (UID: "d9a686f2-3fa7-4b42-b237-53be6ba2b050"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:16:46.701072 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:46.700993    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9a686f2-3fa7-4b42-b237-53be6ba2b050-kube-api-access-lw69q" (OuterVolumeSpecName: "kube-api-access-lw69q") pod "d9a686f2-3fa7-4b42-b237-53be6ba2b050" (UID: "d9a686f2-3fa7-4b42-b237-53be6ba2b050"). InnerVolumeSpecName "kube-api-access-lw69q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:16:46.701072 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:46.701066    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9a686f2-3fa7-4b42-b237-53be6ba2b050-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d9a686f2-3fa7-4b42-b237-53be6ba2b050" (UID: "d9a686f2-3fa7-4b42-b237-53be6ba2b050"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 03:16:46.773064 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:46.773033    2564 generic.go:356] "Generic (PLEG): container finished" podID="d9a686f2-3fa7-4b42-b237-53be6ba2b050" containerID="b5299b825348b563b36a8b2701026e1e36bde22ade897339a3ff0cead5376051" exitCode=0
Sep 14 03:16:46.773244 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:46.773074    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3a752f/embedding-w0-3a752f-kserve-65c77cc784-2gd2r"
Sep 14 03:16:46.773244 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:46.773116    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3a752f/embedding-w0-3a752f-kserve-65c77cc784-2gd2r" event={"ID":"d9a686f2-3fa7-4b42-b237-53be6ba2b050","Type":"ContainerDied","Data":"b5299b825348b563b36a8b2701026e1e36bde22ade897339a3ff0cead5376051"}
Sep 14 03:16:46.773244 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:46.773157    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3a752f/embedding-w0-3a752f-kserve-65c77cc784-2gd2r" event={"ID":"d9a686f2-3fa7-4b42-b237-53be6ba2b050","Type":"ContainerDied","Data":"5f62e04dd82e33826398439bfeed93346b395593ad0b3975fa32f7fdd5813f01"}
Sep 14 03:16:46.773244 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:46.773175    2564 scope.go:117] "RemoveContainer" containerID="b5299b825348b563b36a8b2701026e1e36bde22ade897339a3ff0cead5376051"
Sep 14 03:16:46.785144 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:46.785119    2564 scope.go:117] "RemoveContainer" containerID="b5299b825348b563b36a8b2701026e1e36bde22ade897339a3ff0cead5376051"
Sep 14 03:16:46.785588 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:16:46.785560    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5299b825348b563b36a8b2701026e1e36bde22ade897339a3ff0cead5376051\": container with ID starting with b5299b825348b563b36a8b2701026e1e36bde22ade897339a3ff0cead5376051 not found: ID does not exist" containerID="b5299b825348b563b36a8b2701026e1e36bde22ade897339a3ff0cead5376051"
Sep 14 03:16:46.785668 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:46.785599    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5299b825348b563b36a8b2701026e1e36bde22ade897339a3ff0cead5376051"} err="failed to get container status \"b5299b825348b563b36a8b2701026e1e36bde22ade897339a3ff0cead5376051\": rpc error: code = NotFound desc = could not find container \"b5299b825348b563b36a8b2701026e1e36bde22ade897339a3ff0cead5376051\": container with ID starting with b5299b825348b563b36a8b2701026e1e36bde22ade897339a3ff0cead5376051 not found: ID does not exist"
Sep 14 03:16:46.800967 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:46.800944    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9a686f2-3fa7-4b42-b237-53be6ba2b050-tls-certs\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:16:46.800967 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:46.800964    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9a686f2-3fa7-4b42-b237-53be6ba2b050-dshm\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:16:46.801122 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:46.800975    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lw69q\" (UniqueName: \"kubernetes.io/projected/d9a686f2-3fa7-4b42-b237-53be6ba2b050-kube-api-access-lw69q\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:16:46.803076 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:46.803056    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-3a752f/embedding-w0-3a752f-kserve-65c77cc784-2gd2r"]
Sep 14 03:16:46.808125 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:46.808100    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-3a752f/embedding-w0-3a752f-kserve-65c77cc784-2gd2r"]
Sep 14 03:16:46.948789 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:46.948752    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:16:47.100125 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:47.100047    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9a686f2-3fa7-4b42-b237-53be6ba2b050" path="/var/lib/kubelet/pods/d9a686f2-3fa7-4b42-b237-53be6ba2b050/volumes"
Sep 14 03:16:47.869989 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:47.869956    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:16:48.641186 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:48.641124    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:16:49.362605 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:49.362555    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:16:49.456988 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:49.456937    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:16:49.762665 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:49.762574    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:16:49.985915 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:49.985874    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:16:50.665146 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:50.665107    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:16:51.482092 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:51.482060    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:16:52.230762 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:52.230724    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:16:53.002562 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:53.002530    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:16:53.735551 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:53.735516    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:16:54.452148 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:54.452115    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:16:55.149762 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:55.149728    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:16:56.146325 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:56.146283    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:16:59.743784 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:16:59.743747    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:17:05.445391 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:05.445348    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3a33c1-55db8cd5c4-h87l7"]
Sep 14 03:17:05.445898 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:05.445774    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a33c1-55db8cd5c4-h87l7" podUID="5437a89f-6ac4-46f0-909d-808ecf23e86f" containerName="maas-api" containerID="cri-o://fa2a0cf3f9fd0b381cf9d6e0aac8521cc273d01512e928464f3e75e59deb2348" gracePeriod=30
Sep 14 03:17:05.518257 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:05.518221    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3a33c1-58cdf544cd-v2xw4"]
Sep 14 03:17:05.518516 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:05.518490    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3a33c1-58cdf544cd-v2xw4" podUID="d13e2896-2b10-4f6c-8583-1d60c1390838" containerName="payload-pre-processing" containerID="cri-o://ced79a87a92353f3d89d8b28d34a1d1fdd9ff14d15222ef355c518d122416851" gracePeriod=30
Sep 14 03:17:05.794750 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:05.794719    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a33c1-55db8cd5c4-h87l7"
Sep 14 03:17:05.828527 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:05.828495    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3a33c1-58cdf544cd-v2xw4"
Sep 14 03:17:05.888278 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:05.888242    2564 generic.go:356] "Generic (PLEG): container finished" podID="5437a89f-6ac4-46f0-909d-808ecf23e86f" containerID="fa2a0cf3f9fd0b381cf9d6e0aac8521cc273d01512e928464f3e75e59deb2348" exitCode=0
Sep 14 03:17:05.888497 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:05.888288    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a33c1-55db8cd5c4-h87l7"
Sep 14 03:17:05.888497 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:05.888335    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a33c1-55db8cd5c4-h87l7" event={"ID":"5437a89f-6ac4-46f0-909d-808ecf23e86f","Type":"ContainerDied","Data":"fa2a0cf3f9fd0b381cf9d6e0aac8521cc273d01512e928464f3e75e59deb2348"}
Sep 14 03:17:05.888497 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:05.888434    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a33c1-55db8cd5c4-h87l7" event={"ID":"5437a89f-6ac4-46f0-909d-808ecf23e86f","Type":"ContainerDied","Data":"6c9403029d28f694bb26ab17e09fb44f242005de7ece674eb547522223e1bcfa"}
Sep 14 03:17:05.888497 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:05.888467    2564 scope.go:117] "RemoveContainer" containerID="fa2a0cf3f9fd0b381cf9d6e0aac8521cc273d01512e928464f3e75e59deb2348"
Sep 14 03:17:05.889946 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:05.889920    2564 generic.go:356] "Generic (PLEG): container finished" podID="d13e2896-2b10-4f6c-8583-1d60c1390838" containerID="ced79a87a92353f3d89d8b28d34a1d1fdd9ff14d15222ef355c518d122416851" exitCode=0
Sep 14 03:17:05.890054 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:05.890007    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3a33c1-58cdf544cd-v2xw4"
Sep 14 03:17:05.890054 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:05.890039    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3a33c1-58cdf544cd-v2xw4" event={"ID":"d13e2896-2b10-4f6c-8583-1d60c1390838","Type":"ContainerDied","Data":"ced79a87a92353f3d89d8b28d34a1d1fdd9ff14d15222ef355c518d122416851"}
Sep 14 03:17:05.890177 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:05.890079    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3a33c1-58cdf544cd-v2xw4" event={"ID":"d13e2896-2b10-4f6c-8583-1d60c1390838","Type":"ContainerDied","Data":"6593b733665bfcc328de43ea6d917431b012d400393d6325f212d8d47c74881a"}
Sep 14 03:17:05.892258 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:05.892228    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5437a89f-6ac4-46f0-909d-808ecf23e86f-metrics-certs\") pod \"5437a89f-6ac4-46f0-909d-808ecf23e86f\" (UID: \"5437a89f-6ac4-46f0-909d-808ecf23e86f\") "
Sep 14 03:17:05.892384 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:05.892292    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wq9k5\" (UniqueName: \"kubernetes.io/projected/5437a89f-6ac4-46f0-909d-808ecf23e86f-kube-api-access-wq9k5\") pod \"5437a89f-6ac4-46f0-909d-808ecf23e86f\" (UID: \"5437a89f-6ac4-46f0-909d-808ecf23e86f\") "
Sep 14 03:17:05.892384 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:05.892334    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5437a89f-6ac4-46f0-909d-808ecf23e86f-maas-api-tls\") pod \"5437a89f-6ac4-46f0-909d-808ecf23e86f\" (UID: \"5437a89f-6ac4-46f0-909d-808ecf23e86f\") "
Sep 14 03:17:05.892547 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:05.892383    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5437a89f-6ac4-46f0-909d-808ecf23e86f-openshift-service-ca\") pod \"5437a89f-6ac4-46f0-909d-808ecf23e86f\" (UID: \"5437a89f-6ac4-46f0-909d-808ecf23e86f\") "
Sep 14 03:17:05.892977 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:05.892946    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5437a89f-6ac4-46f0-909d-808ecf23e86f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5437a89f-6ac4-46f0-909d-808ecf23e86f" (UID: "5437a89f-6ac4-46f0-909d-808ecf23e86f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 03:17:05.895033 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:05.895004    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5437a89f-6ac4-46f0-909d-808ecf23e86f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5437a89f-6ac4-46f0-909d-808ecf23e86f" (UID: "5437a89f-6ac4-46f0-909d-808ecf23e86f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 03:17:05.895119 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:05.895034    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5437a89f-6ac4-46f0-909d-808ecf23e86f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5437a89f-6ac4-46f0-909d-808ecf23e86f" (UID: "5437a89f-6ac4-46f0-909d-808ecf23e86f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 03:17:05.895119 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:05.895043    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5437a89f-6ac4-46f0-909d-808ecf23e86f-kube-api-access-wq9k5" (OuterVolumeSpecName: "kube-api-access-wq9k5") pod "5437a89f-6ac4-46f0-909d-808ecf23e86f" (UID: "5437a89f-6ac4-46f0-909d-808ecf23e86f"). InnerVolumeSpecName "kube-api-access-wq9k5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:17:05.898713 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:05.898692    2564 scope.go:117] "RemoveContainer" containerID="fa2a0cf3f9fd0b381cf9d6e0aac8521cc273d01512e928464f3e75e59deb2348"
Sep 14 03:17:05.899065 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:17:05.899047    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa2a0cf3f9fd0b381cf9d6e0aac8521cc273d01512e928464f3e75e59deb2348\": container with ID starting with fa2a0cf3f9fd0b381cf9d6e0aac8521cc273d01512e928464f3e75e59deb2348 not found: ID does not exist" containerID="fa2a0cf3f9fd0b381cf9d6e0aac8521cc273d01512e928464f3e75e59deb2348"
Sep 14 03:17:05.899108 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:05.899074    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa2a0cf3f9fd0b381cf9d6e0aac8521cc273d01512e928464f3e75e59deb2348"} err="failed to get container status \"fa2a0cf3f9fd0b381cf9d6e0aac8521cc273d01512e928464f3e75e59deb2348\": rpc error: code = NotFound desc = could not find container \"fa2a0cf3f9fd0b381cf9d6e0aac8521cc273d01512e928464f3e75e59deb2348\": container with ID starting with fa2a0cf3f9fd0b381cf9d6e0aac8521cc273d01512e928464f3e75e59deb2348 not found: ID does not exist"
Sep 14 03:17:05.899108 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:05.899091    2564 scope.go:117] "RemoveContainer" containerID="ced79a87a92353f3d89d8b28d34a1d1fdd9ff14d15222ef355c518d122416851"
Sep 14 03:17:05.910487 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:05.910465    2564 scope.go:117] "RemoveContainer" containerID="ced79a87a92353f3d89d8b28d34a1d1fdd9ff14d15222ef355c518d122416851"
Sep 14 03:17:05.910847 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:17:05.910823    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ced79a87a92353f3d89d8b28d34a1d1fdd9ff14d15222ef355c518d122416851\": container with ID starting with ced79a87a92353f3d89d8b28d34a1d1fdd9ff14d15222ef355c518d122416851 not found: ID does not exist" containerID="ced79a87a92353f3d89d8b28d34a1d1fdd9ff14d15222ef355c518d122416851"
Sep 14 03:17:05.910928 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:05.910851    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ced79a87a92353f3d89d8b28d34a1d1fdd9ff14d15222ef355c518d122416851"} err="failed to get container status \"ced79a87a92353f3d89d8b28d34a1d1fdd9ff14d15222ef355c518d122416851\": rpc error: code = NotFound desc = could not find container \"ced79a87a92353f3d89d8b28d34a1d1fdd9ff14d15222ef355c518d122416851\": container with ID starting with ced79a87a92353f3d89d8b28d34a1d1fdd9ff14d15222ef355c518d122416851 not found: ID does not exist"
Sep 14 03:17:05.993638 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:05.993546    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qrb7r\" (UniqueName: \"kubernetes.io/projected/d13e2896-2b10-4f6c-8583-1d60c1390838-kube-api-access-qrb7r\") pod \"d13e2896-2b10-4f6c-8583-1d60c1390838\" (UID: \"d13e2896-2b10-4f6c-8583-1d60c1390838\") "
Sep 14 03:17:05.993790 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:05.993708    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d13e2896-2b10-4f6c-8583-1d60c1390838-plugins-config-volume\") pod \"d13e2896-2b10-4f6c-8583-1d60c1390838\" (UID: \"d13e2896-2b10-4f6c-8583-1d60c1390838\") "
Sep 14 03:17:05.994063 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:05.994042    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5437a89f-6ac4-46f0-909d-808ecf23e86f-maas-api-tls\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:17:05.994063 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:05.994063    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5437a89f-6ac4-46f0-909d-808ecf23e86f-openshift-service-ca\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:17:05.994204 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:05.994073    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5437a89f-6ac4-46f0-909d-808ecf23e86f-metrics-certs\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:17:05.994204 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:05.994082    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wq9k5\" (UniqueName: \"kubernetes.io/projected/5437a89f-6ac4-46f0-909d-808ecf23e86f-kube-api-access-wq9k5\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:17:05.995728 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:05.994065    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d13e2896-2b10-4f6c-8583-1d60c1390838-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d13e2896-2b10-4f6c-8583-1d60c1390838" (UID: "d13e2896-2b10-4f6c-8583-1d60c1390838"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 03:17:06.002307 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:06.002194    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d13e2896-2b10-4f6c-8583-1d60c1390838-kube-api-access-qrb7r" (OuterVolumeSpecName: "kube-api-access-qrb7r") pod "d13e2896-2b10-4f6c-8583-1d60c1390838" (UID: "d13e2896-2b10-4f6c-8583-1d60c1390838"). InnerVolumeSpecName "kube-api-access-qrb7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:17:06.095319 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:06.095204    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d13e2896-2b10-4f6c-8583-1d60c1390838-plugins-config-volume\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:17:06.095319 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:06.095245    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrb7r\" (UniqueName: \"kubernetes.io/projected/d13e2896-2b10-4f6c-8583-1d60c1390838-kube-api-access-qrb7r\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:17:06.214815 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:06.214779    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3a33c1-55db8cd5c4-h87l7"]
Sep 14 03:17:06.218761 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:06.218728    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3a33c1-55db8cd5c4-h87l7"]
Sep 14 03:17:06.229595 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:06.229563    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3a33c1-58cdf544cd-v2xw4"]
Sep 14 03:17:06.234139 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:06.234110    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3a33c1-58cdf544cd-v2xw4"]
Sep 14 03:17:06.432497 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:06.432463    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:17:06.532153 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:06.532092    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:17:06.845441 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:06.845385    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:17:07.099229 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:07.099154    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5437a89f-6ac4-46f0-909d-808ecf23e86f" path="/var/lib/kubelet/pods/5437a89f-6ac4-46f0-909d-808ecf23e86f/volumes"
Sep 14 03:17:07.099535 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:07.099522    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d13e2896-2b10-4f6c-8583-1d60c1390838" path="/var/lib/kubelet/pods/d13e2896-2b10-4f6c-8583-1d60c1390838/volumes"
Sep 14 03:17:07.672082 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:07.672048    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:17:07.988309 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:07.988228    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:17:08.081319 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:08.081279    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:17:08.554920 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:08.554887    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:17:09.263799 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:09.263748    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:17:09.578081 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:09.578052    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:17:10.141044 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:10.141008    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:17:10.308495 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:10.308451    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:17:10.944333 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:10.944293    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:17:13.040813 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:13.040779    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:17:13.164007 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:13.163967    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:17:13.360721 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:13.360613    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:17:13.961442 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:13.961382    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:17:14.864244 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:14.864200    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:17:15.654035 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:15.653999    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:17:15.763058 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:15.763023    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:17:16.440743 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:16.440704    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:17:17.333090 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:17.333049    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:17:18.041740 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:18.041695    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:17:19.001645 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:19.001602    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:17:19.666793 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:19.666759    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:17:29.280589 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:29.280558    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-97da78-7b967d5785-ksvwm"]
Sep 14 03:17:29.281034 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:29.280762    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-97da78-7b967d5785-ksvwm" podUID="49ef063e-3050-42ba-aa15-4fc3504d52ef" containerName="maas-api" containerID="cri-o://d93f37c3ec35bcc355a970a069c44a6b6fd33ff9e6c6465870dba7fc699347ce" gracePeriod=30
Sep 14 03:17:29.312344 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:29.312311    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:17:29.539736 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:29.539677    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-97da78-7b967d5785-ksvwm"
Sep 14 03:17:29.626482 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:29.626251    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49ef063e-3050-42ba-aa15-4fc3504d52ef-metrics-certs\") pod \"49ef063e-3050-42ba-aa15-4fc3504d52ef\" (UID: \"49ef063e-3050-42ba-aa15-4fc3504d52ef\") "
Sep 14 03:17:29.626482 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:29.626337    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49ef063e-3050-42ba-aa15-4fc3504d52ef-maas-api-tls\") pod \"49ef063e-3050-42ba-aa15-4fc3504d52ef\" (UID: \"49ef063e-3050-42ba-aa15-4fc3504d52ef\") "
Sep 14 03:17:29.626482 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:29.626373    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49ef063e-3050-42ba-aa15-4fc3504d52ef-openshift-service-ca\") pod \"49ef063e-3050-42ba-aa15-4fc3504d52ef\" (UID: \"49ef063e-3050-42ba-aa15-4fc3504d52ef\") "
Sep 14 03:17:29.626482 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:29.626479    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bwpk2\" (UniqueName: \"kubernetes.io/projected/49ef063e-3050-42ba-aa15-4fc3504d52ef-kube-api-access-bwpk2\") pod \"49ef063e-3050-42ba-aa15-4fc3504d52ef\" (UID: \"49ef063e-3050-42ba-aa15-4fc3504d52ef\") "
Sep 14 03:17:29.629223 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:29.629183    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49ef063e-3050-42ba-aa15-4fc3504d52ef-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "49ef063e-3050-42ba-aa15-4fc3504d52ef" (UID: "49ef063e-3050-42ba-aa15-4fc3504d52ef"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 03:17:29.629472 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:29.629443    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49ef063e-3050-42ba-aa15-4fc3504d52ef-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "49ef063e-3050-42ba-aa15-4fc3504d52ef" (UID: "49ef063e-3050-42ba-aa15-4fc3504d52ef"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 03:17:29.629740 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:29.629709    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49ef063e-3050-42ba-aa15-4fc3504d52ef-kube-api-access-bwpk2" (OuterVolumeSpecName: "kube-api-access-bwpk2") pod "49ef063e-3050-42ba-aa15-4fc3504d52ef" (UID: "49ef063e-3050-42ba-aa15-4fc3504d52ef"). InnerVolumeSpecName "kube-api-access-bwpk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:17:29.630936 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:29.630912    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49ef063e-3050-42ba-aa15-4fc3504d52ef-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "49ef063e-3050-42ba-aa15-4fc3504d52ef" (UID: "49ef063e-3050-42ba-aa15-4fc3504d52ef"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 03:17:29.727685 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:29.727651    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49ef063e-3050-42ba-aa15-4fc3504d52ef-maas-api-tls\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:17:29.727685 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:29.727683    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49ef063e-3050-42ba-aa15-4fc3504d52ef-openshift-service-ca\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:17:29.727880 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:29.727696    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwpk2\" (UniqueName: \"kubernetes.io/projected/49ef063e-3050-42ba-aa15-4fc3504d52ef-kube-api-access-bwpk2\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:17:29.727880 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:29.727708    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49ef063e-3050-42ba-aa15-4fc3504d52ef-metrics-certs\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:17:30.032815 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:30.032781    2564 generic.go:356] "Generic (PLEG): container finished" podID="49ef063e-3050-42ba-aa15-4fc3504d52ef" containerID="d93f37c3ec35bcc355a970a069c44a6b6fd33ff9e6c6465870dba7fc699347ce" exitCode=0
Sep 14 03:17:30.033010 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:30.032834    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-97da78-7b967d5785-ksvwm"
Sep 14 03:17:30.033010 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:30.032868    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-97da78-7b967d5785-ksvwm" event={"ID":"49ef063e-3050-42ba-aa15-4fc3504d52ef","Type":"ContainerDied","Data":"d93f37c3ec35bcc355a970a069c44a6b6fd33ff9e6c6465870dba7fc699347ce"}
Sep 14 03:17:30.033010 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:30.032958    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-97da78-7b967d5785-ksvwm" event={"ID":"49ef063e-3050-42ba-aa15-4fc3504d52ef","Type":"ContainerDied","Data":"d27dffadbefb081e14edc19fa51db43d139d610bd157f7c7dc9db730de3f9ea2"}
Sep 14 03:17:30.033010 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:30.032987    2564 scope.go:117] "RemoveContainer" containerID="d93f37c3ec35bcc355a970a069c44a6b6fd33ff9e6c6465870dba7fc699347ce"
Sep 14 03:17:30.042574 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:30.042556    2564 scope.go:117] "RemoveContainer" containerID="d93f37c3ec35bcc355a970a069c44a6b6fd33ff9e6c6465870dba7fc699347ce"
Sep 14 03:17:30.042808 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:17:30.042790    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d93f37c3ec35bcc355a970a069c44a6b6fd33ff9e6c6465870dba7fc699347ce\": container with ID starting with d93f37c3ec35bcc355a970a069c44a6b6fd33ff9e6c6465870dba7fc699347ce not found: ID does not exist" containerID="d93f37c3ec35bcc355a970a069c44a6b6fd33ff9e6c6465870dba7fc699347ce"
Sep 14 03:17:30.042852 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:30.042814    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d93f37c3ec35bcc355a970a069c44a6b6fd33ff9e6c6465870dba7fc699347ce"} err="failed to get container status \"d93f37c3ec35bcc355a970a069c44a6b6fd33ff9e6c6465870dba7fc699347ce\": rpc error: code = NotFound desc = could not find container \"d93f37c3ec35bcc355a970a069c44a6b6fd33ff9e6c6465870dba7fc699347ce\": container with ID starting with d93f37c3ec35bcc355a970a069c44a6b6fd33ff9e6c6465870dba7fc699347ce not found: ID does not exist"
Sep 14 03:17:30.055050 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:30.055022    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-97da78-7b967d5785-ksvwm"]
Sep 14 03:17:30.057632 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:30.057613    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-97da78-7b967d5785-ksvwm"]
Sep 14 03:17:30.514882 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:30.514775    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:17:30.612953 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:30.612923    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:17:31.099075 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:31.099035    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49ef063e-3050-42ba-aa15-4fc3504d52ef" path="/var/lib/kubelet/pods/49ef063e-3050-42ba-aa15-4fc3504d52ef/volumes"
Sep 14 03:17:31.336175 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:31.336137    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:17:32.156016 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:32.155981    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:17:32.942686 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:32.942638    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:17:33.670398 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:33.670366    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:17:35.281979 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:35.281942    2564 scope.go:117] "RemoveContainer" containerID="78f2cf32c8d1bf3d3b3d3535cd52fd8781ee19bae589c1760772a65e0295df71"
Sep 14 03:17:35.908685 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:35.908649    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:17:35.986665 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:35.986596    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:17:36.200749 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:36.200648    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:17:36.653386 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:36.653355    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:17:37.684307 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:37.684268    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:17:38.576597 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:38.576563    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:17:39.465422 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:39.465285    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:17:40.239477 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:40.239441    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:17:41.100421 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:41.100367    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:17:41.876814 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:41.876774    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:17:42.748564 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:42.748528    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:17:44.342190 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:44.342158    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:17:44.745419 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:44.745311    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:17:45.720827 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:45.720780    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:17:46.433486 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:46.433437    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:17:47.038330 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:47.038296    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:17:47.158586 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:47.158553    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ce5b9b-8f77c865d-d2qkn"]
Sep 14 03:17:47.158820 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:47.158794    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ce5b9b-8f77c865d-d2qkn" podUID="3eed608f-4d31-4708-ac7c-9a29fe254a1b" containerName="payload-pre-processing" containerID="cri-o://95b4f0a0090ddf7f23472f5e6c8139c329d246acaba401dd67d59f66ac91a806" gracePeriod=30
Sep 14 03:17:47.238137 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:47.238101    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:17:47.338570 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:47.338541    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:17:47.427634 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:47.427602    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:17:47.432541 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:47.432510    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ce5b9b-8f77c865d-d2qkn"
Sep 14 03:17:47.609386 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:47.609282    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wghk2\" (UniqueName: \"kubernetes.io/projected/3eed608f-4d31-4708-ac7c-9a29fe254a1b-kube-api-access-wghk2\") pod \"3eed608f-4d31-4708-ac7c-9a29fe254a1b\" (UID: \"3eed608f-4d31-4708-ac7c-9a29fe254a1b\") "
Sep 14 03:17:47.609592 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:47.609449    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3eed608f-4d31-4708-ac7c-9a29fe254a1b-plugins-config-volume\") pod \"3eed608f-4d31-4708-ac7c-9a29fe254a1b\" (UID: \"3eed608f-4d31-4708-ac7c-9a29fe254a1b\") "
Sep 14 03:17:47.609872 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:47.609844    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3eed608f-4d31-4708-ac7c-9a29fe254a1b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3eed608f-4d31-4708-ac7c-9a29fe254a1b" (UID: "3eed608f-4d31-4708-ac7c-9a29fe254a1b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 03:17:47.611238 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:47.611211    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:17:47.611749 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:47.611727    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eed608f-4d31-4708-ac7c-9a29fe254a1b-kube-api-access-wghk2" (OuterVolumeSpecName: "kube-api-access-wghk2") pod "3eed608f-4d31-4708-ac7c-9a29fe254a1b" (UID: "3eed608f-4d31-4708-ac7c-9a29fe254a1b"). InnerVolumeSpecName "kube-api-access-wghk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:17:47.710914 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:47.710876    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wghk2\" (UniqueName: \"kubernetes.io/projected/3eed608f-4d31-4708-ac7c-9a29fe254a1b-kube-api-access-wghk2\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:17:47.710914 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:47.710914    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3eed608f-4d31-4708-ac7c-9a29fe254a1b-plugins-config-volume\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:17:47.746873 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:47.746836    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:17:48.069936 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:48.069892    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:17:48.134592 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:48.134555    2564 generic.go:356] "Generic (PLEG): container finished" podID="3eed608f-4d31-4708-ac7c-9a29fe254a1b" containerID="95b4f0a0090ddf7f23472f5e6c8139c329d246acaba401dd67d59f66ac91a806" exitCode=0
Sep 14 03:17:48.134775 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:48.134626    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ce5b9b-8f77c865d-d2qkn"
Sep 14 03:17:48.134775 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:48.134631    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ce5b9b-8f77c865d-d2qkn" event={"ID":"3eed608f-4d31-4708-ac7c-9a29fe254a1b","Type":"ContainerDied","Data":"95b4f0a0090ddf7f23472f5e6c8139c329d246acaba401dd67d59f66ac91a806"}
Sep 14 03:17:48.134775 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:48.134690    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ce5b9b-8f77c865d-d2qkn" event={"ID":"3eed608f-4d31-4708-ac7c-9a29fe254a1b","Type":"ContainerDied","Data":"49eb5766c4e12b0dcd884a83f6f7261f1a9c6516765aa42fb450c741264edbcf"}
Sep 14 03:17:48.134775 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:48.134713    2564 scope.go:117] "RemoveContainer" containerID="95b4f0a0090ddf7f23472f5e6c8139c329d246acaba401dd67d59f66ac91a806"
Sep 14 03:17:48.144348 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:48.144331    2564 scope.go:117] "RemoveContainer" containerID="95b4f0a0090ddf7f23472f5e6c8139c329d246acaba401dd67d59f66ac91a806"
Sep 14 03:17:48.144718 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:17:48.144694    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95b4f0a0090ddf7f23472f5e6c8139c329d246acaba401dd67d59f66ac91a806\": container with ID starting with 95b4f0a0090ddf7f23472f5e6c8139c329d246acaba401dd67d59f66ac91a806 not found: ID does not exist" containerID="95b4f0a0090ddf7f23472f5e6c8139c329d246acaba401dd67d59f66ac91a806"
Sep 14 03:17:48.144781 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:48.144727    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95b4f0a0090ddf7f23472f5e6c8139c329d246acaba401dd67d59f66ac91a806"} err="failed to get container status \"95b4f0a0090ddf7f23472f5e6c8139c329d246acaba401dd67d59f66ac91a806\": rpc error: code = NotFound desc = could not find container \"95b4f0a0090ddf7f23472f5e6c8139c329d246acaba401dd67d59f66ac91a806\": container with ID starting with 95b4f0a0090ddf7f23472f5e6c8139c329d246acaba401dd67d59f66ac91a806 not found: ID does not exist"
Sep 14 03:17:48.172281 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:48.172248    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ce5b9b-8f77c865d-d2qkn"]
Sep 14 03:17:48.179329 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:48.179300    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ce5b9b-8f77c865d-d2qkn"]
Sep 14 03:17:48.741132 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:48.741098    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:17:49.099564 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:49.099535    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3eed608f-4d31-4708-ac7c-9a29fe254a1b" path="/var/lib/kubelet/pods/3eed608f-4d31-4708-ac7c-9a29fe254a1b/volumes"
Sep 14 03:17:49.318466 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:49.318416    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-9c9f1e/distinct-2-w4-9c9f1e-kserve-58779c476f-qmt8b"]
Sep 14 03:17:49.329105 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:49.329080    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-9c9f1e/distinct-w4-9c9f1e-kserve-579d94566b-tgjjk"]
Sep 14 03:17:49.436760 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:49.436670    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:17:49.551281 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:49.551251    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:17:50.441695 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:50.441651    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:17:51.224531 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:51.224492    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:17:52.026056 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:52.026022    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:17:53.417117 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:53.417084    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:17:54.504861 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:54.504825    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:17:54.601929 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:54.601889    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:17:54.697136 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:54.697097    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:17:54.767949 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:54.767856    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:17:55.658910 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:55.658873    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:17:56.416097 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:56.416062    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:17:57.429583 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:57.429493    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:17:58.123639 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:17:58.123589    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:18:03.777172 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:03.777130    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-9c9f1e-8497955c59-k5c6b"]
Sep 14 03:18:03.777779 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:03.777389    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9c9f1e-8497955c59-k5c6b" podUID="0940ece5-1e43-46ce-86d4-3bc31c3777a2" containerName="maas-api" containerID="cri-o://a04e8f1e6fa7b6e849244550691537e271e91d8259d260dd7e0f39f7e42ec75f" gracePeriod=30
Sep 14 03:18:03.818448 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:03.814998    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:18:03.821735 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:03.821701    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9c9f1e-29822595-z4rrc"]
Sep 14 03:18:03.824019 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:03.823993    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9c9f1e-29822595-z4rrc"]
Sep 14 03:18:03.864078 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:03.864043    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-9c9f1e-785c744797-7hjth"]
Sep 14 03:18:03.864335 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:03.864300    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-9c9f1e-785c744797-7hjth" podUID="cd75830d-dc18-47e2-a0fc-7ea2e0cd21b4" containerName="payload-processing" containerID="cri-o://d524a37470cb1eedfdefd675865381f05f4565893b661c8a7c532237db1d57d7" gracePeriod=30
Sep 14 03:18:03.917590 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:03.917559    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:18:04.018288 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.018187    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:18:04.110351 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.110303    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:18:04.120254 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.120223    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9c9f1e-8497955c59-k5c6b"
Sep 14 03:18:04.152926 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.152897    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-9c9f1e-785c744797-7hjth"
Sep 14 03:18:04.177858 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.177815    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0940ece5-1e43-46ce-86d4-3bc31c3777a2-maas-api-tls\") pod \"0940ece5-1e43-46ce-86d4-3bc31c3777a2\" (UID: \"0940ece5-1e43-46ce-86d4-3bc31c3777a2\") "
Sep 14 03:18:04.178448 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.177933    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-szqxx\" (UniqueName: \"kubernetes.io/projected/0940ece5-1e43-46ce-86d4-3bc31c3777a2-kube-api-access-szqxx\") pod \"0940ece5-1e43-46ce-86d4-3bc31c3777a2\" (UID: \"0940ece5-1e43-46ce-86d4-3bc31c3777a2\") "
Sep 14 03:18:04.178448 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.178060    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0940ece5-1e43-46ce-86d4-3bc31c3777a2-metrics-certs\") pod \"0940ece5-1e43-46ce-86d4-3bc31c3777a2\" (UID: \"0940ece5-1e43-46ce-86d4-3bc31c3777a2\") "
Sep 14 03:18:04.178448 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.178109    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0940ece5-1e43-46ce-86d4-3bc31c3777a2-openshift-service-ca\") pod \"0940ece5-1e43-46ce-86d4-3bc31c3777a2\" (UID: \"0940ece5-1e43-46ce-86d4-3bc31c3777a2\") "
Sep 14 03:18:04.179369 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.179315    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0940ece5-1e43-46ce-86d4-3bc31c3777a2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0940ece5-1e43-46ce-86d4-3bc31c3777a2" (UID: "0940ece5-1e43-46ce-86d4-3bc31c3777a2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 03:18:04.181527 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.181496    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0940ece5-1e43-46ce-86d4-3bc31c3777a2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0940ece5-1e43-46ce-86d4-3bc31c3777a2" (UID: "0940ece5-1e43-46ce-86d4-3bc31c3777a2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 03:18:04.181639 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.181597    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0940ece5-1e43-46ce-86d4-3bc31c3777a2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0940ece5-1e43-46ce-86d4-3bc31c3777a2" (UID: "0940ece5-1e43-46ce-86d4-3bc31c3777a2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 03:18:04.182024 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.181997    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0940ece5-1e43-46ce-86d4-3bc31c3777a2-kube-api-access-szqxx" (OuterVolumeSpecName: "kube-api-access-szqxx") pod "0940ece5-1e43-46ce-86d4-3bc31c3777a2" (UID: "0940ece5-1e43-46ce-86d4-3bc31c3777a2"). InnerVolumeSpecName "kube-api-access-szqxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:18:04.199637 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.199599    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:18:04.223184 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.223140    2564 generic.go:356] "Generic (PLEG): container finished" podID="0940ece5-1e43-46ce-86d4-3bc31c3777a2" containerID="a04e8f1e6fa7b6e849244550691537e271e91d8259d260dd7e0f39f7e42ec75f" exitCode=0
Sep 14 03:18:04.223373 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.223203    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9c9f1e-8497955c59-k5c6b"
Sep 14 03:18:04.223373 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.223237    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9c9f1e-8497955c59-k5c6b" event={"ID":"0940ece5-1e43-46ce-86d4-3bc31c3777a2","Type":"ContainerDied","Data":"a04e8f1e6fa7b6e849244550691537e271e91d8259d260dd7e0f39f7e42ec75f"}
Sep 14 03:18:04.223373 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.223293    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9c9f1e-8497955c59-k5c6b" event={"ID":"0940ece5-1e43-46ce-86d4-3bc31c3777a2","Type":"ContainerDied","Data":"1b12a7ab057325ed0eaed544ae86908d2cf9900dc5b42c6535d58b79432bc9fe"}
Sep 14 03:18:04.223373 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.223314    2564 scope.go:117] "RemoveContainer" containerID="a04e8f1e6fa7b6e849244550691537e271e91d8259d260dd7e0f39f7e42ec75f"
Sep 14 03:18:04.224757 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.224730    2564 generic.go:356] "Generic (PLEG): container finished" podID="cd75830d-dc18-47e2-a0fc-7ea2e0cd21b4" containerID="d524a37470cb1eedfdefd675865381f05f4565893b661c8a7c532237db1d57d7" exitCode=0
Sep 14 03:18:04.224875 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.224780    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-9c9f1e-785c744797-7hjth" event={"ID":"cd75830d-dc18-47e2-a0fc-7ea2e0cd21b4","Type":"ContainerDied","Data":"d524a37470cb1eedfdefd675865381f05f4565893b661c8a7c532237db1d57d7"}
Sep 14 03:18:04.224875 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.224829    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-9c9f1e-785c744797-7hjth" event={"ID":"cd75830d-dc18-47e2-a0fc-7ea2e0cd21b4","Type":"ContainerDied","Data":"6d8a158647c871e6ba0c12838ef944c1aeb84379515bd3216ccaf30d61e82409"}
Sep 14 03:18:04.224875 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.224762    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-9c9f1e-785c744797-7hjth"
Sep 14 03:18:04.233282 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.233262    2564 scope.go:117] "RemoveContainer" containerID="a04e8f1e6fa7b6e849244550691537e271e91d8259d260dd7e0f39f7e42ec75f"
Sep 14 03:18:04.233682 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:18:04.233662    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a04e8f1e6fa7b6e849244550691537e271e91d8259d260dd7e0f39f7e42ec75f\": container with ID starting with a04e8f1e6fa7b6e849244550691537e271e91d8259d260dd7e0f39f7e42ec75f not found: ID does not exist" containerID="a04e8f1e6fa7b6e849244550691537e271e91d8259d260dd7e0f39f7e42ec75f"
Sep 14 03:18:04.233760 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.233691    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a04e8f1e6fa7b6e849244550691537e271e91d8259d260dd7e0f39f7e42ec75f"} err="failed to get container status \"a04e8f1e6fa7b6e849244550691537e271e91d8259d260dd7e0f39f7e42ec75f\": rpc error: code = NotFound desc = could not find container \"a04e8f1e6fa7b6e849244550691537e271e91d8259d260dd7e0f39f7e42ec75f\": container with ID starting with a04e8f1e6fa7b6e849244550691537e271e91d8259d260dd7e0f39f7e42ec75f not found: ID does not exist"
Sep 14 03:18:04.233760 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.233708    2564 scope.go:117] "RemoveContainer" containerID="d524a37470cb1eedfdefd675865381f05f4565893b661c8a7c532237db1d57d7"
Sep 14 03:18:04.243684 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.243643    2564 scope.go:117] "RemoveContainer" containerID="d524a37470cb1eedfdefd675865381f05f4565893b661c8a7c532237db1d57d7"
Sep 14 03:18:04.244304 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:18:04.244240    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d524a37470cb1eedfdefd675865381f05f4565893b661c8a7c532237db1d57d7\": container with ID starting with d524a37470cb1eedfdefd675865381f05f4565893b661c8a7c532237db1d57d7 not found: ID does not exist" containerID="d524a37470cb1eedfdefd675865381f05f4565893b661c8a7c532237db1d57d7"
Sep 14 03:18:04.244304 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.244276    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d524a37470cb1eedfdefd675865381f05f4565893b661c8a7c532237db1d57d7"} err="failed to get container status \"d524a37470cb1eedfdefd675865381f05f4565893b661c8a7c532237db1d57d7\": rpc error: code = NotFound desc = could not find container \"d524a37470cb1eedfdefd675865381f05f4565893b661c8a7c532237db1d57d7\": container with ID starting with d524a37470cb1eedfdefd675865381f05f4565893b661c8a7c532237db1d57d7 not found: ID does not exist"
Sep 14 03:18:04.246480 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.246458    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-9c9f1e-8497955c59-k5c6b"]
Sep 14 03:18:04.250802 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.250773    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-9c9f1e-8497955c59-k5c6b"]
Sep 14 03:18:04.279747 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.279711    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x964n\" (UniqueName: \"kubernetes.io/projected/cd75830d-dc18-47e2-a0fc-7ea2e0cd21b4-kube-api-access-x964n\") pod \"cd75830d-dc18-47e2-a0fc-7ea2e0cd21b4\" (UID: \"cd75830d-dc18-47e2-a0fc-7ea2e0cd21b4\") "
Sep 14 03:18:04.279945 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.279760    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd75830d-dc18-47e2-a0fc-7ea2e0cd21b4-plugins-config-volume\") pod \"cd75830d-dc18-47e2-a0fc-7ea2e0cd21b4\" (UID: \"cd75830d-dc18-47e2-a0fc-7ea2e0cd21b4\") "
Sep 14 03:18:04.280013 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.279999    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szqxx\" (UniqueName: \"kubernetes.io/projected/0940ece5-1e43-46ce-86d4-3bc31c3777a2-kube-api-access-szqxx\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:18:04.280013 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.280011    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0940ece5-1e43-46ce-86d4-3bc31c3777a2-metrics-certs\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:18:04.280123 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.280020    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0940ece5-1e43-46ce-86d4-3bc31c3777a2-openshift-service-ca\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:18:04.280123 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.280029    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0940ece5-1e43-46ce-86d4-3bc31c3777a2-maas-api-tls\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:18:04.280211 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.280151    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd75830d-dc18-47e2-a0fc-7ea2e0cd21b4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cd75830d-dc18-47e2-a0fc-7ea2e0cd21b4" (UID: "cd75830d-dc18-47e2-a0fc-7ea2e0cd21b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 03:18:04.281988 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.281956    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd75830d-dc18-47e2-a0fc-7ea2e0cd21b4-kube-api-access-x964n" (OuterVolumeSpecName: "kube-api-access-x964n") pod "cd75830d-dc18-47e2-a0fc-7ea2e0cd21b4" (UID: "cd75830d-dc18-47e2-a0fc-7ea2e0cd21b4"). InnerVolumeSpecName "kube-api-access-x964n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:18:04.338703 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.338618    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-9c9f1e/distinct-2-w4-9c9f1e-kserve-58779c476f-qmt8b" podUID="4ced0495-da0f-47b8-ab59-d369f274eb29" containerName="main" containerID="cri-o://e006c7368c1c2260aa2ae0c60c877847dc3920aa684e3a6fd97df6569a6a6709" gracePeriod=45
Sep 14 03:18:04.353000 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.352884    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-9c9f1e/distinct-w4-9c9f1e-kserve-579d94566b-tgjjk" podUID="09fd993a-f4ed-44c2-983d-73f7501e6b9b" containerName="main" containerID="cri-o://3bafbacfe580478b16a366fa151010c9508709cd6d16a8611ba4b826e7e89a5b" gracePeriod=45
Sep 14 03:18:04.380702 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.380669    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x964n\" (UniqueName: \"kubernetes.io/projected/cd75830d-dc18-47e2-a0fc-7ea2e0cd21b4-kube-api-access-x964n\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:18:04.380702 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.380698    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd75830d-dc18-47e2-a0fc-7ea2e0cd21b4-plugins-config-volume\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:18:04.517001 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.516958    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:18:04.551823 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.551776    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-9c9f1e-785c744797-7hjth"]
Sep 14 03:18:04.553854 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.553830    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-9c9f1e-785c744797-7hjth"]
Sep 14 03:18:04.591878 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.591792    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9c9f1e/distinct-2-w4-9c9f1e-kserve-58779c476f-qmt8b"
Sep 14 03:18:04.623593 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.622856    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:18:04.637083 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.637055    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9c9f1e/distinct-w4-9c9f1e-kserve-579d94566b-tgjjk"
Sep 14 03:18:04.691020 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.690991    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ced0495-da0f-47b8-ab59-d369f274eb29-dshm\") pod \"4ced0495-da0f-47b8-ab59-d369f274eb29\" (UID: \"4ced0495-da0f-47b8-ab59-d369f274eb29\") "
Sep 14 03:18:04.691195 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.691047    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pbxvp\" (UniqueName: \"kubernetes.io/projected/09fd993a-f4ed-44c2-983d-73f7501e6b9b-kube-api-access-pbxvp\") pod \"09fd993a-f4ed-44c2-983d-73f7501e6b9b\" (UID: \"09fd993a-f4ed-44c2-983d-73f7501e6b9b\") "
Sep 14 03:18:04.691195 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.691066    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ced0495-da0f-47b8-ab59-d369f274eb29-home\") pod \"4ced0495-da0f-47b8-ab59-d369f274eb29\" (UID: \"4ced0495-da0f-47b8-ab59-d369f274eb29\") "
Sep 14 03:18:04.691195 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.691111    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ced0495-da0f-47b8-ab59-d369f274eb29-tmp-dir\") pod \"4ced0495-da0f-47b8-ab59-d369f274eb29\" (UID: \"4ced0495-da0f-47b8-ab59-d369f274eb29\") "
Sep 14 03:18:04.691195 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.691160    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09fd993a-f4ed-44c2-983d-73f7501e6b9b-model-cache\") pod \"09fd993a-f4ed-44c2-983d-73f7501e6b9b\" (UID: \"09fd993a-f4ed-44c2-983d-73f7501e6b9b\") "
Sep 14 03:18:04.691195 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.691191    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09fd993a-f4ed-44c2-983d-73f7501e6b9b-tmp-dir\") pod \"09fd993a-f4ed-44c2-983d-73f7501e6b9b\" (UID: \"09fd993a-f4ed-44c2-983d-73f7501e6b9b\") "
Sep 14 03:18:04.691505 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.691238    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09fd993a-f4ed-44c2-983d-73f7501e6b9b-tls-certs\") pod \"09fd993a-f4ed-44c2-983d-73f7501e6b9b\" (UID: \"09fd993a-f4ed-44c2-983d-73f7501e6b9b\") "
Sep 14 03:18:04.691505 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.691264    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09fd993a-f4ed-44c2-983d-73f7501e6b9b-home\") pod \"09fd993a-f4ed-44c2-983d-73f7501e6b9b\" (UID: \"09fd993a-f4ed-44c2-983d-73f7501e6b9b\") "
Sep 14 03:18:04.691505 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.691311    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ced0495-da0f-47b8-ab59-d369f274eb29-model-cache\") pod \"4ced0495-da0f-47b8-ab59-d369f274eb29\" (UID: \"4ced0495-da0f-47b8-ab59-d369f274eb29\") "
Sep 14 03:18:04.691505 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.691349    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09fd993a-f4ed-44c2-983d-73f7501e6b9b-dshm\") pod \"09fd993a-f4ed-44c2-983d-73f7501e6b9b\" (UID: \"09fd993a-f4ed-44c2-983d-73f7501e6b9b\") "
Sep 14 03:18:04.691505 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.691381    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ced0495-da0f-47b8-ab59-d369f274eb29-tls-certs\") pod \"4ced0495-da0f-47b8-ab59-d369f274eb29\" (UID: \"4ced0495-da0f-47b8-ab59-d369f274eb29\") "
Sep 14 03:18:04.691505 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.691390    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ced0495-da0f-47b8-ab59-d369f274eb29-home" (OuterVolumeSpecName: "home") pod "4ced0495-da0f-47b8-ab59-d369f274eb29" (UID: "4ced0495-da0f-47b8-ab59-d369f274eb29"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:18:04.691505 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.691424    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ced0495-da0f-47b8-ab59-d369f274eb29-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4ced0495-da0f-47b8-ab59-d369f274eb29" (UID: "4ced0495-da0f-47b8-ab59-d369f274eb29"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:18:04.691505 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.691436    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09fd993a-f4ed-44c2-983d-73f7501e6b9b-model-cache" (OuterVolumeSpecName: "model-cache") pod "09fd993a-f4ed-44c2-983d-73f7501e6b9b" (UID: "09fd993a-f4ed-44c2-983d-73f7501e6b9b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:18:04.691505 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.691473    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vsvsz\" (UniqueName: \"kubernetes.io/projected/4ced0495-da0f-47b8-ab59-d369f274eb29-kube-api-access-vsvsz\") pod \"4ced0495-da0f-47b8-ab59-d369f274eb29\" (UID: \"4ced0495-da0f-47b8-ab59-d369f274eb29\") "
Sep 14 03:18:04.691976 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.691635    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09fd993a-f4ed-44c2-983d-73f7501e6b9b-home" (OuterVolumeSpecName: "home") pod "09fd993a-f4ed-44c2-983d-73f7501e6b9b" (UID: "09fd993a-f4ed-44c2-983d-73f7501e6b9b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:18:04.691976 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.691649    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09fd993a-f4ed-44c2-983d-73f7501e6b9b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "09fd993a-f4ed-44c2-983d-73f7501e6b9b" (UID: "09fd993a-f4ed-44c2-983d-73f7501e6b9b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:18:04.691976 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.691879    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09fd993a-f4ed-44c2-983d-73f7501e6b9b-home\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:18:04.691976 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.691900    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ced0495-da0f-47b8-ab59-d369f274eb29-home\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:18:04.691976 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.691640    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ced0495-da0f-47b8-ab59-d369f274eb29-model-cache" (OuterVolumeSpecName: "model-cache") pod "4ced0495-da0f-47b8-ab59-d369f274eb29" (UID: "4ced0495-da0f-47b8-ab59-d369f274eb29"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:18:04.691976 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.691914    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ced0495-da0f-47b8-ab59-d369f274eb29-tmp-dir\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:18:04.691976 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.691929    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09fd993a-f4ed-44c2-983d-73f7501e6b9b-model-cache\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:18:04.691976 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.691944    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09fd993a-f4ed-44c2-983d-73f7501e6b9b-tmp-dir\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:18:04.693572 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.693540    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09fd993a-f4ed-44c2-983d-73f7501e6b9b-dshm" (OuterVolumeSpecName: "dshm") pod "09fd993a-f4ed-44c2-983d-73f7501e6b9b" (UID: "09fd993a-f4ed-44c2-983d-73f7501e6b9b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:18:04.693722 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.693678    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ced0495-da0f-47b8-ab59-d369f274eb29-dshm" (OuterVolumeSpecName: "dshm") pod "4ced0495-da0f-47b8-ab59-d369f274eb29" (UID: "4ced0495-da0f-47b8-ab59-d369f274eb29"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:18:04.694022 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.693998    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ced0495-da0f-47b8-ab59-d369f274eb29-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4ced0495-da0f-47b8-ab59-d369f274eb29" (UID: "4ced0495-da0f-47b8-ab59-d369f274eb29"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 03:18:04.694110 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.694017    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ced0495-da0f-47b8-ab59-d369f274eb29-kube-api-access-vsvsz" (OuterVolumeSpecName: "kube-api-access-vsvsz") pod "4ced0495-da0f-47b8-ab59-d369f274eb29" (UID: "4ced0495-da0f-47b8-ab59-d369f274eb29"). InnerVolumeSpecName "kube-api-access-vsvsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:18:04.694110 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.694055    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09fd993a-f4ed-44c2-983d-73f7501e6b9b-kube-api-access-pbxvp" (OuterVolumeSpecName: "kube-api-access-pbxvp") pod "09fd993a-f4ed-44c2-983d-73f7501e6b9b" (UID: "09fd993a-f4ed-44c2-983d-73f7501e6b9b"). InnerVolumeSpecName "kube-api-access-pbxvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:18:04.694230 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.694159    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09fd993a-f4ed-44c2-983d-73f7501e6b9b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "09fd993a-f4ed-44c2-983d-73f7501e6b9b" (UID: "09fd993a-f4ed-44c2-983d-73f7501e6b9b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 03:18:04.792815 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.792775    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09fd993a-f4ed-44c2-983d-73f7501e6b9b-tls-certs\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:18:04.792815 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.792803    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ced0495-da0f-47b8-ab59-d369f274eb29-model-cache\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:18:04.792815 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.792813    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09fd993a-f4ed-44c2-983d-73f7501e6b9b-dshm\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:18:04.792815 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.792822    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ced0495-da0f-47b8-ab59-d369f274eb29-tls-certs\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:18:04.792815 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.792830    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsvsz\" (UniqueName: \"kubernetes.io/projected/4ced0495-da0f-47b8-ab59-d369f274eb29-kube-api-access-vsvsz\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:18:04.793507 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.792839    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ced0495-da0f-47b8-ab59-d369f274eb29-dshm\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:18:04.793507 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:04.792847    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbxvp\" (UniqueName: \"kubernetes.io/projected/09fd993a-f4ed-44c2-983d-73f7501e6b9b-kube-api-access-pbxvp\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:18:05.100158 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:05.100122    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0940ece5-1e43-46ce-86d4-3bc31c3777a2" path="/var/lib/kubelet/pods/0940ece5-1e43-46ce-86d4-3bc31c3777a2/volumes"
Sep 14 03:18:05.100445 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:05.100432    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a633a62e-b4c2-49d5-82ba-e6ae0b7d02ec" path="/var/lib/kubelet/pods/a633a62e-b4c2-49d5-82ba-e6ae0b7d02ec/volumes"
Sep 14 03:18:05.100734 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:05.100718    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd75830d-dc18-47e2-a0fc-7ea2e0cd21b4" path="/var/lib/kubelet/pods/cd75830d-dc18-47e2-a0fc-7ea2e0cd21b4/volumes"
Sep 14 03:18:05.232985 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:05.232947    2564 generic.go:356] "Generic (PLEG): container finished" podID="4ced0495-da0f-47b8-ab59-d369f274eb29" containerID="e006c7368c1c2260aa2ae0c60c877847dc3920aa684e3a6fd97df6569a6a6709" exitCode=0
Sep 14 03:18:05.233240 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:05.232993    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9c9f1e/distinct-2-w4-9c9f1e-kserve-58779c476f-qmt8b"
Sep 14 03:18:05.233240 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:05.233029    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9c9f1e/distinct-2-w4-9c9f1e-kserve-58779c476f-qmt8b" event={"ID":"4ced0495-da0f-47b8-ab59-d369f274eb29","Type":"ContainerDied","Data":"e006c7368c1c2260aa2ae0c60c877847dc3920aa684e3a6fd97df6569a6a6709"}
Sep 14 03:18:05.233240 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:05.233069    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9c9f1e/distinct-2-w4-9c9f1e-kserve-58779c476f-qmt8b" event={"ID":"4ced0495-da0f-47b8-ab59-d369f274eb29","Type":"ContainerDied","Data":"06e1af2b9d342a8e65e429698ccd2b57427073eea5ce92e0bc15f50ca675894d"}
Sep 14 03:18:05.233240 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:05.233088    2564 scope.go:117] "RemoveContainer" containerID="e006c7368c1c2260aa2ae0c60c877847dc3920aa684e3a6fd97df6569a6a6709"
Sep 14 03:18:05.235667 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:05.235641    2564 generic.go:356] "Generic (PLEG): container finished" podID="09fd993a-f4ed-44c2-983d-73f7501e6b9b" containerID="3bafbacfe580478b16a366fa151010c9508709cd6d16a8611ba4b826e7e89a5b" exitCode=0
Sep 14 03:18:05.235783 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:05.235681    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9c9f1e/distinct-w4-9c9f1e-kserve-579d94566b-tgjjk" event={"ID":"09fd993a-f4ed-44c2-983d-73f7501e6b9b","Type":"ContainerDied","Data":"3bafbacfe580478b16a366fa151010c9508709cd6d16a8611ba4b826e7e89a5b"}
Sep 14 03:18:05.235783 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:05.235728    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9c9f1e/distinct-w4-9c9f1e-kserve-579d94566b-tgjjk" event={"ID":"09fd993a-f4ed-44c2-983d-73f7501e6b9b","Type":"ContainerDied","Data":"cd1c084692167f0d8e5673872b74af5d982f90226836b0a6a02a72b9e7599786"}
Sep 14 03:18:05.235783 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:05.235751    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9c9f1e/distinct-w4-9c9f1e-kserve-579d94566b-tgjjk"
Sep 14 03:18:05.243064 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:05.243044    2564 scope.go:117] "RemoveContainer" containerID="e006c7368c1c2260aa2ae0c60c877847dc3920aa684e3a6fd97df6569a6a6709"
Sep 14 03:18:05.243382 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:18:05.243362    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e006c7368c1c2260aa2ae0c60c877847dc3920aa684e3a6fd97df6569a6a6709\": container with ID starting with e006c7368c1c2260aa2ae0c60c877847dc3920aa684e3a6fd97df6569a6a6709 not found: ID does not exist" containerID="e006c7368c1c2260aa2ae0c60c877847dc3920aa684e3a6fd97df6569a6a6709"
Sep 14 03:18:05.243481 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:05.243393    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e006c7368c1c2260aa2ae0c60c877847dc3920aa684e3a6fd97df6569a6a6709"} err="failed to get container status \"e006c7368c1c2260aa2ae0c60c877847dc3920aa684e3a6fd97df6569a6a6709\": rpc error: code = NotFound desc = could not find container \"e006c7368c1c2260aa2ae0c60c877847dc3920aa684e3a6fd97df6569a6a6709\": container with ID starting with e006c7368c1c2260aa2ae0c60c877847dc3920aa684e3a6fd97df6569a6a6709 not found: ID does not exist"
Sep 14 03:18:05.243481 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:05.243442    2564 scope.go:117] "RemoveContainer" containerID="3bafbacfe580478b16a366fa151010c9508709cd6d16a8611ba4b826e7e89a5b"
Sep 14 03:18:05.246742 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:05.246715    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:18:05.251886 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:05.251828    2564 scope.go:117] "RemoveContainer" containerID="3bafbacfe580478b16a366fa151010c9508709cd6d16a8611ba4b826e7e89a5b"
Sep 14 03:18:05.252350 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:18:05.252322    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bafbacfe580478b16a366fa151010c9508709cd6d16a8611ba4b826e7e89a5b\": container with ID starting with 3bafbacfe580478b16a366fa151010c9508709cd6d16a8611ba4b826e7e89a5b not found: ID does not exist" containerID="3bafbacfe580478b16a366fa151010c9508709cd6d16a8611ba4b826e7e89a5b"
Sep 14 03:18:05.252482 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:05.252361    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bafbacfe580478b16a366fa151010c9508709cd6d16a8611ba4b826e7e89a5b"} err="failed to get container status \"3bafbacfe580478b16a366fa151010c9508709cd6d16a8611ba4b826e7e89a5b\": rpc error: code = NotFound desc = could not find container \"3bafbacfe580478b16a366fa151010c9508709cd6d16a8611ba4b826e7e89a5b\": container with ID starting with 3bafbacfe580478b16a366fa151010c9508709cd6d16a8611ba4b826e7e89a5b not found: ID does not exist"
Sep 14 03:18:05.257580 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:05.257555    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-9c9f1e/distinct-w4-9c9f1e-kserve-579d94566b-tgjjk"]
Sep 14 03:18:05.259662 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:05.259636    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-9c9f1e/distinct-w4-9c9f1e-kserve-579d94566b-tgjjk"]
Sep 14 03:18:05.270384 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:05.270362    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-9c9f1e/distinct-2-w4-9c9f1e-kserve-58779c476f-qmt8b"]
Sep 14 03:18:05.274567 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:05.274542    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-9c9f1e/distinct-2-w4-9c9f1e-kserve-58779c476f-qmt8b"]
Sep 14 03:18:06.124043 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:06.124012    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:18:06.881789 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:06.881752    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:18:07.098643 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:07.098614    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09fd993a-f4ed-44c2-983d-73f7501e6b9b" path="/var/lib/kubelet/pods/09fd993a-f4ed-44c2-983d-73f7501e6b9b/volumes"
Sep 14 03:18:07.099037 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:07.099023    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ced0495-da0f-47b8-ab59-d369f274eb29" path="/var/lib/kubelet/pods/4ced0495-da0f-47b8-ab59-d369f274eb29/volumes"
Sep 14 03:18:07.506644 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:07.506608    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:18:07.598990 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:07.598956    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:18:07.746067 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:07.746036    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:18:07.955989 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:07.955950    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:18:08.048092 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:08.048055    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:18:08.268187 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:08.268092    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:18:08.370231 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:08.370195    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:18:09.087322 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:09.087288    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:18:09.961640 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:09.961607    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:18:10.875881 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:10.875852    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:18:10.987765 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:10.987721    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:18:11.135097 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:11.135016    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:18:14.457766 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:14.457732    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:18:14.558180 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:14.558152    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:18:14.718531 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:14.718440    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:18:21.353870 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:21.353831    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:18:26.200268 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:26.200223    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:18:26.450036 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:26.450002    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:18:26.957939 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:26.957892    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:18:27.086460 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:27.086397    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:18:27.426781 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:27.426736    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:18:33.490242 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:33.490185    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:18:43.688267 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:43.688233    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:18:44.769136 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:44.769096    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:18:54.116714 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:18:54.116675    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:19:19.264549 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:19:19.264518    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-177593/sub-test-model-177593-kserve-5dd94756c5-wtz98"]
Sep 14 03:19:19.265267 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:19:19.264927    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac41e03a-54d4-4b59-8e31-563773bd52ba" containerName="maas-api"
Sep 14 03:19:19.265267 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:19:19.264938    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac41e03a-54d4-4b59-8e31-563773bd52ba" containerName="maas-api"
Sep 14 03:19:19.265267 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:19:19.264951    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d13e2896-2b10-4f6c-8583-1d60c1390838" containerName="payload-pre-processing"
Sep 14 03:19:19.265267 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:19:19.264957    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d13e2896-2b10-4f6c-8583-1d60c1390838" containerName="payload-pre-processing"
Sep 14 03:19:19.265267 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:19:19.264965    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="242753da-219f-47d3-ad2e-f3866269b610" containerName="payload-processing"
Sep 14 03:19:19.265267 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:19:19.264971    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="242753da-219f-47d3-ad2e-f3866269b610" containerName="payload-processing"
Sep 14 03:19:19.265267 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:19:19.264988    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09fd993a-f4ed-44c2-983d-73f7501e6b9b" containerName="main"
Sep 14 03:19:19.265267 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:19:19.264993    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="09fd993a-f4ed-44c2-983d-73f7501e6b9b" containerName="main"
Sep 14 03:19:19.265267 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:19:19.265001    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ced0495-da0f-47b8-ab59-d369f274eb29" containerName="main"
Sep 14 03:19:19.265267 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:19:19.265006    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ced0495-da0f-47b8-ab59-d369f274eb29" containerName="main"
Sep 14 03:19:19.265267 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:19:19.265014    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9a686f2-3fa7-4b42-b237-53be6ba2b050" containerName="main"
Sep 14 03:19:19.265267 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:19:19.265019    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9a686f2-3fa7-4b42-b237-53be6ba2b050" containerName="main"
Sep 14 03:19:19.265267 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:19:19.265027    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3eed608f-4d31-4708-ac7c-9a29fe254a1b" containerName="payload-pre-processing"
Sep 14 03:19:19.265267 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:19:19.265033    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3eed608f-4d31-4708-ac7c-9a29fe254a1b" containerName="payload-pre-processing"
Sep 14 03:19:19.265267 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:19:19.265040    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd75830d-dc18-47e2-a0fc-7ea2e0cd21b4" containerName="payload-processing"
Sep 14 03:19:19.265267 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:19:19.265045    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd75830d-dc18-47e2-a0fc-7ea2e0cd21b4" containerName="payload-processing"
Sep 14 03:19:19.265267 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:19:19.265050    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a69a896-145c-4365-9444-3ea9f537ae6b" containerName="payload-pre-processing"
Sep 14 03:19:19.265267 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:19:19.265055    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a69a896-145c-4365-9444-3ea9f537ae6b" containerName="payload-pre-processing"
Sep 14 03:19:19.265267 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:19:19.265060    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5437a89f-6ac4-46f0-909d-808ecf23e86f" containerName="maas-api"
Sep 14 03:19:19.265267 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:19:19.265065    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5437a89f-6ac4-46f0-909d-808ecf23e86f" containerName="maas-api"
Sep 14 03:19:19.265267 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:19:19.265071    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49ef063e-3050-42ba-aa15-4fc3504d52ef" containerName="maas-api"
Sep 14 03:19:19.265267 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:19:19.265079    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="49ef063e-3050-42ba-aa15-4fc3504d52ef" containerName="maas-api"
Sep 14 03:19:19.265267 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:19:19.265087    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0940ece5-1e43-46ce-86d4-3bc31c3777a2" containerName="maas-api"
Sep 14 03:19:19.265267 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:19:19.265092    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0940ece5-1e43-46ce-86d4-3bc31c3777a2" containerName="maas-api"
Sep 14 03:19:19.265267 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:19:19.265143    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0940ece5-1e43-46ce-86d4-3bc31c3777a2" containerName="maas-api"
Sep 14 03:19:19.265267 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:19:19.265151    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a69a896-145c-4365-9444-3ea9f537ae6b" containerName="payload-pre-processing"
Sep 14 03:19:19.265267 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:19:19.265156    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="3eed608f-4d31-4708-ac7c-9a29fe254a1b" containerName="payload-pre-processing"
Sep 14 03:19:19.265267 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:19:19.265161    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9a686f2-3fa7-4b42-b237-53be6ba2b050" containerName="main"
Sep 14 03:19:19.265267 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:19:19.265167    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="242753da-219f-47d3-ad2e-f3866269b610" containerName="payload-processing"
Sep 14 03:19:19.265267 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:19:19.265172    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ced0495-da0f-47b8-ab59-d369f274eb29" containerName="main"
Sep 14 03:19:19.265267 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:19:19.265215    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5437a89f-6ac4-46f0-909d-808ecf23e86f" containerName="maas-api"
Sep 14 03:19:19.265267 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:19:19.265221    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac41e03a-54d4-4b59-8e31-563773bd52ba" containerName="maas-api"
Sep 14 03:19:19.265267 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:19:19.265235    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="09fd993a-f4ed-44c2-983d-73f7501e6b9b" containerName="main"
Sep 14 03:19:19.265267 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:19:19.265246    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d13e2896-2b10-4f6c-8583-1d60c1390838" containerName="payload-pre-processing"
Sep 14 03:19:19.265267 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:19:19.265257    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd75830d-dc18-47e2-a0fc-7ea2e0cd21b4" containerName="payload-processing"
Sep 14 03:19:19.265267 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:19:19.265265    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="49ef063e-3050-42ba-aa15-4fc3504d52ef" containerName="maas-api"
Sep 14 03:19:19.268663 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:19:19.268647    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-177593/sub-test-model-177593-kserve-5dd94756c5-wtz98"
Sep 14 03:19:19.271641 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:19:19.271518    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-177593\"/\"openshift-service-ca.crt\""
Sep 14 03:19:19.271641 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:19:19.271529    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-177593\"/\"default-dockercfg-54rpf\""
Sep 14 03:19:19.272932 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:19:19.272903    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-177593\"/\"sub-test-model-177593-kserve-self-signed-certs\""
Sep 14 03:19:19.272932 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:19:19.272925    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-177593\"/\"kube-root-ca.crt\""
Sep 14 03:19:19.281291 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:19:19.281269    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-177593/sub-test-model-177593-kserve-5dd94756c5-wtz98"]
Sep 14 03:19:19.290502 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:19:19.290480    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cdsp\" (UniqueName: \"kubernetes.io/projected/563b9999-0dde-4584-995c-004fdcfa21bd-kube-api-access-4cdsp\") pod \"sub-test-model-177593-kserve-5dd94756c5-wtz98\" (UID: \"563b9999-0dde-4584-995c-004fdcfa21bd\") " pod="ai-tenant-e2e-shared-b-w3-177593/sub-test-model-177593-kserve-5dd94756c5-wtz98"
Sep 14 03:19:19.290600 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:19:19.290524    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/563b9999-0dde-4584-995c-004fdcfa21bd-tls-certs\") pod \"sub-test-model-177593-kserve-5dd94756c5-wtz98\" (UID: \"563b9999-0dde-4584-995c-004fdcfa21bd\") " pod="ai-tenant-e2e-shared-b-w3-177593/sub-test-model-177593-kserve-5dd94756c5-wtz98"
Sep 14 03:19:19.290648 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:19:19.290597    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/563b9999-0dde-4584-995c-004fdcfa21bd-tmp-dir\") pod \"sub-test-model-177593-kserve-5dd94756c5-wtz98\" (UID: \"563b9999-0dde-4584-995c-004fdcfa21bd\") " pod="ai-tenant-e2e-shared-b-w3-177593/sub-test-model-177593-kserve-5dd94756c5-wtz98"
Sep 14 03:19:19.290648 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:19:19.290628    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/563b9999-0dde-4584-995c-004fdcfa21bd-home\") pod \"sub-test-model-177593-kserve-5dd94756c5-wtz98\" (UID: \"563b9999-0dde-4584-995c-004fdcfa21bd\") " pod="ai-tenant-e2e-shared-b-w3-177593/sub-test-model-177593-kserve-5dd94756c5-wtz98"
Sep 14 03:19:19.290714 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:19:19.290658    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/563b9999-0dde-4584-995c-004fdcfa21bd-model-cache\") pod \"sub-test-model-177593-kserve-5dd94756c5-wtz98\" (UID: \"563b9999-0dde-4584-995c-004fdcfa21bd\") " pod="ai-tenant-e2e-shared-b-w3-177593/sub-test-model-177593-kserve-5dd94756c5-wtz98"
Sep 14 03:19:19.290779 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:19:19.290762    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/563b9999-0dde-4584-995c-004fdcfa21bd-dshm\") pod \"sub-test-model-177593-kserve-5dd94756c5-wtz98\" (UID: \"563b9999-0dde-4584-995c-004fdcfa21bd\") " pod="ai-tenant-e2e-shared-b-w3-177593/sub-test-model-177593-kserve-5dd94756c5-wtz98"
Sep 14 03:19:19.306835 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:19:19.306768    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:19:19.392137 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:19:19.392102    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/563b9999-0dde-4584-995c-004fdcfa21bd-dshm\") pod \"sub-test-model-177593-kserve-5dd94756c5-wtz98\" (UID: \"563b9999-0dde-4584-995c-004fdcfa21bd\") " pod="ai-tenant-e2e-shared-b-w3-177593/sub-test-model-177593-kserve-5dd94756c5-wtz98"
Sep 14 03:19:19.392297 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:19:19.392149    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cdsp\" (UniqueName: \"kubernetes.io/projected/563b9999-0dde-4584-995c-004fdcfa21bd-kube-api-access-4cdsp\") pod \"sub-test-model-177593-kserve-5dd94756c5-wtz98\" (UID: \"563b9999-0dde-4584-995c-004fdcfa21bd\") " pod="ai-tenant-e2e-shared-b-w3-177593/sub-test-model-177593-kserve-5dd94756c5-wtz98"
Sep 14 03:19:19.392297 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:19:19.392187    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/563b9999-0dde-4584-995c-004fdcfa21bd-tls-certs\") pod \"sub-test-model-177593-kserve-5dd94756c5-wtz98\" (UID: \"563b9999-0dde-4584-995c-004fdcfa21bd\") " pod="ai-tenant-e2e-shared-b-w3-177593/sub-test-model-177593-kserve-5dd94756c5-wtz98"
Sep 14 03:19:19.392297 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:19:19.392237    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/563b9999-0dde-4584-995c-004fdcfa21bd-tmp-dir\") pod \"sub-test-model-177593-kserve-5dd94756c5-wtz98\" (UID: \"563b9999-0dde-4584-995c-004fdcfa21bd\") " pod="ai-tenant-e2e-shared-b-w3-177593/sub-test-model-177593-kserve-5dd94756c5-wtz98"
Sep 14 03:19:19.392297 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:19:19.392274    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/563b9999-0dde-4584-995c-004fdcfa21bd-home\") pod \"sub-test-model-177593-kserve-5dd94756c5-wtz98\" (UID: \"563b9999-0dde-4584-995c-004fdcfa21bd\") " pod="ai-tenant-e2e-shared-b-w3-177593/sub-test-model-177593-kserve-5dd94756c5-wtz98"
Sep 14 03:19:19.392553 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:19:19.392303    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/563b9999-0dde-4584-995c-004fdcfa21bd-model-cache\") pod \"sub-test-model-177593-kserve-5dd94756c5-wtz98\" (UID: \"563b9999-0dde-4584-995c-004fdcfa21bd\") " pod="ai-tenant-e2e-shared-b-w3-177593/sub-test-model-177593-kserve-5dd94756c5-wtz98"
Sep 14 03:19:19.392699 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:19:19.392657    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/563b9999-0dde-4584-995c-004fdcfa21bd-tmp-dir\") pod \"sub-test-model-177593-kserve-5dd94756c5-wtz98\" (UID: \"563b9999-0dde-4584-995c-004fdcfa21bd\") " pod="ai-tenant-e2e-shared-b-w3-177593/sub-test-model-177593-kserve-5dd94756c5-wtz98"
Sep 14 03:19:19.392699 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:19:19.392683    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/563b9999-0dde-4584-995c-004fdcfa21bd-model-cache\") pod \"sub-test-model-177593-kserve-5dd94756c5-wtz98\" (UID: \"563b9999-0dde-4584-995c-004fdcfa21bd\") " pod="ai-tenant-e2e-shared-b-w3-177593/sub-test-model-177593-kserve-5dd94756c5-wtz98"
Sep 14 03:19:19.392856 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:19:19.392717    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/563b9999-0dde-4584-995c-004fdcfa21bd-home\") pod \"sub-test-model-177593-kserve-5dd94756c5-wtz98\" (UID: \"563b9999-0dde-4584-995c-004fdcfa21bd\") " pod="ai-tenant-e2e-shared-b-w3-177593/sub-test-model-177593-kserve-5dd94756c5-wtz98"
Sep 14 03:19:19.394394 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:19:19.394371    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/563b9999-0dde-4584-995c-004fdcfa21bd-dshm\") pod \"sub-test-model-177593-kserve-5dd94756c5-wtz98\" (UID: \"563b9999-0dde-4584-995c-004fdcfa21bd\") " pod="ai-tenant-e2e-shared-b-w3-177593/sub-test-model-177593-kserve-5dd94756c5-wtz98"
Sep 14 03:19:19.394582 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:19:19.394565    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/563b9999-0dde-4584-995c-004fdcfa21bd-tls-certs\") pod \"sub-test-model-177593-kserve-5dd94756c5-wtz98\" (UID: \"563b9999-0dde-4584-995c-004fdcfa21bd\") " pod="ai-tenant-e2e-shared-b-w3-177593/sub-test-model-177593-kserve-5dd94756c5-wtz98"
Sep 14 03:19:19.401319 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:19:19.401295    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cdsp\" (UniqueName: \"kubernetes.io/projected/563b9999-0dde-4584-995c-004fdcfa21bd-kube-api-access-4cdsp\") pod \"sub-test-model-177593-kserve-5dd94756c5-wtz98\" (UID: \"563b9999-0dde-4584-995c-004fdcfa21bd\") " pod="ai-tenant-e2e-shared-b-w3-177593/sub-test-model-177593-kserve-5dd94756c5-wtz98"
Sep 14 03:19:19.578146 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:19:19.578115    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-177593/sub-test-model-177593-kserve-5dd94756c5-wtz98"
Sep 14 03:19:19.852381 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:19:19.852298    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-177593/sub-test-model-177593-kserve-5dd94756c5-wtz98"]
Sep 14 03:19:19.855383 ip-10-0-141-4 kubenswrapper[2564]: W0914 03:19:19.855355    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod563b9999_0dde_4584_995c_004fdcfa21bd.slice/crio-da278da13faa2a09b589d44de8782231c4ac54e65b2d1a01ef2f9e9d1bcc5fb4 WatchSource:0}: Error finding container da278da13faa2a09b589d44de8782231c4ac54e65b2d1a01ef2f9e9d1bcc5fb4: Status 404 returned error can't find the container with id da278da13faa2a09b589d44de8782231c4ac54e65b2d1a01ef2f9e9d1bcc5fb4
Sep 14 03:19:20.634083 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:19:20.634048    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-177593/sub-test-model-177593-kserve-5dd94756c5-wtz98" event={"ID":"563b9999-0dde-4584-995c-004fdcfa21bd","Type":"ContainerStarted","Data":"b641647cf319fff44714e6e3116955c35c0716ccce98beb5ed6450cdaa963fac"}
Sep 14 03:19:20.634469 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:19:20.634093    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-177593/sub-test-model-177593-kserve-5dd94756c5-wtz98" event={"ID":"563b9999-0dde-4584-995c-004fdcfa21bd","Type":"ContainerStarted","Data":"da278da13faa2a09b589d44de8782231c4ac54e65b2d1a01ef2f9e9d1bcc5fb4"}
Sep 14 03:19:20.670990 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:19:20.670946    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-177593/sub-test-model-177593-kserve-5dd94756c5-wtz98" podStartSLOduration=1.670931118 podStartE2EDuration="1.670931118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:19:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:19:20.668802353 +0000 UTC m=+1606.049886553" watchObservedRunningTime="2026-09-14 03:19:20.670931118 +0000 UTC m=+1606.052015332"
Sep 14 03:19:29.578282 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:19:29.578207    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-177593/sub-test-model-177593-kserve-5dd94756c5-wtz98"
Sep 14 03:19:29.578282 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:19:29.578245    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-177593/sub-test-model-177593-kserve-5dd94756c5-wtz98"
Sep 14 03:19:29.590504 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:19:29.590477    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-177593/sub-test-model-177593-kserve-5dd94756c5-wtz98"
Sep 14 03:19:29.689927 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:19:29.689895    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-177593/sub-test-model-177593-kserve-5dd94756c5-wtz98"
Sep 14 03:19:42.713231 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:19:42.713175    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:20:03.832506 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:20:03.832467    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:20:04.653756 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:20:04.653714    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-177593/sub-test-model-177593-kserve-5dd94756c5-wtz98"]
Sep 14 03:20:04.679003 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:20:04.678967    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:20:19.672529 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:20:19.672470    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-177593/sub-test-model-177593-kserve-5dd94756c5-wtz98" podUID="563b9999-0dde-4584-995c-004fdcfa21bd" containerName="main" containerID="cri-o://b641647cf319fff44714e6e3116955c35c0716ccce98beb5ed6450cdaa963fac" gracePeriod=45
Sep 14 03:20:19.680380 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:20:19.680332    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-177593/sub-test-model-177593-kserve-5dd94756c5-wtz98" podUID="563b9999-0dde-4584-995c-004fdcfa21bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": dial tcp 10.134.0.78:8000: connect: connection refused"
Sep 14 03:20:19.920305 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:20:19.920283    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-177593/sub-test-model-177593-kserve-5dd94756c5-wtz98"
Sep 14 03:20:19.935914 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:20:19.935856    2564 generic.go:356] "Generic (PLEG): container finished" podID="563b9999-0dde-4584-995c-004fdcfa21bd" containerID="b641647cf319fff44714e6e3116955c35c0716ccce98beb5ed6450cdaa963fac" exitCode=0
Sep 14 03:20:19.935914 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:20:19.935898    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-177593/sub-test-model-177593-kserve-5dd94756c5-wtz98"
Sep 14 03:20:19.936088 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:20:19.935923    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-177593/sub-test-model-177593-kserve-5dd94756c5-wtz98" event={"ID":"563b9999-0dde-4584-995c-004fdcfa21bd","Type":"ContainerDied","Data":"b641647cf319fff44714e6e3116955c35c0716ccce98beb5ed6450cdaa963fac"}
Sep 14 03:20:19.936088 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:20:19.935963    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-177593/sub-test-model-177593-kserve-5dd94756c5-wtz98" event={"ID":"563b9999-0dde-4584-995c-004fdcfa21bd","Type":"ContainerDied","Data":"da278da13faa2a09b589d44de8782231c4ac54e65b2d1a01ef2f9e9d1bcc5fb4"}
Sep 14 03:20:19.936088 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:20:19.935981    2564 scope.go:117] "RemoveContainer" containerID="b641647cf319fff44714e6e3116955c35c0716ccce98beb5ed6450cdaa963fac"
Sep 14 03:20:19.947122 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:20:19.947101    2564 scope.go:117] "RemoveContainer" containerID="b641647cf319fff44714e6e3116955c35c0716ccce98beb5ed6450cdaa963fac"
Sep 14 03:20:19.947484 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:20:19.947459    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b641647cf319fff44714e6e3116955c35c0716ccce98beb5ed6450cdaa963fac\": container with ID starting with b641647cf319fff44714e6e3116955c35c0716ccce98beb5ed6450cdaa963fac not found: ID does not exist" containerID="b641647cf319fff44714e6e3116955c35c0716ccce98beb5ed6450cdaa963fac"
Sep 14 03:20:19.947533 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:20:19.947500    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b641647cf319fff44714e6e3116955c35c0716ccce98beb5ed6450cdaa963fac"} err="failed to get container status \"b641647cf319fff44714e6e3116955c35c0716ccce98beb5ed6450cdaa963fac\": rpc error: code = NotFound desc = could not find container \"b641647cf319fff44714e6e3116955c35c0716ccce98beb5ed6450cdaa963fac\": container with ID starting with b641647cf319fff44714e6e3116955c35c0716ccce98beb5ed6450cdaa963fac not found: ID does not exist"
Sep 14 03:20:20.087911 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:20:20.087881    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/563b9999-0dde-4584-995c-004fdcfa21bd-dshm\") pod \"563b9999-0dde-4584-995c-004fdcfa21bd\" (UID: \"563b9999-0dde-4584-995c-004fdcfa21bd\") "
Sep 14 03:20:20.088083 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:20:20.087927    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/563b9999-0dde-4584-995c-004fdcfa21bd-model-cache\") pod \"563b9999-0dde-4584-995c-004fdcfa21bd\" (UID: \"563b9999-0dde-4584-995c-004fdcfa21bd\") "
Sep 14 03:20:20.088083 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:20:20.087965    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/563b9999-0dde-4584-995c-004fdcfa21bd-tmp-dir\") pod \"563b9999-0dde-4584-995c-004fdcfa21bd\" (UID: \"563b9999-0dde-4584-995c-004fdcfa21bd\") "
Sep 14 03:20:20.088083 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:20:20.088019    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4cdsp\" (UniqueName: \"kubernetes.io/projected/563b9999-0dde-4584-995c-004fdcfa21bd-kube-api-access-4cdsp\") pod \"563b9999-0dde-4584-995c-004fdcfa21bd\" (UID: \"563b9999-0dde-4584-995c-004fdcfa21bd\") "
Sep 14 03:20:20.088083 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:20:20.088046    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/563b9999-0dde-4584-995c-004fdcfa21bd-tls-certs\") pod \"563b9999-0dde-4584-995c-004fdcfa21bd\" (UID: \"563b9999-0dde-4584-995c-004fdcfa21bd\") "
Sep 14 03:20:20.088369 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:20:20.088085    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/563b9999-0dde-4584-995c-004fdcfa21bd-home\") pod \"563b9999-0dde-4584-995c-004fdcfa21bd\" (UID: \"563b9999-0dde-4584-995c-004fdcfa21bd\") "
Sep 14 03:20:20.088369 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:20:20.088265    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/563b9999-0dde-4584-995c-004fdcfa21bd-model-cache" (OuterVolumeSpecName: "model-cache") pod "563b9999-0dde-4584-995c-004fdcfa21bd" (UID: "563b9999-0dde-4584-995c-004fdcfa21bd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:20:20.088369 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:20:20.088272    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/563b9999-0dde-4584-995c-004fdcfa21bd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "563b9999-0dde-4584-995c-004fdcfa21bd" (UID: "563b9999-0dde-4584-995c-004fdcfa21bd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:20:20.088565 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:20:20.088399    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/563b9999-0dde-4584-995c-004fdcfa21bd-home" (OuterVolumeSpecName: "home") pod "563b9999-0dde-4584-995c-004fdcfa21bd" (UID: "563b9999-0dde-4584-995c-004fdcfa21bd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:20:20.088565 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:20:20.088472    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/563b9999-0dde-4584-995c-004fdcfa21bd-model-cache\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:20:20.088565 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:20:20.088487    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/563b9999-0dde-4584-995c-004fdcfa21bd-tmp-dir\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:20:20.090231 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:20:20.090207    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/563b9999-0dde-4584-995c-004fdcfa21bd-kube-api-access-4cdsp" (OuterVolumeSpecName: "kube-api-access-4cdsp") pod "563b9999-0dde-4584-995c-004fdcfa21bd" (UID: "563b9999-0dde-4584-995c-004fdcfa21bd"). InnerVolumeSpecName "kube-api-access-4cdsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:20:20.090231 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:20:20.090207    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/563b9999-0dde-4584-995c-004fdcfa21bd-dshm" (OuterVolumeSpecName: "dshm") pod "563b9999-0dde-4584-995c-004fdcfa21bd" (UID: "563b9999-0dde-4584-995c-004fdcfa21bd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 03:20:20.090363 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:20:20.090290    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/563b9999-0dde-4584-995c-004fdcfa21bd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "563b9999-0dde-4584-995c-004fdcfa21bd" (UID: "563b9999-0dde-4584-995c-004fdcfa21bd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 03:20:20.189532 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:20:20.189450    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/563b9999-0dde-4584-995c-004fdcfa21bd-dshm\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:20:20.189532 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:20:20.189479    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cdsp\" (UniqueName: \"kubernetes.io/projected/563b9999-0dde-4584-995c-004fdcfa21bd-kube-api-access-4cdsp\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:20:20.189532 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:20:20.189490    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/563b9999-0dde-4584-995c-004fdcfa21bd-tls-certs\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:20:20.189532 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:20:20.189504    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/563b9999-0dde-4584-995c-004fdcfa21bd-home\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:20:20.257803 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:20:20.257773    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-177593/sub-test-model-177593-kserve-5dd94756c5-wtz98"]
Sep 14 03:20:20.262443 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:20:20.262419    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-177593/sub-test-model-177593-kserve-5dd94756c5-wtz98"]
Sep 14 03:20:21.098012 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:20:21.097979    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="563b9999-0dde-4584-995c-004fdcfa21bd" path="/var/lib/kubelet/pods/563b9999-0dde-4584-995c-004fdcfa21bd/volumes"
Sep 14 03:20:27.847398 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:20:27.847357    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:20:45.697815 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:20:45.697760    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:21:23.930197 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:21:23.930149    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:21:24.727139 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:21:24.727102    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:21:25.166988 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:21:25.166956    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:21:25.474701 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:21:25.474590    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:21:25.770258 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:21:25.770211    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:21:27.934950 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:21:27.934914    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:21:28.608884 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:21:28.608846    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:21:28.741585 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:21:28.741544    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:21:28.838277 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:21:28.838237    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:21:28.938992 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:21:28.938872    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:21:29.638686 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:21:29.638626    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:21:30.531053 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:21:30.531012    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d081a3-6dbff6ccd-jfqdr"]
Sep 14 03:21:30.531628 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:21:30.531604    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="563b9999-0dde-4584-995c-004fdcfa21bd" containerName="main"
Sep 14 03:21:30.531628 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:21:30.531628    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="563b9999-0dde-4584-995c-004fdcfa21bd" containerName="main"
Sep 14 03:21:30.531834 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:21:30.531729    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="563b9999-0dde-4584-995c-004fdcfa21bd" containerName="main"
Sep 14 03:21:30.534567 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:21:30.534544    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d081a3-6dbff6ccd-jfqdr"
Sep 14 03:21:30.538057 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:21:30.538033    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-d081a3-serving-cert\""
Sep 14 03:21:30.548233 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:21:30.548200    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d081a3-6dbff6ccd-jfqdr"]
Sep 14 03:21:30.559847 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:21:30.559808    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:21:30.656847 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:21:30.656800    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:21:30.671057 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:21:30.671022    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45433dea-3cc8-4746-9090-83be7ade5bd3-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-d081a3-6dbff6ccd-jfqdr\" (UID: \"45433dea-3cc8-4746-9090-83be7ade5bd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d081a3-6dbff6ccd-jfqdr"
Sep 14 03:21:30.671197 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:21:30.671068    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45433dea-3cc8-4746-9090-83be7ade5bd3-metrics-certs\") pod \"maas-api-e2e-ipp-a-d081a3-6dbff6ccd-jfqdr\" (UID: \"45433dea-3cc8-4746-9090-83be7ade5bd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d081a3-6dbff6ccd-jfqdr"
Sep 14 03:21:30.671197 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:21:30.671134    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz7jb\" (UniqueName: \"kubernetes.io/projected/45433dea-3cc8-4746-9090-83be7ade5bd3-kube-api-access-hz7jb\") pod \"maas-api-e2e-ipp-a-d081a3-6dbff6ccd-jfqdr\" (UID: \"45433dea-3cc8-4746-9090-83be7ade5bd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d081a3-6dbff6ccd-jfqdr"
Sep 14 03:21:30.671274 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:21:30.671228    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45433dea-3cc8-4746-9090-83be7ade5bd3-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d081a3-6dbff6ccd-jfqdr\" (UID: \"45433dea-3cc8-4746-9090-83be7ade5bd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d081a3-6dbff6ccd-jfqdr"
Sep 14 03:21:30.771821 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:21:30.771773    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45433dea-3cc8-4746-9090-83be7ade5bd3-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d081a3-6dbff6ccd-jfqdr\" (UID: \"45433dea-3cc8-4746-9090-83be7ade5bd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d081a3-6dbff6ccd-jfqdr"
Sep 14 03:21:30.772041 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:21:30.771837    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45433dea-3cc8-4746-9090-83be7ade5bd3-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-d081a3-6dbff6ccd-jfqdr\" (UID: \"45433dea-3cc8-4746-9090-83be7ade5bd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d081a3-6dbff6ccd-jfqdr"
Sep 14 03:21:30.772041 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:21:30.771862    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45433dea-3cc8-4746-9090-83be7ade5bd3-metrics-certs\") pod \"maas-api-e2e-ipp-a-d081a3-6dbff6ccd-jfqdr\" (UID: \"45433dea-3cc8-4746-9090-83be7ade5bd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d081a3-6dbff6ccd-jfqdr"
Sep 14 03:21:30.772041 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:21:30.771913    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz7jb\" (UniqueName: \"kubernetes.io/projected/45433dea-3cc8-4746-9090-83be7ade5bd3-kube-api-access-hz7jb\") pod \"maas-api-e2e-ipp-a-d081a3-6dbff6ccd-jfqdr\" (UID: \"45433dea-3cc8-4746-9090-83be7ade5bd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d081a3-6dbff6ccd-jfqdr"
Sep 14 03:21:30.772041 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:21:30.771956    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-d081a3-serving-cert: secret "maas-api-e2e-ipp-a-d081a3-serving-cert" not found
Sep 14 03:21:30.772041 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:21:30.772033    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45433dea-3cc8-4746-9090-83be7ade5bd3-maas-api-tls podName:45433dea-3cc8-4746-9090-83be7ade5bd3 nodeName:}" failed. No retries permitted until 2026-09-14 03:21:31.272011776 +0000 UTC m=+1736.653095953 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/45433dea-3cc8-4746-9090-83be7ade5bd3-maas-api-tls") pod "maas-api-e2e-ipp-a-d081a3-6dbff6ccd-jfqdr" (UID: "45433dea-3cc8-4746-9090-83be7ade5bd3") : secret "maas-api-e2e-ipp-a-d081a3-serving-cert" not found
Sep 14 03:21:30.772671 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:21:30.772653    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45433dea-3cc8-4746-9090-83be7ade5bd3-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-d081a3-6dbff6ccd-jfqdr\" (UID: \"45433dea-3cc8-4746-9090-83be7ade5bd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d081a3-6dbff6ccd-jfqdr"
Sep 14 03:21:30.774318 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:21:30.774299    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45433dea-3cc8-4746-9090-83be7ade5bd3-metrics-certs\") pod \"maas-api-e2e-ipp-a-d081a3-6dbff6ccd-jfqdr\" (UID: \"45433dea-3cc8-4746-9090-83be7ade5bd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d081a3-6dbff6ccd-jfqdr"
Sep 14 03:21:30.784785 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:21:30.784715    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz7jb\" (UniqueName: \"kubernetes.io/projected/45433dea-3cc8-4746-9090-83be7ade5bd3-kube-api-access-hz7jb\") pod \"maas-api-e2e-ipp-a-d081a3-6dbff6ccd-jfqdr\" (UID: \"45433dea-3cc8-4746-9090-83be7ade5bd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d081a3-6dbff6ccd-jfqdr"
Sep 14 03:21:30.949562 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:21:30.949521    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:21:31.061846 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:21:31.061735    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:21:31.154029 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:21:31.153988    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:21:31.277295 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:21:31.277261    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45433dea-3cc8-4746-9090-83be7ade5bd3-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d081a3-6dbff6ccd-jfqdr\" (UID: \"45433dea-3cc8-4746-9090-83be7ade5bd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d081a3-6dbff6ccd-jfqdr"
Sep 14 03:21:31.279805 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:21:31.279781    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45433dea-3cc8-4746-9090-83be7ade5bd3-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d081a3-6dbff6ccd-jfqdr\" (UID: \"45433dea-3cc8-4746-9090-83be7ade5bd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d081a3-6dbff6ccd-jfqdr"
Sep 14 03:21:31.446113 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:21:31.446030    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d081a3-6dbff6ccd-jfqdr"
Sep 14 03:21:31.726674 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:21:31.726641    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d081a3-6dbff6ccd-jfqdr"]
Sep 14 03:21:31.728057 ip-10-0-141-4 kubenswrapper[2564]: W0914 03:21:31.728030    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45433dea_3cc8_4746_9090_83be7ade5bd3.slice/crio-3bf2d73084ea6d8de9e57b4681a7c8962cea90ad4802bb24acea2587386a14a1 WatchSource:0}: Error finding container 3bf2d73084ea6d8de9e57b4681a7c8962cea90ad4802bb24acea2587386a14a1: Status 404 returned error can't find the container with id 3bf2d73084ea6d8de9e57b4681a7c8962cea90ad4802bb24acea2587386a14a1
Sep 14 03:21:31.729392 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:21:31.729367    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 14 03:21:32.318605 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:21:32.318567    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d081a3-6dbff6ccd-jfqdr" event={"ID":"45433dea-3cc8-4746-9090-83be7ade5bd3","Type":"ContainerStarted","Data":"c75dde86a033733b359918ae22a814a96e4cb5c82c0514acf5ba5e4853693658"}
Sep 14 03:21:32.318815 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:21:32.318622    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d081a3-6dbff6ccd-jfqdr" event={"ID":"45433dea-3cc8-4746-9090-83be7ade5bd3","Type":"ContainerStarted","Data":"3bf2d73084ea6d8de9e57b4681a7c8962cea90ad4802bb24acea2587386a14a1"}
Sep 14 03:21:32.318815 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:21:32.318686    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d081a3-6dbff6ccd-jfqdr"
Sep 14 03:21:32.349766 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:21:32.349693    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d081a3-6dbff6ccd-jfqdr" podStartSLOduration=1.907477615 podStartE2EDuration="2.349630802s" totalImagesPullingTime="442.153187ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:21:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:21:32.34046106 +0000 UTC m=+1737.721545260" watchObservedRunningTime="2026-09-14 03:21:32.349630802 +0000 UTC m=+1737.730715001"
Sep 14 03:21:35.377265 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:21:35.377225    2564 scope.go:117] "RemoveContainer" containerID="1f47a0cca98f448fd69878725a945fdbc8bf6a7dcfe7b9c0fbec06de96300b5c"
Sep 14 03:21:38.330050 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:21:38.330018    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d081a3-6dbff6ccd-jfqdr"
Sep 14 03:21:49.933723 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:21:49.933680    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:21:50.034541 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:21:50.034511    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:21:52.687295 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:21:52.687251    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:21:53.450064 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:21:53.450028    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:21:53.589539 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:21:53.589489    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:21:54.274157 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:21:54.274101    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:21:54.566875 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:21:54.566775    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:21:54.636197 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:21:54.636149    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:21:54.970968 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:21:54.970874    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:21:55.398782 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:21:55.398736    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:21:57.136591 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:21:57.136557    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:21:57.898611 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:21:57.898569    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:21:58.015915 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:21:58.015859    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:21:58.101462 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:21:58.101394    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:21:58.939147 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:21:58.939109    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:21:59.753250 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:21:59.753210    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:22:00.580204 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:22:00.580168    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:22:00.675690 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:22:00.675642    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:22:00.775009 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:22:00.774969    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:22:00.975636 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:22:00.975545    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:22:01.075421 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:22:01.075364    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:22:01.139145 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:22:01.139108    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8d29d7-548dddc64b-5tr2f"]
Sep 14 03:22:01.142757 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:22:01.142738    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8d29d7-5896fdfbdc-v2trc"]
Sep 14 03:22:01.142873 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:22:01.142853    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8d29d7-548dddc64b-5tr2f"
Sep 14 03:22:01.146506 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:22:01.146482    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8d29d7-5896fdfbdc-v2trc"
Sep 14 03:22:01.161912 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:22:01.161884    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-8d29d7-dockercfg-k5xnt\""
Sep 14 03:22:01.162070 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:22:01.162019    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-8d29d7\""
Sep 14 03:22:01.179296 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:22:01.179270    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8d29d7-548dddc64b-5tr2f"]
Sep 14 03:22:01.184111 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:22:01.184087    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8d29d7-5896fdfbdc-v2trc"]
Sep 14 03:22:01.189271 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:22:01.189240    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnqsq\" (UniqueName: \"kubernetes.io/projected/891107e4-feb4-41fe-b1ef-8ac6721198dc-kube-api-access-hnqsq\") pod \"payload-pre-processing-e2e-ipp-b-8d29d7-548dddc64b-5tr2f\" (UID: \"891107e4-feb4-41fe-b1ef-8ac6721198dc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8d29d7-548dddc64b-5tr2f"
Sep 14 03:22:01.189389 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:22:01.189370    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/891107e4-feb4-41fe-b1ef-8ac6721198dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8d29d7-548dddc64b-5tr2f\" (UID: \"891107e4-feb4-41fe-b1ef-8ac6721198dc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8d29d7-548dddc64b-5tr2f"
Sep 14 03:22:01.290413 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:22:01.290376    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c74e592f-9364-4a6e-a1a2-9ea5fa74ebf3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8d29d7-5896fdfbdc-v2trc\" (UID: \"c74e592f-9364-4a6e-a1a2-9ea5fa74ebf3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8d29d7-5896fdfbdc-v2trc"
Sep 14 03:22:01.290608 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:22:01.290476    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnqsq\" (UniqueName: \"kubernetes.io/projected/891107e4-feb4-41fe-b1ef-8ac6721198dc-kube-api-access-hnqsq\") pod \"payload-pre-processing-e2e-ipp-b-8d29d7-548dddc64b-5tr2f\" (UID: \"891107e4-feb4-41fe-b1ef-8ac6721198dc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8d29d7-548dddc64b-5tr2f"
Sep 14 03:22:01.290608 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:22:01.290530    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f242l\" (UniqueName: \"kubernetes.io/projected/c74e592f-9364-4a6e-a1a2-9ea5fa74ebf3-kube-api-access-f242l\") pod \"payload-processing-e2e-ipp-b-8d29d7-5896fdfbdc-v2trc\" (UID: \"c74e592f-9364-4a6e-a1a2-9ea5fa74ebf3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8d29d7-5896fdfbdc-v2trc"
Sep 14 03:22:01.290608 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:22:01.290547    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/891107e4-feb4-41fe-b1ef-8ac6721198dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8d29d7-548dddc64b-5tr2f\" (UID: \"891107e4-feb4-41fe-b1ef-8ac6721198dc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8d29d7-548dddc64b-5tr2f"
Sep 14 03:22:01.291155 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:22:01.291138    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/891107e4-feb4-41fe-b1ef-8ac6721198dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8d29d7-548dddc64b-5tr2f\" (UID: \"891107e4-feb4-41fe-b1ef-8ac6721198dc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8d29d7-548dddc64b-5tr2f"
Sep 14 03:22:01.299765 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:22:01.299733    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnqsq\" (UniqueName: \"kubernetes.io/projected/891107e4-feb4-41fe-b1ef-8ac6721198dc-kube-api-access-hnqsq\") pod \"payload-pre-processing-e2e-ipp-b-8d29d7-548dddc64b-5tr2f\" (UID: \"891107e4-feb4-41fe-b1ef-8ac6721198dc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8d29d7-548dddc64b-5tr2f"
Sep 14 03:22:01.391333 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:22:01.391300    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c74e592f-9364-4a6e-a1a2-9ea5fa74ebf3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8d29d7-5896fdfbdc-v2trc\" (UID: \"c74e592f-9364-4a6e-a1a2-9ea5fa74ebf3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8d29d7-5896fdfbdc-v2trc"
Sep 14 03:22:01.391522 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:22:01.391454    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f242l\" (UniqueName: \"kubernetes.io/projected/c74e592f-9364-4a6e-a1a2-9ea5fa74ebf3-kube-api-access-f242l\") pod \"payload-processing-e2e-ipp-b-8d29d7-5896fdfbdc-v2trc\" (UID: \"c74e592f-9364-4a6e-a1a2-9ea5fa74ebf3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8d29d7-5896fdfbdc-v2trc"
Sep 14 03:22:01.392122 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:22:01.392099    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c74e592f-9364-4a6e-a1a2-9ea5fa74ebf3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8d29d7-5896fdfbdc-v2trc\" (UID: \"c74e592f-9364-4a6e-a1a2-9ea5fa74ebf3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8d29d7-5896fdfbdc-v2trc"
Sep 14 03:22:01.400524 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:22:01.400504    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f242l\" (UniqueName: \"kubernetes.io/projected/c74e592f-9364-4a6e-a1a2-9ea5fa74ebf3-kube-api-access-f242l\") pod \"payload-processing-e2e-ipp-b-8d29d7-5896fdfbdc-v2trc\" (UID: \"c74e592f-9364-4a6e-a1a2-9ea5fa74ebf3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8d29d7-5896fdfbdc-v2trc"
Sep 14 03:22:01.455192 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:22:01.455160    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8d29d7-548dddc64b-5tr2f"
Sep 14 03:22:01.462484 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:22:01.462461    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8d29d7-5896fdfbdc-v2trc"
Sep 14 03:22:01.736603 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:22:01.736577    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8d29d7-548dddc64b-5tr2f"]
Sep 14 03:22:01.738094 ip-10-0-141-4 kubenswrapper[2564]: W0914 03:22:01.738064    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod891107e4_feb4_41fe_b1ef_8ac6721198dc.slice/crio-e730d41701196fa64e0900b3525ee987f35536a61e08261de2749e3ffe79b306 WatchSource:0}: Error finding container e730d41701196fa64e0900b3525ee987f35536a61e08261de2749e3ffe79b306: Status 404 returned error can't find the container with id e730d41701196fa64e0900b3525ee987f35536a61e08261de2749e3ffe79b306
Sep 14 03:22:01.750622 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:22:01.750597    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8d29d7-5896fdfbdc-v2trc"]
Sep 14 03:22:01.752752 ip-10-0-141-4 kubenswrapper[2564]: W0914 03:22:01.752726    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc74e592f_9364_4a6e_a1a2_9ea5fa74ebf3.slice/crio-e63ac4455d034cc101188cac0de36699ec3036f33e46081fe53286f565c6de63 WatchSource:0}: Error finding container e63ac4455d034cc101188cac0de36699ec3036f33e46081fe53286f565c6de63: Status 404 returned error can't find the container with id e63ac4455d034cc101188cac0de36699ec3036f33e46081fe53286f565c6de63
Sep 14 03:22:02.489341 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:22:02.489303    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8d29d7-5896fdfbdc-v2trc" event={"ID":"c74e592f-9364-4a6e-a1a2-9ea5fa74ebf3","Type":"ContainerStarted","Data":"d23f836e3aeb0d1b1a1d615cbcdc284a4712a68041183732b5feba8ec6e22455"}
Sep 14 03:22:02.489565 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:22:02.489360    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8d29d7-5896fdfbdc-v2trc" event={"ID":"c74e592f-9364-4a6e-a1a2-9ea5fa74ebf3","Type":"ContainerStarted","Data":"e63ac4455d034cc101188cac0de36699ec3036f33e46081fe53286f565c6de63"}
Sep 14 03:22:02.489565 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:22:02.489537    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8d29d7-5896fdfbdc-v2trc"
Sep 14 03:22:02.491159 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:22:02.491126    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8d29d7-548dddc64b-5tr2f" event={"ID":"891107e4-feb4-41fe-b1ef-8ac6721198dc","Type":"ContainerStarted","Data":"859e57b05a48f23a55f510a9aed3f5cb45b1d221914f6900810d7f60b71db47f"}
Sep 14 03:22:02.491286 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:22:02.491172    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8d29d7-548dddc64b-5tr2f" event={"ID":"891107e4-feb4-41fe-b1ef-8ac6721198dc","Type":"ContainerStarted","Data":"e730d41701196fa64e0900b3525ee987f35536a61e08261de2749e3ffe79b306"}
Sep 14 03:22:02.491286 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:22:02.491196    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8d29d7-548dddc64b-5tr2f"
Sep 14 03:22:02.552389 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:22:02.552332    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-8d29d7-5896fdfbdc-v2trc" podStartSLOduration=1.552313716 podStartE2EDuration="1.552313716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:22:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:22:02.550853476 +0000 UTC m=+1767.931937674" watchObservedRunningTime="2026-09-14 03:22:02.552313716 +0000 UTC m=+1767.933397917"
Sep 14 03:22:02.601680 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:22:02.601613    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8d29d7-548dddc64b-5tr2f" podStartSLOduration=1.601590486 podStartE2EDuration="1.601590486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:22:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:22:02.591997126 +0000 UTC m=+1767.973081339" watchObservedRunningTime="2026-09-14 03:22:02.601590486 +0000 UTC m=+1767.982674687"
Sep 14 03:22:13.499578 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:22:13.499499    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8d29d7-5896fdfbdc-v2trc"
Sep 14 03:22:13.499578 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:22:13.499569    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8d29d7-548dddc64b-5tr2f"
Sep 14 03:22:22.479395 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:22:22.479352    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:22:22.811173 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:22:22.811134    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:22:23.456725 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:22:23.456688    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:22:24.174507 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:22:24.174469    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:22:24.849720 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:22:24.849683    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:22:25.656251 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:22:25.656216    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:22:27.927684 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:22:27.927601    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-265v7_c6f51115-9cf5-49ab-97b5-d0a840313cf4/payload-processing/0.log"
Sep 14 03:22:28.061801 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:22:28.061761    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-8d29d7-5896fdfbdc-v2trc_c74e592f-9364-4a6e-a1a2-9ea5fa74ebf3/payload-processing/0.log"
Sep 14 03:22:31.271426 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:22:31.271361    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:22:43.541143 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:22:43.541104    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:22:44.525250 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:22:44.525203    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:22:44.626244 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:22:44.626210    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:22:53.944559 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:22:53.944516    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-265v7_c6f51115-9cf5-49ab-97b5-d0a840313cf4/payload-processing/0.log"
Sep 14 03:22:54.071126 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:22:54.071072    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-8d29d7-5896fdfbdc-v2trc_c74e592f-9364-4a6e-a1a2-9ea5fa74ebf3/payload-processing/0.log"
Sep 14 03:22:54.325192 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:22:54.325152    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:22:54.425030 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:22:54.424988    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:22:54.746194 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:22:54.746110    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:22:54.847701 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:22:54.847662    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:22:58.064122 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:22:58.064082    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:22:58.333061 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:22:58.332960    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:22:58.421760 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:22:58.421722    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:22:58.767262 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:22:58.767168    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:22:58.947060 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:22:58.947013    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:22:59.796040 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:22:59.795992    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:23:00.516102 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:00.516068    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:23:01.339026 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:01.338972    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:23:01.437757 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:01.437725    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:23:01.559186 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:01.559143    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:23:01.817257 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:01.817214    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:23:01.955701 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:01.955668    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:23:02.071969 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:02.071880    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3b049b-fb696d594-q5x8m"]
Sep 14 03:23:02.075775 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:02.075752    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3b049b-fb696d594-q5x8m"
Sep 14 03:23:02.078685 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:02.078663    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-3b049b-dockercfg-9zmb2\""
Sep 14 03:23:02.078809 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:02.078703    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-3b049b\""
Sep 14 03:23:02.088216 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:02.088187    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3b049b-fb696d594-q5x8m"]
Sep 14 03:23:02.108010 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:02.107969    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cp4g\" (UniqueName: \"kubernetes.io/projected/ce8f6872-4aa2-4405-88ed-d7746c8c4ede-kube-api-access-9cp4g\") pod \"payload-pre-processing-e2e-ipp-cleanup-3b049b-fb696d594-q5x8m\" (UID: \"ce8f6872-4aa2-4405-88ed-d7746c8c4ede\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3b049b-fb696d594-q5x8m"
Sep 14 03:23:02.108192 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:02.108022    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce8f6872-4aa2-4405-88ed-d7746c8c4ede-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-3b049b-fb696d594-q5x8m\" (UID: \"ce8f6872-4aa2-4405-88ed-d7746c8c4ede\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3b049b-fb696d594-q5x8m"
Sep 14 03:23:02.129016 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:02.128977    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-3b049b-64f997666c-h22nf"]
Sep 14 03:23:02.132761 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:02.132733    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3b049b-64f997666c-h22nf"
Sep 14 03:23:02.152090 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:02.152059    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-3b049b-64f997666c-h22nf"]
Sep 14 03:23:02.209167 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:02.209135    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cp4g\" (UniqueName: \"kubernetes.io/projected/ce8f6872-4aa2-4405-88ed-d7746c8c4ede-kube-api-access-9cp4g\") pod \"payload-pre-processing-e2e-ipp-cleanup-3b049b-fb696d594-q5x8m\" (UID: \"ce8f6872-4aa2-4405-88ed-d7746c8c4ede\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3b049b-fb696d594-q5x8m"
Sep 14 03:23:02.209340 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:02.209177    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce8f6872-4aa2-4405-88ed-d7746c8c4ede-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-3b049b-fb696d594-q5x8m\" (UID: \"ce8f6872-4aa2-4405-88ed-d7746c8c4ede\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3b049b-fb696d594-q5x8m"
Sep 14 03:23:02.209340 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:02.209234    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvmmh\" (UniqueName: \"kubernetes.io/projected/54ab7623-06c6-471e-a412-2c3538877408-kube-api-access-tvmmh\") pod \"payload-processing-e2e-ipp-cleanup-3b049b-64f997666c-h22nf\" (UID: \"54ab7623-06c6-471e-a412-2c3538877408\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3b049b-64f997666c-h22nf"
Sep 14 03:23:02.209462 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:02.209436    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54ab7623-06c6-471e-a412-2c3538877408-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-3b049b-64f997666c-h22nf\" (UID: \"54ab7623-06c6-471e-a412-2c3538877408\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3b049b-64f997666c-h22nf"
Sep 14 03:23:02.209945 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:02.209927    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce8f6872-4aa2-4405-88ed-d7746c8c4ede-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-3b049b-fb696d594-q5x8m\" (UID: \"ce8f6872-4aa2-4405-88ed-d7746c8c4ede\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3b049b-fb696d594-q5x8m"
Sep 14 03:23:02.220079 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:02.220051    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cp4g\" (UniqueName: \"kubernetes.io/projected/ce8f6872-4aa2-4405-88ed-d7746c8c4ede-kube-api-access-9cp4g\") pod \"payload-pre-processing-e2e-ipp-cleanup-3b049b-fb696d594-q5x8m\" (UID: \"ce8f6872-4aa2-4405-88ed-d7746c8c4ede\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3b049b-fb696d594-q5x8m"
Sep 14 03:23:02.310438 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:02.310377    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvmmh\" (UniqueName: \"kubernetes.io/projected/54ab7623-06c6-471e-a412-2c3538877408-kube-api-access-tvmmh\") pod \"payload-processing-e2e-ipp-cleanup-3b049b-64f997666c-h22nf\" (UID: \"54ab7623-06c6-471e-a412-2c3538877408\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3b049b-64f997666c-h22nf"
Sep 14 03:23:02.310641 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:02.310540    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54ab7623-06c6-471e-a412-2c3538877408-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-3b049b-64f997666c-h22nf\" (UID: \"54ab7623-06c6-471e-a412-2c3538877408\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3b049b-64f997666c-h22nf"
Sep 14 03:23:02.311246 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:02.311223    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54ab7623-06c6-471e-a412-2c3538877408-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-3b049b-64f997666c-h22nf\" (UID: \"54ab7623-06c6-471e-a412-2c3538877408\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3b049b-64f997666c-h22nf"
Sep 14 03:23:02.320378 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:02.320353    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvmmh\" (UniqueName: \"kubernetes.io/projected/54ab7623-06c6-471e-a412-2c3538877408-kube-api-access-tvmmh\") pod \"payload-processing-e2e-ipp-cleanup-3b049b-64f997666c-h22nf\" (UID: \"54ab7623-06c6-471e-a412-2c3538877408\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3b049b-64f997666c-h22nf"
Sep 14 03:23:02.386198 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:02.386088    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3b049b-fb696d594-q5x8m"
Sep 14 03:23:02.444138 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:02.443865    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3b049b-64f997666c-h22nf"
Sep 14 03:23:02.665575 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:02.665490    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3b049b-fb696d594-q5x8m"]
Sep 14 03:23:02.668643 ip-10-0-141-4 kubenswrapper[2564]: W0914 03:23:02.668615    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce8f6872_4aa2_4405_88ed_d7746c8c4ede.slice/crio-664d82787803cc28d67d9b7bc529486507cd4dc2a69ab4fb712c25c2bc48a4a3 WatchSource:0}: Error finding container 664d82787803cc28d67d9b7bc529486507cd4dc2a69ab4fb712c25c2bc48a4a3: Status 404 returned error can't find the container with id 664d82787803cc28d67d9b7bc529486507cd4dc2a69ab4fb712c25c2bc48a4a3
Sep 14 03:23:02.730142 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:02.730115    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-3b049b-64f997666c-h22nf"]
Sep 14 03:23:02.734757 ip-10-0-141-4 kubenswrapper[2564]: W0914 03:23:02.734729    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54ab7623_06c6_471e_a412_2c3538877408.slice/crio-575f4ca12b5c4c016024b46d2c98368d0dc43ddda4312c461cc877e7c4d86559 WatchSource:0}: Error finding container 575f4ca12b5c4c016024b46d2c98368d0dc43ddda4312c461cc877e7c4d86559: Status 404 returned error can't find the container with id 575f4ca12b5c4c016024b46d2c98368d0dc43ddda4312c461cc877e7c4d86559
Sep 14 03:23:02.812430 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:02.812342    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3b049b-fb696d594-q5x8m" event={"ID":"ce8f6872-4aa2-4405-88ed-d7746c8c4ede","Type":"ContainerStarted","Data":"ac2cd56f903032078ed75d5b3f7604576ea7a125d6bdcf237afa3b428f710514"}
Sep 14 03:23:02.812613 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:02.812442    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3b049b-fb696d594-q5x8m" event={"ID":"ce8f6872-4aa2-4405-88ed-d7746c8c4ede","Type":"ContainerStarted","Data":"664d82787803cc28d67d9b7bc529486507cd4dc2a69ab4fb712c25c2bc48a4a3"}
Sep 14 03:23:02.812613 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:02.812476    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3b049b-fb696d594-q5x8m"
Sep 14 03:23:02.814246 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:02.814219    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3b049b-64f997666c-h22nf" event={"ID":"54ab7623-06c6-471e-a412-2c3538877408","Type":"ContainerStarted","Data":"9868e218e9b0b00ff681dc77d8aa47ce1f2c4c3333ca7c9fa8797a1aa1c08acf"}
Sep 14 03:23:02.814390 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:02.814264    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3b049b-64f997666c-h22nf" event={"ID":"54ab7623-06c6-471e-a412-2c3538877408","Type":"ContainerStarted","Data":"575f4ca12b5c4c016024b46d2c98368d0dc43ddda4312c461cc877e7c4d86559"}
Sep 14 03:23:02.814390 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:02.814289    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3b049b-64f997666c-h22nf"
Sep 14 03:23:02.835013 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:02.834957    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3b049b-fb696d594-q5x8m" podStartSLOduration=0.834937504 podStartE2EDuration="834.937504ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:23:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:23:02.833067024 +0000 UTC m=+1828.214151224" watchObservedRunningTime="2026-09-14 03:23:02.834937504 +0000 UTC m=+1828.216021702"
Sep 14 03:23:02.853258 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:02.853193    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3b049b-64f997666c-h22nf" podStartSLOduration=0.853170563 podStartE2EDuration="853.170563ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:23:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:23:02.852093931 +0000 UTC m=+1828.233178130" watchObservedRunningTime="2026-09-14 03:23:02.853170563 +0000 UTC m=+1828.234254764"
Sep 14 03:23:13.821598 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:13.821560    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3b049b-64f997666c-h22nf"
Sep 14 03:23:13.821994 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:13.821740    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3b049b-fb696d594-q5x8m"
Sep 14 03:23:23.743895 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:23.743862    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:23:24.445245 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:24.445207    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:23:25.235739 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:25.235695    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:23:26.140311 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:26.140271    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:23:26.940654 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:26.940612    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:23:27.648313 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:27.648278    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:23:28.505924 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:28.505879    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:23:28.920823 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:28.920789    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:23:29.846628 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:29.846573    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:23:30.515530 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:30.515490    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:23:31.333938 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:31.333892    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:23:32.039059 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:32.039002    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:23:32.837944 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:32.837905    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:23:33.605742 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:33.605708    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:23:39.920046 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:39.919997    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:23:40.530467 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:40.530435    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-3b049b-64f997666c-h22nf"]
Sep 14 03:23:40.530704 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:40.530678    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3b049b-64f997666c-h22nf" podUID="54ab7623-06c6-471e-a412-2c3538877408" containerName="payload-processing" containerID="cri-o://9868e218e9b0b00ff681dc77d8aa47ce1f2c4c3333ca7c9fa8797a1aa1c08acf" gracePeriod=30
Sep 14 03:23:40.533336 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:40.533306    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:23:40.540550 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:40.540523    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3b049b-fb696d594-q5x8m"]
Sep 14 03:23:40.540994 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:40.540756    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3b049b-fb696d594-q5x8m" podUID="ce8f6872-4aa2-4405-88ed-d7746c8c4ede" containerName="payload-pre-processing" containerID="cri-o://ac2cd56f903032078ed75d5b3f7604576ea7a125d6bdcf237afa3b428f710514" gracePeriod=30
Sep 14 03:23:40.618316 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:40.618279    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:23:40.725025 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:40.724994    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:23:40.817582 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:40.817548    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3b049b-64f997666c-h22nf"
Sep 14 03:23:40.820639 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:40.820598    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:23:40.821607 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:40.821589    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3b049b-fb696d594-q5x8m"
Sep 14 03:23:40.910386 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:40.910349    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54ab7623-06c6-471e-a412-2c3538877408-plugins-config-volume\") pod \"54ab7623-06c6-471e-a412-2c3538877408\" (UID: \"54ab7623-06c6-471e-a412-2c3538877408\") "
Sep 14 03:23:40.910565 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:40.910399    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tvmmh\" (UniqueName: \"kubernetes.io/projected/54ab7623-06c6-471e-a412-2c3538877408-kube-api-access-tvmmh\") pod \"54ab7623-06c6-471e-a412-2c3538877408\" (UID: \"54ab7623-06c6-471e-a412-2c3538877408\") "
Sep 14 03:23:40.910565 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:40.910473    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9cp4g\" (UniqueName: \"kubernetes.io/projected/ce8f6872-4aa2-4405-88ed-d7746c8c4ede-kube-api-access-9cp4g\") pod \"ce8f6872-4aa2-4405-88ed-d7746c8c4ede\" (UID: \"ce8f6872-4aa2-4405-88ed-d7746c8c4ede\") "
Sep 14 03:23:40.910565 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:40.910497    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce8f6872-4aa2-4405-88ed-d7746c8c4ede-plugins-config-volume\") pod \"ce8f6872-4aa2-4405-88ed-d7746c8c4ede\" (UID: \"ce8f6872-4aa2-4405-88ed-d7746c8c4ede\") "
Sep 14 03:23:40.910881 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:40.910853    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54ab7623-06c6-471e-a412-2c3538877408-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "54ab7623-06c6-471e-a412-2c3538877408" (UID: "54ab7623-06c6-471e-a412-2c3538877408"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 03:23:40.910949 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:40.910895    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce8f6872-4aa2-4405-88ed-d7746c8c4ede-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ce8f6872-4aa2-4405-88ed-d7746c8c4ede" (UID: "ce8f6872-4aa2-4405-88ed-d7746c8c4ede"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 03:23:40.912627 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:40.912602    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce8f6872-4aa2-4405-88ed-d7746c8c4ede-kube-api-access-9cp4g" (OuterVolumeSpecName: "kube-api-access-9cp4g") pod "ce8f6872-4aa2-4405-88ed-d7746c8c4ede" (UID: "ce8f6872-4aa2-4405-88ed-d7746c8c4ede"). InnerVolumeSpecName "kube-api-access-9cp4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:23:40.912627 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:40.912614    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54ab7623-06c6-471e-a412-2c3538877408-kube-api-access-tvmmh" (OuterVolumeSpecName: "kube-api-access-tvmmh") pod "54ab7623-06c6-471e-a412-2c3538877408" (UID: "54ab7623-06c6-471e-a412-2c3538877408"). InnerVolumeSpecName "kube-api-access-tvmmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:23:41.012428 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:41.012361    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvmmh\" (UniqueName: \"kubernetes.io/projected/54ab7623-06c6-471e-a412-2c3538877408-kube-api-access-tvmmh\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:23:41.012428 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:41.012395    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cp4g\" (UniqueName: \"kubernetes.io/projected/ce8f6872-4aa2-4405-88ed-d7746c8c4ede-kube-api-access-9cp4g\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:23:41.012428 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:41.012433    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce8f6872-4aa2-4405-88ed-d7746c8c4ede-plugins-config-volume\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:23:41.012877 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:41.012446    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54ab7623-06c6-471e-a412-2c3538877408-plugins-config-volume\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:23:41.019006 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:41.018979    2564 generic.go:356] "Generic (PLEG): container finished" podID="54ab7623-06c6-471e-a412-2c3538877408" containerID="9868e218e9b0b00ff681dc77d8aa47ce1f2c4c3333ca7c9fa8797a1aa1c08acf" exitCode=0
Sep 14 03:23:41.019163 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:41.019025    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3b049b-64f997666c-h22nf"
Sep 14 03:23:41.019163 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:41.019070    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3b049b-64f997666c-h22nf" event={"ID":"54ab7623-06c6-471e-a412-2c3538877408","Type":"ContainerDied","Data":"9868e218e9b0b00ff681dc77d8aa47ce1f2c4c3333ca7c9fa8797a1aa1c08acf"}
Sep 14 03:23:41.019163 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:41.019120    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3b049b-64f997666c-h22nf" event={"ID":"54ab7623-06c6-471e-a412-2c3538877408","Type":"ContainerDied","Data":"575f4ca12b5c4c016024b46d2c98368d0dc43ddda4312c461cc877e7c4d86559"}
Sep 14 03:23:41.019320 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:41.019148    2564 scope.go:117] "RemoveContainer" containerID="9868e218e9b0b00ff681dc77d8aa47ce1f2c4c3333ca7c9fa8797a1aa1c08acf"
Sep 14 03:23:41.020508 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:41.020485    2564 generic.go:356] "Generic (PLEG): container finished" podID="ce8f6872-4aa2-4405-88ed-d7746c8c4ede" containerID="ac2cd56f903032078ed75d5b3f7604576ea7a125d6bdcf237afa3b428f710514" exitCode=0
Sep 14 03:23:41.020608 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:41.020527    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3b049b-fb696d594-q5x8m"
Sep 14 03:23:41.020608 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:41.020557    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3b049b-fb696d594-q5x8m" event={"ID":"ce8f6872-4aa2-4405-88ed-d7746c8c4ede","Type":"ContainerDied","Data":"ac2cd56f903032078ed75d5b3f7604576ea7a125d6bdcf237afa3b428f710514"}
Sep 14 03:23:41.020695 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:41.020609    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3b049b-fb696d594-q5x8m" event={"ID":"ce8f6872-4aa2-4405-88ed-d7746c8c4ede","Type":"ContainerDied","Data":"664d82787803cc28d67d9b7bc529486507cd4dc2a69ab4fb712c25c2bc48a4a3"}
Sep 14 03:23:41.031970 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:41.031950    2564 scope.go:117] "RemoveContainer" containerID="9868e218e9b0b00ff681dc77d8aa47ce1f2c4c3333ca7c9fa8797a1aa1c08acf"
Sep 14 03:23:41.032382 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:23:41.032360    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9868e218e9b0b00ff681dc77d8aa47ce1f2c4c3333ca7c9fa8797a1aa1c08acf\": container with ID starting with 9868e218e9b0b00ff681dc77d8aa47ce1f2c4c3333ca7c9fa8797a1aa1c08acf not found: ID does not exist" containerID="9868e218e9b0b00ff681dc77d8aa47ce1f2c4c3333ca7c9fa8797a1aa1c08acf"
Sep 14 03:23:41.032753 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:41.032393    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9868e218e9b0b00ff681dc77d8aa47ce1f2c4c3333ca7c9fa8797a1aa1c08acf"} err="failed to get container status \"9868e218e9b0b00ff681dc77d8aa47ce1f2c4c3333ca7c9fa8797a1aa1c08acf\": rpc error: code = NotFound desc = could not find container \"9868e218e9b0b00ff681dc77d8aa47ce1f2c4c3333ca7c9fa8797a1aa1c08acf\": container with ID starting with 9868e218e9b0b00ff681dc77d8aa47ce1f2c4c3333ca7c9fa8797a1aa1c08acf not found: ID does not exist"
Sep 14 03:23:41.032809 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:41.032763    2564 scope.go:117] "RemoveContainer" containerID="ac2cd56f903032078ed75d5b3f7604576ea7a125d6bdcf237afa3b428f710514"
Sep 14 03:23:41.041380 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:41.041359    2564 scope.go:117] "RemoveContainer" containerID="ac2cd56f903032078ed75d5b3f7604576ea7a125d6bdcf237afa3b428f710514"
Sep 14 03:23:41.041662 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:23:41.041646    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac2cd56f903032078ed75d5b3f7604576ea7a125d6bdcf237afa3b428f710514\": container with ID starting with ac2cd56f903032078ed75d5b3f7604576ea7a125d6bdcf237afa3b428f710514 not found: ID does not exist" containerID="ac2cd56f903032078ed75d5b3f7604576ea7a125d6bdcf237afa3b428f710514"
Sep 14 03:23:41.041712 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:41.041668    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac2cd56f903032078ed75d5b3f7604576ea7a125d6bdcf237afa3b428f710514"} err="failed to get container status \"ac2cd56f903032078ed75d5b3f7604576ea7a125d6bdcf237afa3b428f710514\": rpc error: code = NotFound desc = could not find container \"ac2cd56f903032078ed75d5b3f7604576ea7a125d6bdcf237afa3b428f710514\": container with ID starting with ac2cd56f903032078ed75d5b3f7604576ea7a125d6bdcf237afa3b428f710514 not found: ID does not exist"
Sep 14 03:23:41.047716 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:41.047696    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-3b049b-64f997666c-h22nf"]
Sep 14 03:23:41.052619 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:41.052597    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-3b049b-64f997666c-h22nf"]
Sep 14 03:23:41.066747 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:41.066720    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3b049b-fb696d594-q5x8m"]
Sep 14 03:23:41.069861 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:41.069841    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3b049b-fb696d594-q5x8m"]
Sep 14 03:23:41.099827 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:41.099803    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54ab7623-06c6-471e-a412-2c3538877408" path="/var/lib/kubelet/pods/54ab7623-06c6-471e-a412-2c3538877408/volumes"
Sep 14 03:23:41.100237 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:41.100222    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce8f6872-4aa2-4405-88ed-d7746c8c4ede" path="/var/lib/kubelet/pods/ce8f6872-4aa2-4405-88ed-d7746c8c4ede/volumes"
Sep 14 03:23:41.129008 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:41.128975    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:23:41.229906 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:41.229868    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:23:41.891234 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:41.891143    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:23:42.669610 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:42.669577    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:23:43.579618 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:43.579579    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:23:44.384712 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:44.384679    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:23:47.390175 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:47.390137    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:23:47.490454 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:47.490396    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:23:47.702513 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:47.702413    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:23:51.826025 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:51.825987    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:23:52.701277 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:52.701240    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:23:53.462331 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:53.462292    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:23:54.307989 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:54.307951    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:23:55.046389 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:55.046353    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:23:55.811787 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:23:55.811752    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:24:07.100664 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:24:07.100560    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d081a3-6dbff6ccd-jfqdr"]
Sep 14 03:24:07.101196 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:24:07.100864    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d081a3-6dbff6ccd-jfqdr" podUID="45433dea-3cc8-4746-9090-83be7ade5bd3" containerName="maas-api" containerID="cri-o://c75dde86a033733b359918ae22a814a96e4cb5c82c0514acf5ba5e4853693658" gracePeriod=30
Sep 14 03:24:07.128849 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:24:07.128815    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:24:07.252972 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:24:07.252933    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:24:07.353321 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:24:07.353255    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d081a3-6dbff6ccd-jfqdr"
Sep 14 03:24:07.370497 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:24:07.370469    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45433dea-3cc8-4746-9090-83be7ade5bd3-openshift-service-ca\") pod \"45433dea-3cc8-4746-9090-83be7ade5bd3\" (UID: \"45433dea-3cc8-4746-9090-83be7ade5bd3\") "
Sep 14 03:24:07.370648 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:24:07.370514    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45433dea-3cc8-4746-9090-83be7ade5bd3-maas-api-tls\") pod \"45433dea-3cc8-4746-9090-83be7ade5bd3\" (UID: \"45433dea-3cc8-4746-9090-83be7ade5bd3\") "
Sep 14 03:24:07.370648 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:24:07.370576    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45433dea-3cc8-4746-9090-83be7ade5bd3-metrics-certs\") pod \"45433dea-3cc8-4746-9090-83be7ade5bd3\" (UID: \"45433dea-3cc8-4746-9090-83be7ade5bd3\") "
Sep 14 03:24:07.370648 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:24:07.370609    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hz7jb\" (UniqueName: \"kubernetes.io/projected/45433dea-3cc8-4746-9090-83be7ade5bd3-kube-api-access-hz7jb\") pod \"45433dea-3cc8-4746-9090-83be7ade5bd3\" (UID: \"45433dea-3cc8-4746-9090-83be7ade5bd3\") "
Sep 14 03:24:07.370890 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:24:07.370869    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45433dea-3cc8-4746-9090-83be7ade5bd3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "45433dea-3cc8-4746-9090-83be7ade5bd3" (UID: "45433dea-3cc8-4746-9090-83be7ade5bd3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 03:24:07.370966 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:24:07.370951    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45433dea-3cc8-4746-9090-83be7ade5bd3-openshift-service-ca\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:24:07.372822 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:24:07.372795    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45433dea-3cc8-4746-9090-83be7ade5bd3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "45433dea-3cc8-4746-9090-83be7ade5bd3" (UID: "45433dea-3cc8-4746-9090-83be7ade5bd3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 03:24:07.373246 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:24:07.373222    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45433dea-3cc8-4746-9090-83be7ade5bd3-kube-api-access-hz7jb" (OuterVolumeSpecName: "kube-api-access-hz7jb") pod "45433dea-3cc8-4746-9090-83be7ade5bd3" (UID: "45433dea-3cc8-4746-9090-83be7ade5bd3"). InnerVolumeSpecName "kube-api-access-hz7jb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:24:07.373376 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:24:07.373248    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45433dea-3cc8-4746-9090-83be7ade5bd3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "45433dea-3cc8-4746-9090-83be7ade5bd3" (UID: "45433dea-3cc8-4746-9090-83be7ade5bd3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 03:24:07.415348 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:24:07.415307    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:24:07.471770 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:24:07.471734    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45433dea-3cc8-4746-9090-83be7ade5bd3-metrics-certs\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:24:07.471770 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:24:07.471763    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hz7jb\" (UniqueName: \"kubernetes.io/projected/45433dea-3cc8-4746-9090-83be7ade5bd3-kube-api-access-hz7jb\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:24:07.471770 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:24:07.471773    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45433dea-3cc8-4746-9090-83be7ade5bd3-maas-api-tls\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:24:07.496910 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:24:07.496860    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:24:07.596936 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:24:07.596899    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:24:07.848108 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:24:07.848070    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:24:07.949540 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:24:07.949496    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:24:08.171925 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:24:08.171845    2564 generic.go:356] "Generic (PLEG): container finished" podID="45433dea-3cc8-4746-9090-83be7ade5bd3" containerID="c75dde86a033733b359918ae22a814a96e4cb5c82c0514acf5ba5e4853693658" exitCode=0
Sep 14 03:24:08.171925 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:24:08.171891    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d081a3-6dbff6ccd-jfqdr"
Sep 14 03:24:08.171925 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:24:08.171896    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d081a3-6dbff6ccd-jfqdr" event={"ID":"45433dea-3cc8-4746-9090-83be7ade5bd3","Type":"ContainerDied","Data":"c75dde86a033733b359918ae22a814a96e4cb5c82c0514acf5ba5e4853693658"}
Sep 14 03:24:08.172376 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:24:08.171948    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d081a3-6dbff6ccd-jfqdr" event={"ID":"45433dea-3cc8-4746-9090-83be7ade5bd3","Type":"ContainerDied","Data":"3bf2d73084ea6d8de9e57b4681a7c8962cea90ad4802bb24acea2587386a14a1"}
Sep 14 03:24:08.172376 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:24:08.171980    2564 scope.go:117] "RemoveContainer" containerID="c75dde86a033733b359918ae22a814a96e4cb5c82c0514acf5ba5e4853693658"
Sep 14 03:24:08.181922 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:24:08.181905    2564 scope.go:117] "RemoveContainer" containerID="c75dde86a033733b359918ae22a814a96e4cb5c82c0514acf5ba5e4853693658"
Sep 14 03:24:08.182181 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:24:08.182165    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c75dde86a033733b359918ae22a814a96e4cb5c82c0514acf5ba5e4853693658\": container with ID starting with c75dde86a033733b359918ae22a814a96e4cb5c82c0514acf5ba5e4853693658 not found: ID does not exist" containerID="c75dde86a033733b359918ae22a814a96e4cb5c82c0514acf5ba5e4853693658"
Sep 14 03:24:08.182225 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:24:08.182190    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c75dde86a033733b359918ae22a814a96e4cb5c82c0514acf5ba5e4853693658"} err="failed to get container status \"c75dde86a033733b359918ae22a814a96e4cb5c82c0514acf5ba5e4853693658\": rpc error: code = NotFound desc = could not find container \"c75dde86a033733b359918ae22a814a96e4cb5c82c0514acf5ba5e4853693658\": container with ID starting with c75dde86a033733b359918ae22a814a96e4cb5c82c0514acf5ba5e4853693658 not found: ID does not exist"
Sep 14 03:24:08.198074 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:24:08.198044    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d081a3-6dbff6ccd-jfqdr"]
Sep 14 03:24:08.199963 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:24:08.199944    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d081a3-6dbff6ccd-jfqdr"]
Sep 14 03:24:08.846297 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:24:08.846256    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:24:09.098273 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:24:09.098199    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45433dea-3cc8-4746-9090-83be7ade5bd3" path="/var/lib/kubelet/pods/45433dea-3cc8-4746-9090-83be7ade5bd3/volumes"
Sep 14 03:24:09.666457 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:24:09.666400    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:24:10.460091 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:24:10.460051    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:24:12.073045 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:24:12.073005    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:24:12.174609 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:24:12.174565    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:24:12.973920 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:24:12.973874    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:24:49.739669 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:24:49.739631    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:24:50.172580 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:24:50.172544    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:24:55.965198 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:24:55.965155    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:24:56.066667 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:24:56.066626    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:24:56.281382 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:24:56.281347    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:24:56.754896 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:24:56.754753    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:24:57.636051 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:24:57.635968    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:24:58.440068 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:24:58.440029    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:24:59.402817 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:24:59.402779    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:25:00.293850 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:00.293818    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:25:03.713248 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:03.713199    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:25:03.813262 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:03.813226    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:25:07.184048 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:07.184008    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:25:07.212859 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:07.212825    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8d29d7-5896fdfbdc-v2trc"]
Sep 14 03:25:07.213091 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:07.213066    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-8d29d7-5896fdfbdc-v2trc" podUID="c74e592f-9364-4a6e-a1a2-9ea5fa74ebf3" containerName="payload-processing" containerID="cri-o://d23f836e3aeb0d1b1a1d615cbcdc284a4712a68041183732b5feba8ec6e22455" gracePeriod=30
Sep 14 03:25:07.220201 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:07.220175    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8d29d7-548dddc64b-5tr2f"]
Sep 14 03:25:07.220605 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:07.220574    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8d29d7-548dddc64b-5tr2f" podUID="891107e4-feb4-41fe-b1ef-8ac6721198dc" containerName="payload-pre-processing" containerID="cri-o://859e57b05a48f23a55f510a9aed3f5cb45b1d221914f6900810d7f60b71db47f" gracePeriod=30
Sep 14 03:25:07.258553 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:07.258480    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:25:07.320340 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:07.320305    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:25:07.427395 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:07.427357    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:25:07.480336 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:07.480306    2564 generic.go:356] "Generic (PLEG): container finished" podID="c74e592f-9364-4a6e-a1a2-9ea5fa74ebf3" containerID="d23f836e3aeb0d1b1a1d615cbcdc284a4712a68041183732b5feba8ec6e22455" exitCode=0
Sep 14 03:25:07.480527 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:07.480435    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8d29d7-5896fdfbdc-v2trc" event={"ID":"c74e592f-9364-4a6e-a1a2-9ea5fa74ebf3","Type":"ContainerDied","Data":"d23f836e3aeb0d1b1a1d615cbcdc284a4712a68041183732b5feba8ec6e22455"}
Sep 14 03:25:07.483069 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:07.482961    2564 generic.go:356] "Generic (PLEG): container finished" podID="891107e4-feb4-41fe-b1ef-8ac6721198dc" containerID="859e57b05a48f23a55f510a9aed3f5cb45b1d221914f6900810d7f60b71db47f" exitCode=0
Sep 14 03:25:07.483069 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:07.483001    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8d29d7-548dddc64b-5tr2f" event={"ID":"891107e4-feb4-41fe-b1ef-8ac6721198dc","Type":"ContainerDied","Data":"859e57b05a48f23a55f510a9aed3f5cb45b1d221914f6900810d7f60b71db47f"}
Sep 14 03:25:07.502628 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:07.502604    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8d29d7-5896fdfbdc-v2trc"
Sep 14 03:25:07.505865 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:07.505842    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8d29d7-548dddc64b-5tr2f"
Sep 14 03:25:07.660726 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:07.660691    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c74e592f-9364-4a6e-a1a2-9ea5fa74ebf3-plugins-config-volume\") pod \"c74e592f-9364-4a6e-a1a2-9ea5fa74ebf3\" (UID: \"c74e592f-9364-4a6e-a1a2-9ea5fa74ebf3\") "
Sep 14 03:25:07.660929 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:07.660785    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hnqsq\" (UniqueName: \"kubernetes.io/projected/891107e4-feb4-41fe-b1ef-8ac6721198dc-kube-api-access-hnqsq\") pod \"891107e4-feb4-41fe-b1ef-8ac6721198dc\" (UID: \"891107e4-feb4-41fe-b1ef-8ac6721198dc\") "
Sep 14 03:25:07.660929 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:07.660902    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/891107e4-feb4-41fe-b1ef-8ac6721198dc-plugins-config-volume\") pod \"891107e4-feb4-41fe-b1ef-8ac6721198dc\" (UID: \"891107e4-feb4-41fe-b1ef-8ac6721198dc\") "
Sep 14 03:25:07.661046 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:07.661036    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f242l\" (UniqueName: \"kubernetes.io/projected/c74e592f-9364-4a6e-a1a2-9ea5fa74ebf3-kube-api-access-f242l\") pod \"c74e592f-9364-4a6e-a1a2-9ea5fa74ebf3\" (UID: \"c74e592f-9364-4a6e-a1a2-9ea5fa74ebf3\") "
Sep 14 03:25:07.661257 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:07.661216    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c74e592f-9364-4a6e-a1a2-9ea5fa74ebf3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c74e592f-9364-4a6e-a1a2-9ea5fa74ebf3" (UID: "c74e592f-9364-4a6e-a1a2-9ea5fa74ebf3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 03:25:07.661257 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:07.661247    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/891107e4-feb4-41fe-b1ef-8ac6721198dc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "891107e4-feb4-41fe-b1ef-8ac6721198dc" (UID: "891107e4-feb4-41fe-b1ef-8ac6721198dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 03:25:07.661382 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:07.661370    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c74e592f-9364-4a6e-a1a2-9ea5fa74ebf3-plugins-config-volume\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:25:07.661449 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:07.661385    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/891107e4-feb4-41fe-b1ef-8ac6721198dc-plugins-config-volume\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:25:07.663082 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:07.663061    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/891107e4-feb4-41fe-b1ef-8ac6721198dc-kube-api-access-hnqsq" (OuterVolumeSpecName: "kube-api-access-hnqsq") pod "891107e4-feb4-41fe-b1ef-8ac6721198dc" (UID: "891107e4-feb4-41fe-b1ef-8ac6721198dc"). InnerVolumeSpecName "kube-api-access-hnqsq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:25:07.663155 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:07.663133    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c74e592f-9364-4a6e-a1a2-9ea5fa74ebf3-kube-api-access-f242l" (OuterVolumeSpecName: "kube-api-access-f242l") pod "c74e592f-9364-4a6e-a1a2-9ea5fa74ebf3" (UID: "c74e592f-9364-4a6e-a1a2-9ea5fa74ebf3"). InnerVolumeSpecName "kube-api-access-f242l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:25:07.762272 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:07.762189    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnqsq\" (UniqueName: \"kubernetes.io/projected/891107e4-feb4-41fe-b1ef-8ac6721198dc-kube-api-access-hnqsq\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:25:07.762272 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:07.762216    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f242l\" (UniqueName: \"kubernetes.io/projected/c74e592f-9364-4a6e-a1a2-9ea5fa74ebf3-kube-api-access-f242l\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:25:07.789910 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:07.789872    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:25:07.886627 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:07.886591    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:25:08.491095 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:08.491055    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8d29d7-5896fdfbdc-v2trc" event={"ID":"c74e592f-9364-4a6e-a1a2-9ea5fa74ebf3","Type":"ContainerDied","Data":"e63ac4455d034cc101188cac0de36699ec3036f33e46081fe53286f565c6de63"}
Sep 14 03:25:08.491572 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:08.491070    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8d29d7-5896fdfbdc-v2trc"
Sep 14 03:25:08.491572 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:08.491129    2564 scope.go:117] "RemoveContainer" containerID="d23f836e3aeb0d1b1a1d615cbcdc284a4712a68041183732b5feba8ec6e22455"
Sep 14 03:25:08.492737 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:08.492703    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8d29d7-548dddc64b-5tr2f" event={"ID":"891107e4-feb4-41fe-b1ef-8ac6721198dc","Type":"ContainerDied","Data":"e730d41701196fa64e0900b3525ee987f35536a61e08261de2749e3ffe79b306"}
Sep 14 03:25:08.492838 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:08.492743    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8d29d7-548dddc64b-5tr2f"
Sep 14 03:25:08.502080 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:08.502060    2564 scope.go:117] "RemoveContainer" containerID="859e57b05a48f23a55f510a9aed3f5cb45b1d221914f6900810d7f60b71db47f"
Sep 14 03:25:08.514833 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:08.514809    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8d29d7-5896fdfbdc-v2trc"]
Sep 14 03:25:08.520977 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:08.520953    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8d29d7-5896fdfbdc-v2trc"]
Sep 14 03:25:08.534971 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:08.534950    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8d29d7-548dddc64b-5tr2f"]
Sep 14 03:25:08.539766 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:08.539745    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8d29d7-548dddc64b-5tr2f"]
Sep 14 03:25:08.666757 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:08.666718    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:25:09.098744 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:09.098714    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="891107e4-feb4-41fe-b1ef-8ac6721198dc" path="/var/lib/kubelet/pods/891107e4-feb4-41fe-b1ef-8ac6721198dc/volumes"
Sep 14 03:25:09.099198 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:09.099179    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c74e592f-9364-4a6e-a1a2-9ea5fa74ebf3" path="/var/lib/kubelet/pods/c74e592f-9364-4a6e-a1a2-9ea5fa74ebf3/volumes"
Sep 14 03:25:09.342286 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:09.342250    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:25:13.692664 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:13.692633    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:25:13.795615 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:13.795580    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:25:13.968970 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:13.968887    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:25:15.227916 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:15.227878    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:25:15.987240 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:15.987210    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:25:16.706556 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:16.706514    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:25:19.851657 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:19.851616    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:25:25.014184 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:25.014150    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:25:25.062269 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:25.062235    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2b32aa-7d7c8cddcd-crjgz"]
Sep 14 03:25:25.062545 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:25.062500    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2b32aa-7d7c8cddcd-crjgz" podUID="681e44c9-583c-4b8e-909e-59f8efbc837a" containerName="payload-pre-processing" containerID="cri-o://48843065543e2bedc40c08daaa066935fcd81b437d765e6c7d250b6890f65370" gracePeriod=30
Sep 14 03:25:25.112782 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:25.112747    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:25:25.213086 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:25.213050    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:25:25.294796 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:25.294764    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:25:25.321953 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:25.321930    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2b32aa-7d7c8cddcd-crjgz"
Sep 14 03:25:25.425217 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:25.425190    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/681e44c9-583c-4b8e-909e-59f8efbc837a-plugins-config-volume\") pod \"681e44c9-583c-4b8e-909e-59f8efbc837a\" (UID: \"681e44c9-583c-4b8e-909e-59f8efbc837a\") "
Sep 14 03:25:25.425368 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:25.425257    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6p9ws\" (UniqueName: \"kubernetes.io/projected/681e44c9-583c-4b8e-909e-59f8efbc837a-kube-api-access-6p9ws\") pod \"681e44c9-583c-4b8e-909e-59f8efbc837a\" (UID: \"681e44c9-583c-4b8e-909e-59f8efbc837a\") "
Sep 14 03:25:25.425591 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:25.425567    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/681e44c9-583c-4b8e-909e-59f8efbc837a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "681e44c9-583c-4b8e-909e-59f8efbc837a" (UID: "681e44c9-583c-4b8e-909e-59f8efbc837a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 03:25:25.428657 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:25.427229    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/681e44c9-583c-4b8e-909e-59f8efbc837a-kube-api-access-6p9ws" (OuterVolumeSpecName: "kube-api-access-6p9ws") pod "681e44c9-583c-4b8e-909e-59f8efbc837a" (UID: "681e44c9-583c-4b8e-909e-59f8efbc837a"). InnerVolumeSpecName "kube-api-access-6p9ws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:25:25.525851 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:25.525818    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/681e44c9-583c-4b8e-909e-59f8efbc837a-plugins-config-volume\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:25:25.525851 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:25.525847    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6p9ws\" (UniqueName: \"kubernetes.io/projected/681e44c9-583c-4b8e-909e-59f8efbc837a-kube-api-access-6p9ws\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:25:25.580221 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:25.580123    2564 generic.go:356] "Generic (PLEG): container finished" podID="681e44c9-583c-4b8e-909e-59f8efbc837a" containerID="48843065543e2bedc40c08daaa066935fcd81b437d765e6c7d250b6890f65370" exitCode=0
Sep 14 03:25:25.580221 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:25.580176    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2b32aa-7d7c8cddcd-crjgz"
Sep 14 03:25:25.580221 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:25.580207    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2b32aa-7d7c8cddcd-crjgz" event={"ID":"681e44c9-583c-4b8e-909e-59f8efbc837a","Type":"ContainerDied","Data":"48843065543e2bedc40c08daaa066935fcd81b437d765e6c7d250b6890f65370"}
Sep 14 03:25:25.580466 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:25.580278    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2b32aa-7d7c8cddcd-crjgz" event={"ID":"681e44c9-583c-4b8e-909e-59f8efbc837a","Type":"ContainerDied","Data":"cfe937c5d4f6ace4888fd6c5d62f28771b2b9948b37106f7a5222725e2251534"}
Sep 14 03:25:25.580466 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:25.580301    2564 scope.go:117] "RemoveContainer" containerID="48843065543e2bedc40c08daaa066935fcd81b437d765e6c7d250b6890f65370"
Sep 14 03:25:25.588688 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:25.588672    2564 scope.go:117] "RemoveContainer" containerID="48843065543e2bedc40c08daaa066935fcd81b437d765e6c7d250b6890f65370"
Sep 14 03:25:25.588946 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:25:25.588929    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48843065543e2bedc40c08daaa066935fcd81b437d765e6c7d250b6890f65370\": container with ID starting with 48843065543e2bedc40c08daaa066935fcd81b437d765e6c7d250b6890f65370 not found: ID does not exist" containerID="48843065543e2bedc40c08daaa066935fcd81b437d765e6c7d250b6890f65370"
Sep 14 03:25:25.589005 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:25.588954    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48843065543e2bedc40c08daaa066935fcd81b437d765e6c7d250b6890f65370"} err="failed to get container status \"48843065543e2bedc40c08daaa066935fcd81b437d765e6c7d250b6890f65370\": rpc error: code = NotFound desc = could not find container \"48843065543e2bedc40c08daaa066935fcd81b437d765e6c7d250b6890f65370\": container with ID starting with 48843065543e2bedc40c08daaa066935fcd81b437d765e6c7d250b6890f65370 not found: ID does not exist"
Sep 14 03:25:25.609125 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:25.609105    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2b32aa-7d7c8cddcd-crjgz"]
Sep 14 03:25:25.613106 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:25.613083    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2b32aa-7d7c8cddcd-crjgz"]
Sep 14 03:25:25.685471 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:25.685442    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:25:25.782530 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:25.782503    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:25:26.446703 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:26.446674    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:25:27.098586 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:27.098557    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="681e44c9-583c-4b8e-909e-59f8efbc837a" path="/var/lib/kubelet/pods/681e44c9-583c-4b8e-909e-59f8efbc837a/volumes"
Sep 14 03:25:32.552447 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:32.552388    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:25:32.685550 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:32.685513    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:25:32.987064 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:32.986983    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:25:34.586345 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:34.586280    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:25:35.340768 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:35.340730    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:25:43.907124 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:43.907094    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-177593-bc8cdccc7-9psb4"]
Sep 14 03:25:43.907673 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:43.907293    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-177593-bc8cdccc7-9psb4" podUID="91302c2a-0745-436a-a6ec-4eda74bd18d7" containerName="maas-api" containerID="cri-o://f08b9a1f8602de5e829e8013800e0db0dc22d29c19e70be0a467cd79a6afa32f" gracePeriod=30
Sep 14 03:25:43.946479 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:43.946385    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:25:44.024275 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:44.024246    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-177593-b77c9949d-8gtv4"]
Sep 14 03:25:44.024495 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:44.024472    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-177593-b77c9949d-8gtv4" podUID="6ea449a0-fb86-44c4-a89f-e8314595862d" containerName="payload-pre-processing" containerID="cri-o://e3c33d0c54f6fd8a07e1405224d784d92ac1ae6be692b344def4e18764fdb5fc" gracePeriod=30
Sep 14 03:25:44.059197 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:44.059168    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:25:44.162159 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:44.162104    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-177593-bc8cdccc7-9psb4"
Sep 14 03:25:44.165794 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:44.165767    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:25:44.273707 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:44.273680    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:25:44.278362 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:44.278339    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-177593-b77c9949d-8gtv4"
Sep 14 03:25:44.299220 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:44.299135    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9t4bs\" (UniqueName: \"kubernetes.io/projected/91302c2a-0745-436a-a6ec-4eda74bd18d7-kube-api-access-9t4bs\") pod \"91302c2a-0745-436a-a6ec-4eda74bd18d7\" (UID: \"91302c2a-0745-436a-a6ec-4eda74bd18d7\") "
Sep 14 03:25:44.299220 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:44.299210    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91302c2a-0745-436a-a6ec-4eda74bd18d7-metrics-certs\") pod \"91302c2a-0745-436a-a6ec-4eda74bd18d7\" (UID: \"91302c2a-0745-436a-a6ec-4eda74bd18d7\") "
Sep 14 03:25:44.299522 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:44.299258    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91302c2a-0745-436a-a6ec-4eda74bd18d7-openshift-service-ca\") pod \"91302c2a-0745-436a-a6ec-4eda74bd18d7\" (UID: \"91302c2a-0745-436a-a6ec-4eda74bd18d7\") "
Sep 14 03:25:44.299522 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:44.299376    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91302c2a-0745-436a-a6ec-4eda74bd18d7-maas-api-tls\") pod \"91302c2a-0745-436a-a6ec-4eda74bd18d7\" (UID: \"91302c2a-0745-436a-a6ec-4eda74bd18d7\") "
Sep 14 03:25:44.300523 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:44.300495    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91302c2a-0745-436a-a6ec-4eda74bd18d7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "91302c2a-0745-436a-a6ec-4eda74bd18d7" (UID: "91302c2a-0745-436a-a6ec-4eda74bd18d7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 03:25:44.302737 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:44.302348    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91302c2a-0745-436a-a6ec-4eda74bd18d7-kube-api-access-9t4bs" (OuterVolumeSpecName: "kube-api-access-9t4bs") pod "91302c2a-0745-436a-a6ec-4eda74bd18d7" (UID: "91302c2a-0745-436a-a6ec-4eda74bd18d7"). InnerVolumeSpecName "kube-api-access-9t4bs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:25:44.302834 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:44.302773    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91302c2a-0745-436a-a6ec-4eda74bd18d7-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "91302c2a-0745-436a-a6ec-4eda74bd18d7" (UID: "91302c2a-0745-436a-a6ec-4eda74bd18d7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 03:25:44.302834 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:44.302799    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91302c2a-0745-436a-a6ec-4eda74bd18d7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "91302c2a-0745-436a-a6ec-4eda74bd18d7" (UID: "91302c2a-0745-436a-a6ec-4eda74bd18d7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 03:25:44.400579 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:44.400518    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ea449a0-fb86-44c4-a89f-e8314595862d-plugins-config-volume\") pod \"6ea449a0-fb86-44c4-a89f-e8314595862d\" (UID: \"6ea449a0-fb86-44c4-a89f-e8314595862d\") "
Sep 14 03:25:44.400579 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:44.400578    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l8j5g\" (UniqueName: \"kubernetes.io/projected/6ea449a0-fb86-44c4-a89f-e8314595862d-kube-api-access-l8j5g\") pod \"6ea449a0-fb86-44c4-a89f-e8314595862d\" (UID: \"6ea449a0-fb86-44c4-a89f-e8314595862d\") "
Sep 14 03:25:44.400947 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:44.400922    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9t4bs\" (UniqueName: \"kubernetes.io/projected/91302c2a-0745-436a-a6ec-4eda74bd18d7-kube-api-access-9t4bs\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:25:44.400947 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:44.400946    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91302c2a-0745-436a-a6ec-4eda74bd18d7-metrics-certs\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:25:44.401089 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:44.400956    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91302c2a-0745-436a-a6ec-4eda74bd18d7-openshift-service-ca\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:25:44.401089 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:44.400964    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91302c2a-0745-436a-a6ec-4eda74bd18d7-maas-api-tls\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:25:44.401089 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:44.400987    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ea449a0-fb86-44c4-a89f-e8314595862d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6ea449a0-fb86-44c4-a89f-e8314595862d" (UID: "6ea449a0-fb86-44c4-a89f-e8314595862d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 03:25:44.402658 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:44.402638    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ea449a0-fb86-44c4-a89f-e8314595862d-kube-api-access-l8j5g" (OuterVolumeSpecName: "kube-api-access-l8j5g") pod "6ea449a0-fb86-44c4-a89f-e8314595862d" (UID: "6ea449a0-fb86-44c4-a89f-e8314595862d"). InnerVolumeSpecName "kube-api-access-l8j5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:25:44.416311 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:44.416217    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:25:44.502345 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:44.502307    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ea449a0-fb86-44c4-a89f-e8314595862d-plugins-config-volume\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:25:44.502345 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:44.502336    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8j5g\" (UniqueName: \"kubernetes.io/projected/6ea449a0-fb86-44c4-a89f-e8314595862d-kube-api-access-l8j5g\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:25:44.681134 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:44.681047    2564 generic.go:356] "Generic (PLEG): container finished" podID="91302c2a-0745-436a-a6ec-4eda74bd18d7" containerID="f08b9a1f8602de5e829e8013800e0db0dc22d29c19e70be0a467cd79a6afa32f" exitCode=0
Sep 14 03:25:44.681134 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:44.681106    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-177593-bc8cdccc7-9psb4"
Sep 14 03:25:44.681363 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:44.681135    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-177593-bc8cdccc7-9psb4" event={"ID":"91302c2a-0745-436a-a6ec-4eda74bd18d7","Type":"ContainerDied","Data":"f08b9a1f8602de5e829e8013800e0db0dc22d29c19e70be0a467cd79a6afa32f"}
Sep 14 03:25:44.681363 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:44.681230    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-177593-bc8cdccc7-9psb4" event={"ID":"91302c2a-0745-436a-a6ec-4eda74bd18d7","Type":"ContainerDied","Data":"706ece2dbfb65c2a4cccb776b9a20e78f4c4576cf8ddbd3333bbcb9f193f3f44"}
Sep 14 03:25:44.681363 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:44.681258    2564 scope.go:117] "RemoveContainer" containerID="f08b9a1f8602de5e829e8013800e0db0dc22d29c19e70be0a467cd79a6afa32f"
Sep 14 03:25:44.682703 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:44.682685    2564 generic.go:356] "Generic (PLEG): container finished" podID="6ea449a0-fb86-44c4-a89f-e8314595862d" containerID="e3c33d0c54f6fd8a07e1405224d784d92ac1ae6be692b344def4e18764fdb5fc" exitCode=0
Sep 14 03:25:44.682793 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:44.682710    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-177593-b77c9949d-8gtv4" event={"ID":"6ea449a0-fb86-44c4-a89f-e8314595862d","Type":"ContainerDied","Data":"e3c33d0c54f6fd8a07e1405224d784d92ac1ae6be692b344def4e18764fdb5fc"}
Sep 14 03:25:44.682793 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:44.682757    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-177593-b77c9949d-8gtv4"
Sep 14 03:25:44.682793 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:44.682764    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-177593-b77c9949d-8gtv4" event={"ID":"6ea449a0-fb86-44c4-a89f-e8314595862d","Type":"ContainerDied","Data":"3834edebcf987751c7d75c67d8bb9bd888adb4bc90be4b5ba343f45b4a8e3949"}
Sep 14 03:25:44.692018 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:44.692002    2564 scope.go:117] "RemoveContainer" containerID="f08b9a1f8602de5e829e8013800e0db0dc22d29c19e70be0a467cd79a6afa32f"
Sep 14 03:25:44.692422 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:25:44.692383    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f08b9a1f8602de5e829e8013800e0db0dc22d29c19e70be0a467cd79a6afa32f\": container with ID starting with f08b9a1f8602de5e829e8013800e0db0dc22d29c19e70be0a467cd79a6afa32f not found: ID does not exist" containerID="f08b9a1f8602de5e829e8013800e0db0dc22d29c19e70be0a467cd79a6afa32f"
Sep 14 03:25:44.692512 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:44.692427    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f08b9a1f8602de5e829e8013800e0db0dc22d29c19e70be0a467cd79a6afa32f"} err="failed to get container status \"f08b9a1f8602de5e829e8013800e0db0dc22d29c19e70be0a467cd79a6afa32f\": rpc error: code = NotFound desc = could not find container \"f08b9a1f8602de5e829e8013800e0db0dc22d29c19e70be0a467cd79a6afa32f\": container with ID starting with f08b9a1f8602de5e829e8013800e0db0dc22d29c19e70be0a467cd79a6afa32f not found: ID does not exist"
Sep 14 03:25:44.692512 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:44.692443    2564 scope.go:117] "RemoveContainer" containerID="e3c33d0c54f6fd8a07e1405224d784d92ac1ae6be692b344def4e18764fdb5fc"
Sep 14 03:25:44.701116 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:44.701099    2564 scope.go:117] "RemoveContainer" containerID="e3c33d0c54f6fd8a07e1405224d784d92ac1ae6be692b344def4e18764fdb5fc"
Sep 14 03:25:44.701355 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:25:44.701338    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3c33d0c54f6fd8a07e1405224d784d92ac1ae6be692b344def4e18764fdb5fc\": container with ID starting with e3c33d0c54f6fd8a07e1405224d784d92ac1ae6be692b344def4e18764fdb5fc not found: ID does not exist" containerID="e3c33d0c54f6fd8a07e1405224d784d92ac1ae6be692b344def4e18764fdb5fc"
Sep 14 03:25:44.701398 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:44.701361    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3c33d0c54f6fd8a07e1405224d784d92ac1ae6be692b344def4e18764fdb5fc"} err="failed to get container status \"e3c33d0c54f6fd8a07e1405224d784d92ac1ae6be692b344def4e18764fdb5fc\": rpc error: code = NotFound desc = could not find container \"e3c33d0c54f6fd8a07e1405224d784d92ac1ae6be692b344def4e18764fdb5fc\": container with ID starting with e3c33d0c54f6fd8a07e1405224d784d92ac1ae6be692b344def4e18764fdb5fc not found: ID does not exist"
Sep 14 03:25:44.710975 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:44.710951    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:25:44.711070 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:44.711053    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-177593-bc8cdccc7-9psb4"]
Sep 14 03:25:44.726201 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:44.726170    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-177593-bc8cdccc7-9psb4"]
Sep 14 03:25:44.739122 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:44.739095    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-177593-b77c9949d-8gtv4"]
Sep 14 03:25:44.741720 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:44.741702    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-177593-b77c9949d-8gtv4"]
Sep 14 03:25:44.798605 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:44.798580    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:25:45.098429 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:45.098371    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ea449a0-fb86-44c4-a89f-e8314595862d" path="/var/lib/kubelet/pods/6ea449a0-fb86-44c4-a89f-e8314595862d/volumes"
Sep 14 03:25:45.098974 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:45.098953    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91302c2a-0745-436a-a6ec-4eda74bd18d7" path="/var/lib/kubelet/pods/91302c2a-0745-436a-a6ec-4eda74bd18d7/volumes"
Sep 14 03:25:51.105772 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:51.105736    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:25:51.183421 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:51.183379    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:25:51.389815 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:51.389726    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:25:53.913525 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:53.913489    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-57d64b7cfb-pbg2k"]
Sep 14 03:25:53.914008 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:53.913710    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-57d64b7cfb-pbg2k" podUID="cec92154-773e-4dd0-b882-6b4e98bc51e4" containerName="manager" containerID="cri-o://c7d25db228ef20e17d423426354210a33626e37d89744b6dc61bef4f0356bdf4" gracePeriod=30
Sep 14 03:25:54.155337 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:54.155311    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57d64b7cfb-pbg2k"
Sep 14 03:25:54.299915 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:54.299886    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cec92154-773e-4dd0-b882-6b4e98bc51e4-webhook-cert\") pod \"cec92154-773e-4dd0-b882-6b4e98bc51e4\" (UID: \"cec92154-773e-4dd0-b882-6b4e98bc51e4\") "
Sep 14 03:25:54.300102 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:54.299939    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cec92154-773e-4dd0-b882-6b4e98bc51e4-metrics-certs\") pod \"cec92154-773e-4dd0-b882-6b4e98bc51e4\" (UID: \"cec92154-773e-4dd0-b882-6b4e98bc51e4\") "
Sep 14 03:25:54.300102 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:54.299964    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qdkhf\" (UniqueName: \"kubernetes.io/projected/cec92154-773e-4dd0-b882-6b4e98bc51e4-kube-api-access-qdkhf\") pod \"cec92154-773e-4dd0-b882-6b4e98bc51e4\" (UID: \"cec92154-773e-4dd0-b882-6b4e98bc51e4\") "
Sep 14 03:25:54.302053 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:54.302006    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cec92154-773e-4dd0-b882-6b4e98bc51e4-kube-api-access-qdkhf" (OuterVolumeSpecName: "kube-api-access-qdkhf") pod "cec92154-773e-4dd0-b882-6b4e98bc51e4" (UID: "cec92154-773e-4dd0-b882-6b4e98bc51e4"). InnerVolumeSpecName "kube-api-access-qdkhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:25:54.302053 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:54.302042    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cec92154-773e-4dd0-b882-6b4e98bc51e4-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "cec92154-773e-4dd0-b882-6b4e98bc51e4" (UID: "cec92154-773e-4dd0-b882-6b4e98bc51e4"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 03:25:54.302204 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:54.302064    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cec92154-773e-4dd0-b882-6b4e98bc51e4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "cec92154-773e-4dd0-b882-6b4e98bc51e4" (UID: "cec92154-773e-4dd0-b882-6b4e98bc51e4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 03:25:54.401069 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:54.401034    2564 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cec92154-773e-4dd0-b882-6b4e98bc51e4-webhook-cert\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:25:54.401069 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:54.401061    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cec92154-773e-4dd0-b882-6b4e98bc51e4-metrics-certs\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:25:54.401069 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:54.401075    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdkhf\" (UniqueName: \"kubernetes.io/projected/cec92154-773e-4dd0-b882-6b4e98bc51e4-kube-api-access-qdkhf\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:25:54.736122 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:54.736022    2564 generic.go:356] "Generic (PLEG): container finished" podID="cec92154-773e-4dd0-b882-6b4e98bc51e4" containerID="c7d25db228ef20e17d423426354210a33626e37d89744b6dc61bef4f0356bdf4" exitCode=0
Sep 14 03:25:54.736122 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:54.736074    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57d64b7cfb-pbg2k"
Sep 14 03:25:54.736122 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:54.736107    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57d64b7cfb-pbg2k" event={"ID":"cec92154-773e-4dd0-b882-6b4e98bc51e4","Type":"ContainerDied","Data":"c7d25db228ef20e17d423426354210a33626e37d89744b6dc61bef4f0356bdf4"}
Sep 14 03:25:54.736392 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:54.736159    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57d64b7cfb-pbg2k" event={"ID":"cec92154-773e-4dd0-b882-6b4e98bc51e4","Type":"ContainerDied","Data":"9aa0d8ff860bfe9857308733d14d3cf1146003231659dac6cac05d4bb343360b"}
Sep 14 03:25:54.736392 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:54.736181    2564 scope.go:117] "RemoveContainer" containerID="c7d25db228ef20e17d423426354210a33626e37d89744b6dc61bef4f0356bdf4"
Sep 14 03:25:54.745709 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:54.745692    2564 scope.go:117] "RemoveContainer" containerID="c7d25db228ef20e17d423426354210a33626e37d89744b6dc61bef4f0356bdf4"
Sep 14 03:25:54.745963 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:25:54.745944    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7d25db228ef20e17d423426354210a33626e37d89744b6dc61bef4f0356bdf4\": container with ID starting with c7d25db228ef20e17d423426354210a33626e37d89744b6dc61bef4f0356bdf4 not found: ID does not exist" containerID="c7d25db228ef20e17d423426354210a33626e37d89744b6dc61bef4f0356bdf4"
Sep 14 03:25:54.746010 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:54.745970    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7d25db228ef20e17d423426354210a33626e37d89744b6dc61bef4f0356bdf4"} err="failed to get container status \"c7d25db228ef20e17d423426354210a33626e37d89744b6dc61bef4f0356bdf4\": rpc error: code = NotFound desc = could not find container \"c7d25db228ef20e17d423426354210a33626e37d89744b6dc61bef4f0356bdf4\": container with ID starting with c7d25db228ef20e17d423426354210a33626e37d89744b6dc61bef4f0356bdf4 not found: ID does not exist"
Sep 14 03:25:54.757201 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:54.757170    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-57d64b7cfb-pbg2k"]
Sep 14 03:25:54.761506 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:54.761483    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-57d64b7cfb-pbg2k"]
Sep 14 03:25:55.098111 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:25:55.098085    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cec92154-773e-4dd0-b882-6b4e98bc51e4" path="/var/lib/kubelet/pods/cec92154-773e-4dd0-b882-6b4e98bc51e4/volumes"
Sep 14 03:26:55.589386 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:26:55.589353    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-57d64b7cfb-v9hxr"]
Sep 14 03:26:55.589875 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:26:55.589767    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce8f6872-4aa2-4405-88ed-d7746c8c4ede" containerName="payload-pre-processing"
Sep 14 03:26:55.589875 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:26:55.589780    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce8f6872-4aa2-4405-88ed-d7746c8c4ede" containerName="payload-pre-processing"
Sep 14 03:26:55.589875 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:26:55.589792    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="681e44c9-583c-4b8e-909e-59f8efbc837a" containerName="payload-pre-processing"
Sep 14 03:26:55.589875 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:26:55.589797    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="681e44c9-583c-4b8e-909e-59f8efbc837a" containerName="payload-pre-processing"
Sep 14 03:26:55.589875 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:26:55.589805    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91302c2a-0745-436a-a6ec-4eda74bd18d7" containerName="maas-api"
Sep 14 03:26:55.589875 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:26:55.589811    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="91302c2a-0745-436a-a6ec-4eda74bd18d7" containerName="maas-api"
Sep 14 03:26:55.589875 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:26:55.589825    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54ab7623-06c6-471e-a412-2c3538877408" containerName="payload-processing"
Sep 14 03:26:55.589875 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:26:55.589830    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="54ab7623-06c6-471e-a412-2c3538877408" containerName="payload-processing"
Sep 14 03:26:55.589875 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:26:55.589838    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ea449a0-fb86-44c4-a89f-e8314595862d" containerName="payload-pre-processing"
Sep 14 03:26:55.589875 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:26:55.589845    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ea449a0-fb86-44c4-a89f-e8314595862d" containerName="payload-pre-processing"
Sep 14 03:26:55.589875 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:26:55.589852    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cec92154-773e-4dd0-b882-6b4e98bc51e4" containerName="manager"
Sep 14 03:26:55.589875 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:26:55.589857    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="cec92154-773e-4dd0-b882-6b4e98bc51e4" containerName="manager"
Sep 14 03:26:55.589875 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:26:55.589864    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45433dea-3cc8-4746-9090-83be7ade5bd3" containerName="maas-api"
Sep 14 03:26:55.589875 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:26:55.589869    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="45433dea-3cc8-4746-9090-83be7ade5bd3" containerName="maas-api"
Sep 14 03:26:55.589875 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:26:55.589874    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="891107e4-feb4-41fe-b1ef-8ac6721198dc" containerName="payload-pre-processing"
Sep 14 03:26:55.589875 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:26:55.589889    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="891107e4-feb4-41fe-b1ef-8ac6721198dc" containerName="payload-pre-processing"
Sep 14 03:26:55.590438 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:26:55.589897    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c74e592f-9364-4a6e-a1a2-9ea5fa74ebf3" containerName="payload-processing"
Sep 14 03:26:55.590438 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:26:55.589902    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c74e592f-9364-4a6e-a1a2-9ea5fa74ebf3" containerName="payload-processing"
Sep 14 03:26:55.590438 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:26:55.589956    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c74e592f-9364-4a6e-a1a2-9ea5fa74ebf3" containerName="payload-processing"
Sep 14 03:26:55.590438 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:26:55.589964    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce8f6872-4aa2-4405-88ed-d7746c8c4ede" containerName="payload-pre-processing"
Sep 14 03:26:55.590438 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:26:55.589971    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="91302c2a-0745-436a-a6ec-4eda74bd18d7" containerName="maas-api"
Sep 14 03:26:55.590438 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:26:55.589977    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="cec92154-773e-4dd0-b882-6b4e98bc51e4" containerName="manager"
Sep 14 03:26:55.590438 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:26:55.589984    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ea449a0-fb86-44c4-a89f-e8314595862d" containerName="payload-pre-processing"
Sep 14 03:26:55.590438 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:26:55.589989    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="681e44c9-583c-4b8e-909e-59f8efbc837a" containerName="payload-pre-processing"
Sep 14 03:26:55.590438 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:26:55.589995    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="54ab7623-06c6-471e-a412-2c3538877408" containerName="payload-processing"
Sep 14 03:26:55.590438 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:26:55.590000    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="891107e4-feb4-41fe-b1ef-8ac6721198dc" containerName="payload-pre-processing"
Sep 14 03:26:55.590438 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:26:55.590005    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="45433dea-3cc8-4746-9090-83be7ade5bd3" containerName="maas-api"
Sep 14 03:26:55.593177 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:26:55.593162    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57d64b7cfb-v9hxr"
Sep 14 03:26:55.597728 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:26:55.597705    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 14 03:26:55.597872 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:26:55.597705    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 14 03:26:55.597872 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:26:55.597764    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 14 03:26:55.597872 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:26:55.597828    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-m8df5\""
Sep 14 03:26:55.607500 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:26:55.607478    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-57d64b7cfb-v9hxr"]
Sep 14 03:26:55.750882 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:26:55.750829    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc16f852-37c9-40a8-b4a2-ad8681df9890-metrics-certs\") pod \"maas-controller-57d64b7cfb-v9hxr\" (UID: \"fc16f852-37c9-40a8-b4a2-ad8681df9890\") " pod="opendatahub/maas-controller-57d64b7cfb-v9hxr"
Sep 14 03:26:55.751068 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:26:55.750982    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j84hh\" (UniqueName: \"kubernetes.io/projected/fc16f852-37c9-40a8-b4a2-ad8681df9890-kube-api-access-j84hh\") pod \"maas-controller-57d64b7cfb-v9hxr\" (UID: \"fc16f852-37c9-40a8-b4a2-ad8681df9890\") " pod="opendatahub/maas-controller-57d64b7cfb-v9hxr"
Sep 14 03:26:55.751068 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:26:55.751011    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fc16f852-37c9-40a8-b4a2-ad8681df9890-webhook-cert\") pod \"maas-controller-57d64b7cfb-v9hxr\" (UID: \"fc16f852-37c9-40a8-b4a2-ad8681df9890\") " pod="opendatahub/maas-controller-57d64b7cfb-v9hxr"
Sep 14 03:26:55.851746 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:26:55.851670    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j84hh\" (UniqueName: \"kubernetes.io/projected/fc16f852-37c9-40a8-b4a2-ad8681df9890-kube-api-access-j84hh\") pod \"maas-controller-57d64b7cfb-v9hxr\" (UID: \"fc16f852-37c9-40a8-b4a2-ad8681df9890\") " pod="opendatahub/maas-controller-57d64b7cfb-v9hxr"
Sep 14 03:26:55.851746 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:26:55.851704    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fc16f852-37c9-40a8-b4a2-ad8681df9890-webhook-cert\") pod \"maas-controller-57d64b7cfb-v9hxr\" (UID: \"fc16f852-37c9-40a8-b4a2-ad8681df9890\") " pod="opendatahub/maas-controller-57d64b7cfb-v9hxr"
Sep 14 03:26:55.851922 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:26:55.851752    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc16f852-37c9-40a8-b4a2-ad8681df9890-metrics-certs\") pod \"maas-controller-57d64b7cfb-v9hxr\" (UID: \"fc16f852-37c9-40a8-b4a2-ad8681df9890\") " pod="opendatahub/maas-controller-57d64b7cfb-v9hxr"
Sep 14 03:26:55.854150 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:26:55.854116    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fc16f852-37c9-40a8-b4a2-ad8681df9890-webhook-cert\") pod \"maas-controller-57d64b7cfb-v9hxr\" (UID: \"fc16f852-37c9-40a8-b4a2-ad8681df9890\") " pod="opendatahub/maas-controller-57d64b7cfb-v9hxr"
Sep 14 03:26:55.854270 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:26:55.854171    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc16f852-37c9-40a8-b4a2-ad8681df9890-metrics-certs\") pod \"maas-controller-57d64b7cfb-v9hxr\" (UID: \"fc16f852-37c9-40a8-b4a2-ad8681df9890\") " pod="opendatahub/maas-controller-57d64b7cfb-v9hxr"
Sep 14 03:26:55.861800 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:26:55.861775    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j84hh\" (UniqueName: \"kubernetes.io/projected/fc16f852-37c9-40a8-b4a2-ad8681df9890-kube-api-access-j84hh\") pod \"maas-controller-57d64b7cfb-v9hxr\" (UID: \"fc16f852-37c9-40a8-b4a2-ad8681df9890\") " pod="opendatahub/maas-controller-57d64b7cfb-v9hxr"
Sep 14 03:26:55.904007 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:26:55.903983    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57d64b7cfb-v9hxr"
Sep 14 03:26:56.177307 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:26:56.177276    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-57d64b7cfb-v9hxr"]
Sep 14 03:26:56.179274 ip-10-0-141-4 kubenswrapper[2564]: W0914 03:26:56.179249    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc16f852_37c9_40a8_b4a2_ad8681df9890.slice/crio-adccd82761c0375d2288827bbe5e3fe2b8edbc64c5ac348dafd849e02b3d365b WatchSource:0}: Error finding container adccd82761c0375d2288827bbe5e3fe2b8edbc64c5ac348dafd849e02b3d365b: Status 404 returned error can't find the container with id adccd82761c0375d2288827bbe5e3fe2b8edbc64c5ac348dafd849e02b3d365b
Sep 14 03:26:56.180703 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:26:56.180686    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 14 03:26:57.043598 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:26:57.043556    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57d64b7cfb-v9hxr" event={"ID":"fc16f852-37c9-40a8-b4a2-ad8681df9890","Type":"ContainerStarted","Data":"61584d2a4f9c26694165c2638cce41a08f5edf4cffab48b5fad9c72126d057ab"}
Sep 14 03:26:57.044037 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:26:57.043621    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57d64b7cfb-v9hxr" event={"ID":"fc16f852-37c9-40a8-b4a2-ad8681df9890","Type":"ContainerStarted","Data":"adccd82761c0375d2288827bbe5e3fe2b8edbc64c5ac348dafd849e02b3d365b"}
Sep 14 03:26:57.044037 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:26:57.043718    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-57d64b7cfb-v9hxr"
Sep 14 03:26:57.070990 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:26:57.070942    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-57d64b7cfb-v9hxr" podStartSLOduration=1.602441687 podStartE2EDuration="2.070923501s" totalImagesPullingTime="468.481814ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:26:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:26:57.068232785 +0000 UTC m=+2062.449316983" watchObservedRunningTime="2026-09-14 03:26:57.070923501 +0000 UTC m=+2062.452007700"
Sep 14 03:27:08.051229 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:27:08.051200    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-57d64b7cfb-v9hxr"
Sep 14 03:27:17.280363 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:27:17.280330    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:27:17.387693 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:27:17.387653    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-57d64b7cfb-v9hxr"]
Sep 14 03:27:17.387894 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:27:17.387869    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-57d64b7cfb-v9hxr" podUID="fc16f852-37c9-40a8-b4a2-ad8681df9890" containerName="manager" containerID="cri-o://61584d2a4f9c26694165c2638cce41a08f5edf4cffab48b5fad9c72126d057ab" gracePeriod=30
Sep 14 03:27:17.633718 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:27:17.633692    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57d64b7cfb-v9hxr"
Sep 14 03:27:17.742914 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:27:17.742885    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j84hh\" (UniqueName: \"kubernetes.io/projected/fc16f852-37c9-40a8-b4a2-ad8681df9890-kube-api-access-j84hh\") pod \"fc16f852-37c9-40a8-b4a2-ad8681df9890\" (UID: \"fc16f852-37c9-40a8-b4a2-ad8681df9890\") "
Sep 14 03:27:17.742914 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:27:17.742918    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fc16f852-37c9-40a8-b4a2-ad8681df9890-webhook-cert\") pod \"fc16f852-37c9-40a8-b4a2-ad8681df9890\" (UID: \"fc16f852-37c9-40a8-b4a2-ad8681df9890\") "
Sep 14 03:27:17.743174 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:27:17.743003    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc16f852-37c9-40a8-b4a2-ad8681df9890-metrics-certs\") pod \"fc16f852-37c9-40a8-b4a2-ad8681df9890\" (UID: \"fc16f852-37c9-40a8-b4a2-ad8681df9890\") "
Sep 14 03:27:17.744953 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:27:17.744918    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc16f852-37c9-40a8-b4a2-ad8681df9890-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "fc16f852-37c9-40a8-b4a2-ad8681df9890" (UID: "fc16f852-37c9-40a8-b4a2-ad8681df9890"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 03:27:17.745072 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:27:17.744986    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc16f852-37c9-40a8-b4a2-ad8681df9890-kube-api-access-j84hh" (OuterVolumeSpecName: "kube-api-access-j84hh") pod "fc16f852-37c9-40a8-b4a2-ad8681df9890" (UID: "fc16f852-37c9-40a8-b4a2-ad8681df9890"). InnerVolumeSpecName "kube-api-access-j84hh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:27:17.745072 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:27:17.744994    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc16f852-37c9-40a8-b4a2-ad8681df9890-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "fc16f852-37c9-40a8-b4a2-ad8681df9890" (UID: "fc16f852-37c9-40a8-b4a2-ad8681df9890"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 03:27:17.844703 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:27:17.844621    2564 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fc16f852-37c9-40a8-b4a2-ad8681df9890-webhook-cert\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:27:17.844703 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:27:17.844650    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc16f852-37c9-40a8-b4a2-ad8681df9890-metrics-certs\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:27:17.844703 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:27:17.844663    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j84hh\" (UniqueName: \"kubernetes.io/projected/fc16f852-37c9-40a8-b4a2-ad8681df9890-kube-api-access-j84hh\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:27:18.152362 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:27:18.152279    2564 generic.go:356] "Generic (PLEG): container finished" podID="fc16f852-37c9-40a8-b4a2-ad8681df9890" containerID="61584d2a4f9c26694165c2638cce41a08f5edf4cffab48b5fad9c72126d057ab" exitCode=0
Sep 14 03:27:18.152362 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:27:18.152325    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57d64b7cfb-v9hxr"
Sep 14 03:27:18.152564 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:27:18.152319    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57d64b7cfb-v9hxr" event={"ID":"fc16f852-37c9-40a8-b4a2-ad8681df9890","Type":"ContainerDied","Data":"61584d2a4f9c26694165c2638cce41a08f5edf4cffab48b5fad9c72126d057ab"}
Sep 14 03:27:18.152564 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:27:18.152455    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57d64b7cfb-v9hxr" event={"ID":"fc16f852-37c9-40a8-b4a2-ad8681df9890","Type":"ContainerDied","Data":"adccd82761c0375d2288827bbe5e3fe2b8edbc64c5ac348dafd849e02b3d365b"}
Sep 14 03:27:18.152564 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:27:18.152478    2564 scope.go:117] "RemoveContainer" containerID="61584d2a4f9c26694165c2638cce41a08f5edf4cffab48b5fad9c72126d057ab"
Sep 14 03:27:18.161787 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:27:18.161769    2564 scope.go:117] "RemoveContainer" containerID="61584d2a4f9c26694165c2638cce41a08f5edf4cffab48b5fad9c72126d057ab"
Sep 14 03:27:18.162057 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:27:18.162039    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61584d2a4f9c26694165c2638cce41a08f5edf4cffab48b5fad9c72126d057ab\": container with ID starting with 61584d2a4f9c26694165c2638cce41a08f5edf4cffab48b5fad9c72126d057ab not found: ID does not exist" containerID="61584d2a4f9c26694165c2638cce41a08f5edf4cffab48b5fad9c72126d057ab"
Sep 14 03:27:18.162120 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:27:18.162065    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61584d2a4f9c26694165c2638cce41a08f5edf4cffab48b5fad9c72126d057ab"} err="failed to get container status \"61584d2a4f9c26694165c2638cce41a08f5edf4cffab48b5fad9c72126d057ab\": rpc error: code = NotFound desc = could not find container \"61584d2a4f9c26694165c2638cce41a08f5edf4cffab48b5fad9c72126d057ab\": container with ID starting with 61584d2a4f9c26694165c2638cce41a08f5edf4cffab48b5fad9c72126d057ab not found: ID does not exist"
Sep 14 03:27:18.174535 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:27:18.174514    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-57d64b7cfb-v9hxr"]
Sep 14 03:27:18.177827 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:27:18.177809    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-57d64b7cfb-v9hxr"]
Sep 14 03:27:18.832056 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:27:18.832022    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:27:18.946677 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:27:18.946649    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-57d64b7cfb-55l9k"]
Sep 14 03:27:18.947081 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:27:18.947065    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc16f852-37c9-40a8-b4a2-ad8681df9890" containerName="manager"
Sep 14 03:27:18.947081 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:27:18.947082    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc16f852-37c9-40a8-b4a2-ad8681df9890" containerName="manager"
Sep 14 03:27:18.947205 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:27:18.947151    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc16f852-37c9-40a8-b4a2-ad8681df9890" containerName="manager"
Sep 14 03:27:18.951319 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:27:18.951303    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57d64b7cfb-55l9k"
Sep 14 03:27:18.954894 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:27:18.954869    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 14 03:27:18.955007 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:27:18.954872    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-m8df5\""
Sep 14 03:27:18.955007 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:27:18.954928    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 14 03:27:18.955127 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:27:18.955006    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 14 03:27:18.960468 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:27:18.960442    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-57d64b7cfb-55l9k"]
Sep 14 03:27:19.054918 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:27:19.054886    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f275a993-2efb-434d-8594-7c0a5ccbb80f-metrics-certs\") pod \"maas-controller-57d64b7cfb-55l9k\" (UID: \"f275a993-2efb-434d-8594-7c0a5ccbb80f\") " pod="opendatahub/maas-controller-57d64b7cfb-55l9k"
Sep 14 03:27:19.055086 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:27:19.054945    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-229hx\" (UniqueName: \"kubernetes.io/projected/f275a993-2efb-434d-8594-7c0a5ccbb80f-kube-api-access-229hx\") pod \"maas-controller-57d64b7cfb-55l9k\" (UID: \"f275a993-2efb-434d-8594-7c0a5ccbb80f\") " pod="opendatahub/maas-controller-57d64b7cfb-55l9k"
Sep 14 03:27:19.055086 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:27:19.055040    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f275a993-2efb-434d-8594-7c0a5ccbb80f-webhook-cert\") pod \"maas-controller-57d64b7cfb-55l9k\" (UID: \"f275a993-2efb-434d-8594-7c0a5ccbb80f\") " pod="opendatahub/maas-controller-57d64b7cfb-55l9k"
Sep 14 03:27:19.099385 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:27:19.099312    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc16f852-37c9-40a8-b4a2-ad8681df9890" path="/var/lib/kubelet/pods/fc16f852-37c9-40a8-b4a2-ad8681df9890/volumes"
Sep 14 03:27:19.155440 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:27:19.155398    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f275a993-2efb-434d-8594-7c0a5ccbb80f-webhook-cert\") pod \"maas-controller-57d64b7cfb-55l9k\" (UID: \"f275a993-2efb-434d-8594-7c0a5ccbb80f\") " pod="opendatahub/maas-controller-57d64b7cfb-55l9k"
Sep 14 03:27:19.155567 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:27:19.155452    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f275a993-2efb-434d-8594-7c0a5ccbb80f-metrics-certs\") pod \"maas-controller-57d64b7cfb-55l9k\" (UID: \"f275a993-2efb-434d-8594-7c0a5ccbb80f\") " pod="opendatahub/maas-controller-57d64b7cfb-55l9k"
Sep 14 03:27:19.155567 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:27:19.155503    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-229hx\" (UniqueName: \"kubernetes.io/projected/f275a993-2efb-434d-8594-7c0a5ccbb80f-kube-api-access-229hx\") pod \"maas-controller-57d64b7cfb-55l9k\" (UID: \"f275a993-2efb-434d-8594-7c0a5ccbb80f\") " pod="opendatahub/maas-controller-57d64b7cfb-55l9k"
Sep 14 03:27:19.157947 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:27:19.157916    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f275a993-2efb-434d-8594-7c0a5ccbb80f-metrics-certs\") pod \"maas-controller-57d64b7cfb-55l9k\" (UID: \"f275a993-2efb-434d-8594-7c0a5ccbb80f\") " pod="opendatahub/maas-controller-57d64b7cfb-55l9k"
Sep 14 03:27:19.158266 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:27:19.158246    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f275a993-2efb-434d-8594-7c0a5ccbb80f-webhook-cert\") pod \"maas-controller-57d64b7cfb-55l9k\" (UID: \"f275a993-2efb-434d-8594-7c0a5ccbb80f\") " pod="opendatahub/maas-controller-57d64b7cfb-55l9k"
Sep 14 03:27:19.164614 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:27:19.164593    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-229hx\" (UniqueName: \"kubernetes.io/projected/f275a993-2efb-434d-8594-7c0a5ccbb80f-kube-api-access-229hx\") pod \"maas-controller-57d64b7cfb-55l9k\" (UID: \"f275a993-2efb-434d-8594-7c0a5ccbb80f\") " pod="opendatahub/maas-controller-57d64b7cfb-55l9k"
Sep 14 03:27:19.262719 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:27:19.262684    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57d64b7cfb-55l9k"
Sep 14 03:27:19.532116 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:27:19.532093    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-57d64b7cfb-55l9k"]
Sep 14 03:27:19.533974 ip-10-0-141-4 kubenswrapper[2564]: W0914 03:27:19.533948    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf275a993_2efb_434d_8594_7c0a5ccbb80f.slice/crio-2af7e0143f52883b7ec275763f8e6f08bd6c775ecf5e4c3cb210fe4214cb0bee WatchSource:0}: Error finding container 2af7e0143f52883b7ec275763f8e6f08bd6c775ecf5e4c3cb210fe4214cb0bee: Status 404 returned error can't find the container with id 2af7e0143f52883b7ec275763f8e6f08bd6c775ecf5e4c3cb210fe4214cb0bee
Sep 14 03:27:20.166911 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:27:20.166828    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57d64b7cfb-55l9k" event={"ID":"f275a993-2efb-434d-8594-7c0a5ccbb80f","Type":"ContainerStarted","Data":"5860582108cf26ac6e20d413fd0cf320079ecd0d32580544565c4c99706a4b1a"}
Sep 14 03:27:20.166911 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:27:20.166883    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57d64b7cfb-55l9k" event={"ID":"f275a993-2efb-434d-8594-7c0a5ccbb80f","Type":"ContainerStarted","Data":"2af7e0143f52883b7ec275763f8e6f08bd6c775ecf5e4c3cb210fe4214cb0bee"}
Sep 14 03:27:20.167286 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:27:20.166915    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-57d64b7cfb-55l9k"
Sep 14 03:27:20.185980 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:27:20.185926    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-57d64b7cfb-55l9k" podStartSLOduration=1.8435116950000001 podStartE2EDuration="2.185911206s" totalImagesPullingTime="342.399511ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:27:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:27:20.184359886 +0000 UTC m=+2085.565444097" watchObservedRunningTime="2026-09-14 03:27:20.185911206 +0000 UTC m=+2085.566995405"
Sep 14 03:27:31.173757 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:27:31.173731    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-57d64b7cfb-55l9k"
Sep 14 03:28:02.412207 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:28:02.412170    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:28:03.947655 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:28:03.947618    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:28:40.509356 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:28:40.509317    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:28:43.572325 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:28:43.572289    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:28:45.820962 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:28:45.820913    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:29:59.735125 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:29:59.735037    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:30:17.699563 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:30:17.699524    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:30:22.145857 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:30:22.145823    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:31:05.388261 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:31:05.388224    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:31:07.465800 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:31:07.465752    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:31:36.068860 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:31:36.068785    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-559c5f7949-hmb7b"]
Sep 14 03:31:36.071897 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:31:36.071880    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-559c5f7949-hmb7b"
Sep 14 03:31:36.082560 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:31:36.082530    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-559c5f7949-hmb7b"]
Sep 14 03:31:36.123821 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:31:36.123787    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn7kk\" (UniqueName: \"kubernetes.io/projected/0db99058-dfec-4aa8-87e1-cd74369b453c-kube-api-access-qn7kk\") pod \"maas-api-559c5f7949-hmb7b\" (UID: \"0db99058-dfec-4aa8-87e1-cd74369b453c\") " pod="odh-ai-gateway-infra/maas-api-559c5f7949-hmb7b"
Sep 14 03:31:36.124020 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:31:36.123849    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0db99058-dfec-4aa8-87e1-cd74369b453c-maas-api-tls\") pod \"maas-api-559c5f7949-hmb7b\" (UID: \"0db99058-dfec-4aa8-87e1-cd74369b453c\") " pod="odh-ai-gateway-infra/maas-api-559c5f7949-hmb7b"
Sep 14 03:31:36.124020 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:31:36.123883    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0db99058-dfec-4aa8-87e1-cd74369b453c-metrics-certs\") pod \"maas-api-559c5f7949-hmb7b\" (UID: \"0db99058-dfec-4aa8-87e1-cd74369b453c\") " pod="odh-ai-gateway-infra/maas-api-559c5f7949-hmb7b"
Sep 14 03:31:36.124020 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:31:36.123923    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0db99058-dfec-4aa8-87e1-cd74369b453c-openshift-service-ca\") pod \"maas-api-559c5f7949-hmb7b\" (UID: \"0db99058-dfec-4aa8-87e1-cd74369b453c\") " pod="odh-ai-gateway-infra/maas-api-559c5f7949-hmb7b"
Sep 14 03:31:36.225203 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:31:36.225171    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn7kk\" (UniqueName: \"kubernetes.io/projected/0db99058-dfec-4aa8-87e1-cd74369b453c-kube-api-access-qn7kk\") pod \"maas-api-559c5f7949-hmb7b\" (UID: \"0db99058-dfec-4aa8-87e1-cd74369b453c\") " pod="odh-ai-gateway-infra/maas-api-559c5f7949-hmb7b"
Sep 14 03:31:36.225397 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:31:36.225223    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0db99058-dfec-4aa8-87e1-cd74369b453c-maas-api-tls\") pod \"maas-api-559c5f7949-hmb7b\" (UID: \"0db99058-dfec-4aa8-87e1-cd74369b453c\") " pod="odh-ai-gateway-infra/maas-api-559c5f7949-hmb7b"
Sep 14 03:31:36.225397 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:31:36.225247    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0db99058-dfec-4aa8-87e1-cd74369b453c-metrics-certs\") pod \"maas-api-559c5f7949-hmb7b\" (UID: \"0db99058-dfec-4aa8-87e1-cd74369b453c\") " pod="odh-ai-gateway-infra/maas-api-559c5f7949-hmb7b"
Sep 14 03:31:36.225397 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:31:36.225284    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0db99058-dfec-4aa8-87e1-cd74369b453c-openshift-service-ca\") pod \"maas-api-559c5f7949-hmb7b\" (UID: \"0db99058-dfec-4aa8-87e1-cd74369b453c\") " pod="odh-ai-gateway-infra/maas-api-559c5f7949-hmb7b"
Sep 14 03:31:36.226045 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:31:36.226021    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0db99058-dfec-4aa8-87e1-cd74369b453c-openshift-service-ca\") pod \"maas-api-559c5f7949-hmb7b\" (UID: \"0db99058-dfec-4aa8-87e1-cd74369b453c\") " pod="odh-ai-gateway-infra/maas-api-559c5f7949-hmb7b"
Sep 14 03:31:36.227903 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:31:36.227882    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0db99058-dfec-4aa8-87e1-cd74369b453c-metrics-certs\") pod \"maas-api-559c5f7949-hmb7b\" (UID: \"0db99058-dfec-4aa8-87e1-cd74369b453c\") " pod="odh-ai-gateway-infra/maas-api-559c5f7949-hmb7b"
Sep 14 03:31:36.227985 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:31:36.227924    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0db99058-dfec-4aa8-87e1-cd74369b453c-maas-api-tls\") pod \"maas-api-559c5f7949-hmb7b\" (UID: \"0db99058-dfec-4aa8-87e1-cd74369b453c\") " pod="odh-ai-gateway-infra/maas-api-559c5f7949-hmb7b"
Sep 14 03:31:36.235052 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:31:36.235030    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn7kk\" (UniqueName: \"kubernetes.io/projected/0db99058-dfec-4aa8-87e1-cd74369b453c-kube-api-access-qn7kk\") pod \"maas-api-559c5f7949-hmb7b\" (UID: \"0db99058-dfec-4aa8-87e1-cd74369b453c\") " pod="odh-ai-gateway-infra/maas-api-559c5f7949-hmb7b"
Sep 14 03:31:36.383282 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:31:36.383186    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-559c5f7949-hmb7b"
Sep 14 03:31:36.659751 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:31:36.659706    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-559c5f7949-hmb7b"]
Sep 14 03:31:36.661045 ip-10-0-141-4 kubenswrapper[2564]: W0914 03:31:36.661021    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0db99058_dfec_4aa8_87e1_cd74369b453c.slice/crio-229ff4b099d8afa62d63bf160a030e4491bedee6835db513963b148883a47516 WatchSource:0}: Error finding container 229ff4b099d8afa62d63bf160a030e4491bedee6835db513963b148883a47516: Status 404 returned error can't find the container with id 229ff4b099d8afa62d63bf160a030e4491bedee6835db513963b148883a47516
Sep 14 03:31:37.440524 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:31:37.440492    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-559c5f7949-hmb7b" event={"ID":"0db99058-dfec-4aa8-87e1-cd74369b453c","Type":"ContainerStarted","Data":"53c78409a6bed28ed53ad17cd80ab8a2948cd0e4c416265bc8f9fecaf864f20d"}
Sep 14 03:31:37.440901 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:31:37.440539    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-559c5f7949-hmb7b" event={"ID":"0db99058-dfec-4aa8-87e1-cd74369b453c","Type":"ContainerStarted","Data":"229ff4b099d8afa62d63bf160a030e4491bedee6835db513963b148883a47516"}
Sep 14 03:31:37.440901 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:31:37.440578    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-559c5f7949-hmb7b"
Sep 14 03:31:37.467643 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:31:37.467587    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-559c5f7949-hmb7b" podStartSLOduration=1.124937589 podStartE2EDuration="1.467571303s" totalImagesPullingTime="342.633714ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:31:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:31:37.466087537 +0000 UTC m=+2342.847171737" watchObservedRunningTime="2026-09-14 03:31:37.467571303 +0000 UTC m=+2342.848655501"
Sep 14 03:31:43.451649 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:31:43.451622    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-559c5f7949-hmb7b"
Sep 14 03:31:43.511197 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:31:43.511162    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-564fdcb8d5-kbxgg"]
Sep 14 03:31:43.511370 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:31:43.511343    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-564fdcb8d5-kbxgg" podUID="2be1f88b-a0b7-4bdf-8dbc-bfeb9db9f8e7" containerName="maas-api" containerID="cri-o://45bf645935b5d7c5121fbb73df8108d53155e5b9f29b98305d3bb7a114d70d82" gracePeriod=30
Sep 14 03:31:44.751447 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:31:44.751396    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-564fdcb8d5-kbxgg"
Sep 14 03:31:44.805160 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:31:44.805124    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pfs8z\" (UniqueName: \"kubernetes.io/projected/2be1f88b-a0b7-4bdf-8dbc-bfeb9db9f8e7-kube-api-access-pfs8z\") pod \"2be1f88b-a0b7-4bdf-8dbc-bfeb9db9f8e7\" (UID: \"2be1f88b-a0b7-4bdf-8dbc-bfeb9db9f8e7\") "
Sep 14 03:31:44.805344 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:31:44.805201    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2be1f88b-a0b7-4bdf-8dbc-bfeb9db9f8e7-openshift-service-ca\") pod \"2be1f88b-a0b7-4bdf-8dbc-bfeb9db9f8e7\" (UID: \"2be1f88b-a0b7-4bdf-8dbc-bfeb9db9f8e7\") "
Sep 14 03:31:44.805344 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:31:44.805228    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2be1f88b-a0b7-4bdf-8dbc-bfeb9db9f8e7-metrics-certs\") pod \"2be1f88b-a0b7-4bdf-8dbc-bfeb9db9f8e7\" (UID: \"2be1f88b-a0b7-4bdf-8dbc-bfeb9db9f8e7\") "
Sep 14 03:31:44.805344 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:31:44.805278    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2be1f88b-a0b7-4bdf-8dbc-bfeb9db9f8e7-maas-api-tls\") pod \"2be1f88b-a0b7-4bdf-8dbc-bfeb9db9f8e7\" (UID: \"2be1f88b-a0b7-4bdf-8dbc-bfeb9db9f8e7\") "
Sep 14 03:31:44.805658 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:31:44.805624    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2be1f88b-a0b7-4bdf-8dbc-bfeb9db9f8e7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2be1f88b-a0b7-4bdf-8dbc-bfeb9db9f8e7" (UID: "2be1f88b-a0b7-4bdf-8dbc-bfeb9db9f8e7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 03:31:44.807313 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:31:44.807289    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2be1f88b-a0b7-4bdf-8dbc-bfeb9db9f8e7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2be1f88b-a0b7-4bdf-8dbc-bfeb9db9f8e7" (UID: "2be1f88b-a0b7-4bdf-8dbc-bfeb9db9f8e7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 03:31:44.807414 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:31:44.807359    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2be1f88b-a0b7-4bdf-8dbc-bfeb9db9f8e7-kube-api-access-pfs8z" (OuterVolumeSpecName: "kube-api-access-pfs8z") pod "2be1f88b-a0b7-4bdf-8dbc-bfeb9db9f8e7" (UID: "2be1f88b-a0b7-4bdf-8dbc-bfeb9db9f8e7"). InnerVolumeSpecName "kube-api-access-pfs8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 03:31:44.807414 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:31:44.807374    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2be1f88b-a0b7-4bdf-8dbc-bfeb9db9f8e7-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2be1f88b-a0b7-4bdf-8dbc-bfeb9db9f8e7" (UID: "2be1f88b-a0b7-4bdf-8dbc-bfeb9db9f8e7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 03:31:44.906027 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:31:44.905934    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfs8z\" (UniqueName: \"kubernetes.io/projected/2be1f88b-a0b7-4bdf-8dbc-bfeb9db9f8e7-kube-api-access-pfs8z\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:31:44.906027 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:31:44.905967    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2be1f88b-a0b7-4bdf-8dbc-bfeb9db9f8e7-openshift-service-ca\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:31:44.906027 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:31:44.905978    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2be1f88b-a0b7-4bdf-8dbc-bfeb9db9f8e7-metrics-certs\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:31:44.906027 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:31:44.905986    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2be1f88b-a0b7-4bdf-8dbc-bfeb9db9f8e7-maas-api-tls\") on node \"ip-10-0-141-4.ec2.internal\" DevicePath \"\""
Sep 14 03:31:45.481552 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:31:45.481515    2564 generic.go:356] "Generic (PLEG): container finished" podID="2be1f88b-a0b7-4bdf-8dbc-bfeb9db9f8e7" containerID="45bf645935b5d7c5121fbb73df8108d53155e5b9f29b98305d3bb7a114d70d82" exitCode=0
Sep 14 03:31:45.481723 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:31:45.481594    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-564fdcb8d5-kbxgg"
Sep 14 03:31:45.481723 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:31:45.481596    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-564fdcb8d5-kbxgg" event={"ID":"2be1f88b-a0b7-4bdf-8dbc-bfeb9db9f8e7","Type":"ContainerDied","Data":"45bf645935b5d7c5121fbb73df8108d53155e5b9f29b98305d3bb7a114d70d82"}
Sep 14 03:31:45.481723 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:31:45.481696    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-564fdcb8d5-kbxgg" event={"ID":"2be1f88b-a0b7-4bdf-8dbc-bfeb9db9f8e7","Type":"ContainerDied","Data":"8201d47ecd5917d8721df4cdbf969fd7bef4a73e049ea14bef8941fabcd58107"}
Sep 14 03:31:45.481824 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:31:45.481726    2564 scope.go:117] "RemoveContainer" containerID="45bf645935b5d7c5121fbb73df8108d53155e5b9f29b98305d3bb7a114d70d82"
Sep 14 03:31:45.490778 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:31:45.490756    2564 scope.go:117] "RemoveContainer" containerID="45bf645935b5d7c5121fbb73df8108d53155e5b9f29b98305d3bb7a114d70d82"
Sep 14 03:31:45.491118 ip-10-0-141-4 kubenswrapper[2564]: E0914 03:31:45.491093    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45bf645935b5d7c5121fbb73df8108d53155e5b9f29b98305d3bb7a114d70d82\": container with ID starting with 45bf645935b5d7c5121fbb73df8108d53155e5b9f29b98305d3bb7a114d70d82 not found: ID does not exist" containerID="45bf645935b5d7c5121fbb73df8108d53155e5b9f29b98305d3bb7a114d70d82"
Sep 14 03:31:45.491203 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:31:45.491128    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45bf645935b5d7c5121fbb73df8108d53155e5b9f29b98305d3bb7a114d70d82"} err="failed to get container status \"45bf645935b5d7c5121fbb73df8108d53155e5b9f29b98305d3bb7a114d70d82\": rpc error: code = NotFound desc = could not find container \"45bf645935b5d7c5121fbb73df8108d53155e5b9f29b98305d3bb7a114d70d82\": container with ID starting with 45bf645935b5d7c5121fbb73df8108d53155e5b9f29b98305d3bb7a114d70d82 not found: ID does not exist"
Sep 14 03:31:45.503183 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:31:45.503159    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-564fdcb8d5-kbxgg"]
Sep 14 03:31:45.508799 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:31:45.508774    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-564fdcb8d5-kbxgg"]
Sep 14 03:31:47.098226 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:31:47.098194    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2be1f88b-a0b7-4bdf-8dbc-bfeb9db9f8e7" path="/var/lib/kubelet/pods/2be1f88b-a0b7-4bdf-8dbc-bfeb9db9f8e7/volumes"
Sep 14 03:31:56.492116 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:31:56.492083    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5f4895b846-pp7ks_411c0296-8329-4d08-9902-3255d8463066/authorino/0.log"
Sep 14 03:32:02.435466 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:02.435432    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-57d64b7cfb-55l9k_f275a993-2efb-434d-8594-7c0a5ccbb80f/manager/0.log"
Sep 14 03:32:02.828395 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:02.828367    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67ffb7699c-45bzd_8ffdca28-8d3a-4588-a121-972b21c116ac/copy-manifests/0.log"
Sep 14 03:32:02.839627 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:02.839597    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67ffb7699c-45bzd_8ffdca28-8d3a-4588-a121-972b21c116ac/manager/0.log"
Sep 14 03:32:03.479829 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:03.479777    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-559c5f7949-hmb7b_0db99058-dfec-4aa8-87e1-cd74369b453c/maas-api/0.log"
Sep 14 03:32:03.955014 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:03.954979    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-vz9hc_8c3a6db4-6599-4e2f-8af6-d0c937edc94b/postgres/0.log"
Sep 14 03:32:04.760671 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:04.760639    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5f4895b846-pp7ks_411c0296-8329-4d08-9902-3255d8463066/authorino/0.log"
Sep 14 03:32:05.024747 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:05.024716    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-4dfs6_bc928c23-7ce5-4a1a-a199-1a999e819964/manager/0.log"
Sep 14 03:32:06.230467 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:06.230438    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-798674575b-cjnkz_f6249c90-2e4c-4814-9df6-08221ea26ae3/kube-auth-proxy/0.log"
Sep 14 03:32:06.608076 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:06.608042    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-27xw5_ce48fdb0-6167-4a7e-ac9a-20fffb8e7500/payload-pre-processing/0.log"
Sep 14 03:32:06.745875 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:06.745837    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-265v7_c6f51115-9cf5-49ab-97b5-d0a840313cf4/payload-processing/0.log"
Sep 14 03:32:07.520070 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:07.520015    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-fqms6_19776359-d16f-4d5d-a874-fda5a9d0a9a8/main/0.log"
Sep 14 03:32:07.778646 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:07.778609    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f94x6m_3e3a9df9-6fa6-499f-a364-ff126aa6ea0c/main/0.log"
Sep 14 03:32:13.984169 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:13.984131    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:32:14.156017 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:14.155976    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log" limit=1024
Sep 14 03:32:17.629132 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:17.629097    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-zhkkf_18063d29-58fa-4599-8e9c-93c829fa0fb7/global-pull-secret-syncer/0.log"
Sep 14 03:32:17.722829 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:17.722795    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-2hr6w_09389d2f-4da9-4e12-91e1-d616caa472d7/konnectivity-agent/0.log"
Sep 14 03:32:18.005395 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:18.005290    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-4.ec2.internal_d52ca59bd77793b4e757e0f76d119156/haproxy/0.log"
Sep 14 03:32:22.526440 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:22.526400    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5f4895b846-pp7ks_411c0296-8329-4d08-9902-3255d8463066/authorino/0.log"
Sep 14 03:32:22.866232 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:22.866200    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-4dfs6_bc928c23-7ce5-4a1a-a199-1a999e819964/manager/0.log"
Sep 14 03:32:24.825228 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:24.825190    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ee2d96e8-59f2-47f6-af92-13b693d4d7ed/alertmanager/0.log"
Sep 14 03:32:24.865451 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:24.865429    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ee2d96e8-59f2-47f6-af92-13b693d4d7ed/config-reloader/0.log"
Sep 14 03:32:24.909284 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:24.909258    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ee2d96e8-59f2-47f6-af92-13b693d4d7ed/kube-rbac-proxy-web/0.log"
Sep 14 03:32:24.976998 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:24.976971    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ee2d96e8-59f2-47f6-af92-13b693d4d7ed/kube-rbac-proxy/0.log"
Sep 14 03:32:25.031905 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:25.031878    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ee2d96e8-59f2-47f6-af92-13b693d4d7ed/kube-rbac-proxy-metric/0.log"
Sep 14 03:32:25.086914 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:25.086845    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ee2d96e8-59f2-47f6-af92-13b693d4d7ed/prom-label-proxy/0.log"
Sep 14 03:32:25.139282 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:25.139254    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ee2d96e8-59f2-47f6-af92-13b693d4d7ed/init-config-reloader/0.log"
Sep 14 03:32:25.255014 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:25.254982    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-rrskf_09b1dd75-0309-48b0-a9dd-2a997d1444c4/kube-state-metrics/0.log"
Sep 14 03:32:25.288083 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:25.288052    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-rrskf_09b1dd75-0309-48b0-a9dd-2a997d1444c4/kube-rbac-proxy-main/0.log"
Sep 14 03:32:25.319610 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:25.319581    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-rrskf_09b1dd75-0309-48b0-a9dd-2a997d1444c4/kube-rbac-proxy-self/0.log"
Sep 14 03:32:25.369837 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:25.369724    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7f54f5c747-7g79s_bffd2f8e-b7bb-49f5-b563-9c1c1af69a67/metrics-server/0.log"
Sep 14 03:32:25.414029 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:25.414005    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-znr76_349c747c-2054-4d9a-9d6c-9bda627193fd/monitoring-plugin/0.log"
Sep 14 03:32:25.605275 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:25.605249    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tlvfd_68ff2110-81b2-469c-ba95-dab13090e6a8/node-exporter/0.log"
Sep 14 03:32:25.641343 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:25.641263    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tlvfd_68ff2110-81b2-469c-ba95-dab13090e6a8/kube-rbac-proxy/0.log"
Sep 14 03:32:25.696785 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:25.696758    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tlvfd_68ff2110-81b2-469c-ba95-dab13090e6a8/init-textfile/0.log"
Sep 14 03:32:25.878706 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:25.878669    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-qqz7z_6102f536-8eb3-4666-91cd-7781a4cca236/kube-rbac-proxy-main/0.log"
Sep 14 03:32:25.910548 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:25.910466    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-qqz7z_6102f536-8eb3-4666-91cd-7781a4cca236/kube-rbac-proxy-self/0.log"
Sep 14 03:32:25.943369 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:25.943339    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-qqz7z_6102f536-8eb3-4666-91cd-7781a4cca236/openshift-state-metrics/0.log"
Sep 14 03:32:25.992586 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:25.992561    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9dc99581-91d5-43c1-95d6-3c42e3f21de8/prometheus/0.log"
Sep 14 03:32:26.024358 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:26.024333    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9dc99581-91d5-43c1-95d6-3c42e3f21de8/config-reloader/0.log"
Sep 14 03:32:26.054717 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:26.054697    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9dc99581-91d5-43c1-95d6-3c42e3f21de8/thanos-sidecar/0.log"
Sep 14 03:32:26.095425 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:26.095380    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9dc99581-91d5-43c1-95d6-3c42e3f21de8/kube-rbac-proxy-web/0.log"
Sep 14 03:32:26.135780 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:26.135761    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9dc99581-91d5-43c1-95d6-3c42e3f21de8/kube-rbac-proxy/0.log"
Sep 14 03:32:26.165721 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:26.165661    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9dc99581-91d5-43c1-95d6-3c42e3f21de8/kube-rbac-proxy-thanos/0.log"
Sep 14 03:32:26.193343 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:26.193325    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9dc99581-91d5-43c1-95d6-3c42e3f21de8/init-config-reloader/0.log"
Sep 14 03:32:26.234692 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:26.234667    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-5gzd6_660447a7-e634-4653-95f6-736d85f5bdd9/prometheus-operator/0.log"
Sep 14 03:32:26.275179 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:26.275149    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-5gzd6_660447a7-e634-4653-95f6-736d85f5bdd9/kube-rbac-proxy/0.log"
Sep 14 03:32:26.345275 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:26.345225    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-df686ffff-z4729_1dffd99b-f463-4251-96de-0b8d7f6d5b71/telemeter-client/0.log"
Sep 14 03:32:26.374940 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:26.374911    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-df686ffff-z4729_1dffd99b-f463-4251-96de-0b8d7f6d5b71/reload/0.log"
Sep 14 03:32:26.407317 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:26.407286    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-df686ffff-z4729_1dffd99b-f463-4251-96de-0b8d7f6d5b71/kube-rbac-proxy/0.log"
Sep 14 03:32:26.447084 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:26.447009    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76b45698db-7zh9r_cc3d5c64-3f98-4267-9469-27eb14252478/thanos-query/0.log"
Sep 14 03:32:26.473153 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:26.473120    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76b45698db-7zh9r_cc3d5c64-3f98-4267-9469-27eb14252478/kube-rbac-proxy-web/0.log"
Sep 14 03:32:26.513880 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:26.513850    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76b45698db-7zh9r_cc3d5c64-3f98-4267-9469-27eb14252478/kube-rbac-proxy/0.log"
Sep 14 03:32:26.568351 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:26.568323    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76b45698db-7zh9r_cc3d5c64-3f98-4267-9469-27eb14252478/prom-label-proxy/0.log"
Sep 14 03:32:26.602222 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:26.602195    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76b45698db-7zh9r_cc3d5c64-3f98-4267-9469-27eb14252478/kube-rbac-proxy-rules/0.log"
Sep 14 03:32:26.638690 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:26.638654    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76b45698db-7zh9r_cc3d5c64-3f98-4267-9469-27eb14252478/kube-rbac-proxy-metrics/0.log"
Sep 14 03:32:27.038472 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:27.038429    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-f2kzn/perf-node-gather-daemonset-2nvdb"]
Sep 14 03:32:27.038906 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:27.038888    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2be1f88b-a0b7-4bdf-8dbc-bfeb9db9f8e7" containerName="maas-api"
Sep 14 03:32:27.038906 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:27.038900    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2be1f88b-a0b7-4bdf-8dbc-bfeb9db9f8e7" containerName="maas-api"
Sep 14 03:32:27.038993 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:27.038961    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2be1f88b-a0b7-4bdf-8dbc-bfeb9db9f8e7" containerName="maas-api"
Sep 14 03:32:27.042436 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:27.042398    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-f2kzn/perf-node-gather-daemonset-2nvdb"
Sep 14 03:32:27.045787 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:27.045764    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-f2kzn\"/\"openshift-service-ca.crt\""
Sep 14 03:32:27.045917 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:27.045864    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-f2kzn\"/\"kube-root-ca.crt\""
Sep 14 03:32:27.046784 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:27.046768    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-f2kzn\"/\"default-dockercfg-v4n8b\""
Sep 14 03:32:27.058007 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:27.057986    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-f2kzn/perf-node-gather-daemonset-2nvdb"]
Sep 14 03:32:27.100455 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:27.100429    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/dafae1ac-4a1d-487d-9b99-e99e720eeec4-proc\") pod \"perf-node-gather-daemonset-2nvdb\" (UID: \"dafae1ac-4a1d-487d-9b99-e99e720eeec4\") " pod="openshift-must-gather-f2kzn/perf-node-gather-daemonset-2nvdb"
Sep 14 03:32:27.100622 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:27.100499    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dafae1ac-4a1d-487d-9b99-e99e720eeec4-lib-modules\") pod \"perf-node-gather-daemonset-2nvdb\" (UID: \"dafae1ac-4a1d-487d-9b99-e99e720eeec4\") " pod="openshift-must-gather-f2kzn/perf-node-gather-daemonset-2nvdb"
Sep 14 03:32:27.100622 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:27.100557    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64bwq\" (UniqueName: \"kubernetes.io/projected/dafae1ac-4a1d-487d-9b99-e99e720eeec4-kube-api-access-64bwq\") pod \"perf-node-gather-daemonset-2nvdb\" (UID: \"dafae1ac-4a1d-487d-9b99-e99e720eeec4\") " pod="openshift-must-gather-f2kzn/perf-node-gather-daemonset-2nvdb"
Sep 14 03:32:27.100622 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:27.100586    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dafae1ac-4a1d-487d-9b99-e99e720eeec4-sys\") pod \"perf-node-gather-daemonset-2nvdb\" (UID: \"dafae1ac-4a1d-487d-9b99-e99e720eeec4\") " pod="openshift-must-gather-f2kzn/perf-node-gather-daemonset-2nvdb"
Sep 14 03:32:27.100733 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:27.100633    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/dafae1ac-4a1d-487d-9b99-e99e720eeec4-podres\") pod \"perf-node-gather-daemonset-2nvdb\" (UID: \"dafae1ac-4a1d-487d-9b99-e99e720eeec4\") " pod="openshift-must-gather-f2kzn/perf-node-gather-daemonset-2nvdb"
Sep 14 03:32:27.202195 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:27.202157    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dafae1ac-4a1d-487d-9b99-e99e720eeec4-lib-modules\") pod \"perf-node-gather-daemonset-2nvdb\" (UID: \"dafae1ac-4a1d-487d-9b99-e99e720eeec4\") " pod="openshift-must-gather-f2kzn/perf-node-gather-daemonset-2nvdb"
Sep 14 03:32:27.202363 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:27.202216    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64bwq\" (UniqueName: \"kubernetes.io/projected/dafae1ac-4a1d-487d-9b99-e99e720eeec4-kube-api-access-64bwq\") pod \"perf-node-gather-daemonset-2nvdb\" (UID: \"dafae1ac-4a1d-487d-9b99-e99e720eeec4\") " pod="openshift-must-gather-f2kzn/perf-node-gather-daemonset-2nvdb"
Sep 14 03:32:27.202363 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:27.202247    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dafae1ac-4a1d-487d-9b99-e99e720eeec4-sys\") pod \"perf-node-gather-daemonset-2nvdb\" (UID: \"dafae1ac-4a1d-487d-9b99-e99e720eeec4\") " pod="openshift-must-gather-f2kzn/perf-node-gather-daemonset-2nvdb"
Sep 14 03:32:27.202363 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:27.202325    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dafae1ac-4a1d-487d-9b99-e99e720eeec4-lib-modules\") pod \"perf-node-gather-daemonset-2nvdb\" (UID: \"dafae1ac-4a1d-487d-9b99-e99e720eeec4\") " pod="openshift-must-gather-f2kzn/perf-node-gather-daemonset-2nvdb"
Sep 14 03:32:27.202363 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:27.202341    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/dafae1ac-4a1d-487d-9b99-e99e720eeec4-podres\") pod \"perf-node-gather-daemonset-2nvdb\" (UID: \"dafae1ac-4a1d-487d-9b99-e99e720eeec4\") " pod="openshift-must-gather-f2kzn/perf-node-gather-daemonset-2nvdb"
Sep 14 03:32:27.202635 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:27.202374    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dafae1ac-4a1d-487d-9b99-e99e720eeec4-sys\") pod \"perf-node-gather-daemonset-2nvdb\" (UID: \"dafae1ac-4a1d-487d-9b99-e99e720eeec4\") " pod="openshift-must-gather-f2kzn/perf-node-gather-daemonset-2nvdb"
Sep 14 03:32:27.202635 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:27.202489    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/dafae1ac-4a1d-487d-9b99-e99e720eeec4-podres\") pod \"perf-node-gather-daemonset-2nvdb\" (UID: \"dafae1ac-4a1d-487d-9b99-e99e720eeec4\") " pod="openshift-must-gather-f2kzn/perf-node-gather-daemonset-2nvdb"
Sep 14 03:32:27.202635 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:27.202510    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/dafae1ac-4a1d-487d-9b99-e99e720eeec4-proc\") pod \"perf-node-gather-daemonset-2nvdb\" (UID: \"dafae1ac-4a1d-487d-9b99-e99e720eeec4\") " pod="openshift-must-gather-f2kzn/perf-node-gather-daemonset-2nvdb"
Sep 14 03:32:27.202635 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:27.202608    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/dafae1ac-4a1d-487d-9b99-e99e720eeec4-proc\") pod \"perf-node-gather-daemonset-2nvdb\" (UID: \"dafae1ac-4a1d-487d-9b99-e99e720eeec4\") " pod="openshift-must-gather-f2kzn/perf-node-gather-daemonset-2nvdb"
Sep 14 03:32:27.214720 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:27.214698    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64bwq\" (UniqueName: \"kubernetes.io/projected/dafae1ac-4a1d-487d-9b99-e99e720eeec4-kube-api-access-64bwq\") pod \"perf-node-gather-daemonset-2nvdb\" (UID: \"dafae1ac-4a1d-487d-9b99-e99e720eeec4\") " pod="openshift-must-gather-f2kzn/perf-node-gather-daemonset-2nvdb"
Sep 14 03:32:27.352346 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:27.352232    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-f2kzn/perf-node-gather-daemonset-2nvdb"
Sep 14 03:32:27.625925 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:27.625892    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-f2kzn/perf-node-gather-daemonset-2nvdb"]
Sep 14 03:32:27.626635 ip-10-0-141-4 kubenswrapper[2564]: W0914 03:32:27.626610    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poddafae1ac_4a1d_487d_9b99_e99e720eeec4.slice/crio-2940218f7adef6f9d336dbc9ac5db535b92347bdf4bd90aad241c67709b37860 WatchSource:0}: Error finding container 2940218f7adef6f9d336dbc9ac5db535b92347bdf4bd90aad241c67709b37860: Status 404 returned error can't find the container with id 2940218f7adef6f9d336dbc9ac5db535b92347bdf4bd90aad241c67709b37860
Sep 14 03:32:27.689291 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:27.689262    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f2kzn/perf-node-gather-daemonset-2nvdb" event={"ID":"dafae1ac-4a1d-487d-9b99-e99e720eeec4","Type":"ContainerStarted","Data":"2940218f7adef6f9d336dbc9ac5db535b92347bdf4bd90aad241c67709b37860"}
Sep 14 03:32:28.696667 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:28.696633    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f2kzn/perf-node-gather-daemonset-2nvdb" event={"ID":"dafae1ac-4a1d-487d-9b99-e99e720eeec4","Type":"ContainerStarted","Data":"0e91be8ae8bfff6c467f8e68fbe4df97d94d149245c12e0e82ed3e7b42f242c6"}
Sep 14 03:32:28.697088 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:28.696742    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-f2kzn/perf-node-gather-daemonset-2nvdb"
Sep 14 03:32:28.719281 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:28.719235    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-f2kzn/perf-node-gather-daemonset-2nvdb" podStartSLOduration=1.7192206140000001 podStartE2EDuration="1.719220614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 03:32:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:32:28.718075884 +0000 UTC m=+2394.099160083" watchObservedRunningTime="2026-09-14 03:32:28.719220614 +0000 UTC m=+2394.100304811"
Sep 14 03:32:31.080069 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:31.080041    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nwtp8_c9ce7325-f74d-4a0f-8dec-51099aaa1cf3/dns/0.log"
Sep 14 03:32:31.110906 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:31.110870    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nwtp8_c9ce7325-f74d-4a0f-8dec-51099aaa1cf3/kube-rbac-proxy/0.log"
Sep 14 03:32:31.244057 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:31.244022    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-c57s5_789839ca-54f5-4cab-815c-777c86244553/dns-node-resolver/0.log"
Sep 14 03:32:32.092354 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:32.092325    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-whvw4_c3ff549c-82fc-4dec-b893-6aa2b2fa1259/node-ca/0.log"
Sep 14 03:32:33.693878 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:33.693845    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-798674575b-cjnkz_f6249c90-2e4c-4814-9df6-08221ea26ae3/kube-auth-proxy/0.log"
Sep 14 03:32:33.834306 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:33.834274    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-27xw5_ce48fdb0-6167-4a7e-ac9a-20fffb8e7500/payload-pre-processing/0.log"
Sep 14 03:32:33.884903 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:33.884875    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-265v7_c6f51115-9cf5-49ab-97b5-d0a840313cf4/payload-processing/0.log"
Sep 14 03:32:34.712343 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:34.712318    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-f2kzn/perf-node-gather-daemonset-2nvdb"
Sep 14 03:32:35.055103 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:35.055071    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jhnxs_e30169cd-3ac5-476e-bdae-555456b42f7a/serve-healthcheck-canary/0.log"
Sep 14 03:32:35.813334 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:35.813248    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-d7hms_09f4cb2d-a780-460e-aa2d-86eca496c0d1/kube-rbac-proxy/0.log"
Sep 14 03:32:35.854444 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:35.854384    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-d7hms_09f4cb2d-a780-460e-aa2d-86eca496c0d1/exporter/0.log"
Sep 14 03:32:35.882710 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:35.882680    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-d7hms_09f4cb2d-a780-460e-aa2d-86eca496c0d1/extractor/0.log"
Sep 14 03:32:37.682638 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:37.682611    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-798674575b-cjnkz_f6249c90-2e4c-4814-9df6-08221ea26ae3/kube-auth-proxy/0.log"
Sep 14 03:32:37.785987 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:37.785957    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-27xw5_ce48fdb0-6167-4a7e-ac9a-20fffb8e7500/payload-pre-processing/0.log"
Sep 14 03:32:37.832123 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:37.832086    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-265v7_c6f51115-9cf5-49ab-97b5-d0a840313cf4/payload-processing/0.log"
Sep 14 03:32:39.038767 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:39.038731    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-57d64b7cfb-55l9k_f275a993-2efb-434d-8594-7c0a5ccbb80f/manager/0.log"
Sep 14 03:32:39.200554 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:39.200518    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67ffb7699c-45bzd_8ffdca28-8d3a-4588-a121-972b21c116ac/manager/0.log"
Sep 14 03:32:39.225321 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:39.225295    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67ffb7699c-45bzd_8ffdca28-8d3a-4588-a121-972b21c116ac/copy-manifests/0.log"
Sep 14 03:32:40.012651 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:40.012622    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-798674575b-cjnkz_f6249c90-2e4c-4814-9df6-08221ea26ae3/kube-auth-proxy/0.log"
Sep 14 03:32:40.090619 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:40.090544    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-27xw5_ce48fdb0-6167-4a7e-ac9a-20fffb8e7500/payload-pre-processing/0.log"
Sep 14 03:32:40.120719 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:40.120691    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-265v7_c6f51115-9cf5-49ab-97b5-d0a840313cf4/payload-processing/0.log"
Sep 14 03:32:40.731962 ip-10-0-141-4 kubenswrapper[2564]: I0914 03:32:40.731931    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-59b9ccfbf8-2xdrk_71f93d7d-55d4-43d5-bdb4-53e942788755/manager/0.log"